It's finally come: The day you want to release the chain of rules and instead use the rules to rein in creations of your own. Maybe the inclination has always been there, urging you forward, or maybe it's that your players know each of the monsters in the game a little too well. How much of a challenge does each change pose to your players? Even though each shift boils down to simply the experience of comparison, there are different factors to weigh in regards to homebrewing creatures, abilities and rules.
Experience of comparison is a little different than plain, broad experience. An experienced GM has a lot of time behind them but there's little telling what type of time that might be. They could run exclusively others' adventures, for instance. Grasping comparison especially is crucial for bringing your own rules to the table. Suppose the rule, ability or creature in question seems amiss at first play. Instead of writing it off, you benefit the game more if that base rule, ability or creature is played out in a variety of ways. Understand what it is you want to change inside and out, that way when you do implement a change you understand exactly what it needs to replace and what it needs to take into consideration. Aside from replacement, you could wish to add something new. Find as many other similar things which already exist in the game as you can and get a good feel for how they operate. Very often this can give you ideas on how to balance your creation.
Creatures pose a special challenge because they are regularly designed for the players to fight and overcome. Challenges of this sort normally generate experience points, and each of them pose a different level of difficulty. Some monsters are greatly enhanced by the presence of certain obstacles or environments (such as darkness or underground), which is something a GM must take into consideration for the group. If you are shooting for a target difficulty level then play as many of the other creatures in that difficulty level as you can. Test your monster against what the group will look like at that level and possibly even other groups of the same level. Find out how your creature plays out in both it's most advantageous situation and it's least. Also remember that any creature you introduce to the setting has the potential to be a player's next choice of race (for some), companion or minion.
Abilties are trickier than creatures. Not every creature brings a new ability to the table but abilities bend the rules slightly, sometimes in very unique ways. Your strategy here begins with testing out other similar abilities and even other abilities you consider to be on the same power level. If it's a monster ability, give it first to monsters already in existence and test it's efficacy. Grant characters the abilities in playtests and gauge if they swing battles or not. Run it through as many scenarios as you can, always looking for ways it can be used to break instead of merely bend the rules. Find who can most use the ability with the greatest effect and determine if it's too powerful for them at that given stage of the game.
When bending the rules is insufficient, it's time to outright break them. A broken rule which is rewritten so that the new rule is more fair for your game table is called a house rule. If your group detests the way two-weapon fighting works or that spellcasters get so few spells or that enchanted weapons have to take so long to create, you as GM can easily rule around that. Contrasted with creatures and abilities, this is both the greatest boon and greatest bane to your game table. Depending on the rule in question, entire systems can have a shift in power. This can stem to something so small as to dictate which spellcasting classes are the most powerful to a large scale alteration in the relative power between martial and stealthy characters. When you create house rules, the game loses it's universality and becomes "your group's" game. Some players I know might not tweak all characters gaining spellcasting too bad while others would brutalize the game with even a paltry amount of spells on their thief. Do what your group feels is best and remember that you can always alter your house rules to incorporate new circumstances. The final important note about house rules is that you should be upfront with your players, both current and future, so that everyone knows exactly what game they are playing. Happy homebrewing!