I know that if I postpone my Pokemon's evolution, I can learn certain moves earlier. For example, Mareep learns Thunder at level 55 while its fully evolved form, Ampharos, only gets it at level 79.
Obviously, Pokemon who change types by evolving get the benefits of their new type, but for everything else, why should I evolve early as opposed to waiting for good attacks?
The advantage gained is improved stats. The evolved form of a Pokemon has better stats than its earlier forms. However, when you evolve your Pokemon their stats are recalculated from level 1. So your fully evolved Pokemon will have the same stats at level 100 regardless of which level you evolve it at.
There are two disadvantages to evolving a Pokemon though. The first one is pretty minor, and is that a Pokemon takes longer to learn moves in an evolved form (as you noted in your question). The other disadvantage is fairly significant though, and only applies to Pokemon which evolve with an evolution stone. The majority of Pokemon which evolve using a stone won't learn any moves when evolved, so you don't want to evolve those Pokemon until they've learned all their possible moves. Evolving them earlier will really, really limit their usefulness.
The evolved form of a Pokémon always has better (or equally distributed *) base stats than its pre-evolved form (with the exception of Shedninja, because of its 1HP stat). Essentially, your attacks will be dealing more damage, and you will be able to take more damage.
The only reason you would want to stop your Pokémon from evolving is if it learns a move that its evolution doesn't learn (for example, Growlithe learns Flare Blitz but Arcanine doesn't, so you may not want to use a Fire Stone on it right when you acquire one).
* Scizor is interesting in that it has the same total base stats as Scyther – when evolving, Scyther's speed gets reduced, but the points removed go to Scizor's attack and defense stats.
There are two principle reasons to evolve your pokemon.
The first is base stats. Almost without exception, an evolved pokemon's base statistics, that is to say, ATTACK, SP. ATTACK, DEFENSE, SP. DEFENSE, SPEED, and HP are higher than its previous form. This translates to an immediate boost in battling abilities.
The second is that pokemon movepools often expand with evolution - the pokemon is able to learn abilities it would otherwise not be able to (often related to its new type, if applicable). While generally speaking, non-evolved pokemon learn attacks earlier than their evolved counterparts, it is not always true, and depending on the individual scenario, there may be no reason to delay evolution.
Additionally, if the move in question is learn-able via TM or Move Tutor, it can be unnecessary to wait until a certain level in the first place - you're just missing out on the higher stats at that point.
As of Generation 6, there is another advantage to keeping your Pokemon unevolved: Bonus Exp. Points.
Any Pokemon that evolves at a certain level will receive bonus Exp. Points if its level is at least the level it evolves at. So if you have an Abra (for example) at level 16 or higher, it will receive bonus Exp. Points until you let it evolve to Kadabra. This applies as soon as the Pokemon reaches that level and afterwards gains Exp. Points, so it can sometimes happen unknowingly in Trainer battles if your Pokemon reaches the Level it evolves at, but the opponent still has Pokemon left and you have that Pokemon gain Exp. Points from them.
What this means is that starting with X/Y, if you just want to get your Pokemon to high levels, you should not let it evolve as it will gain levels faster as long as it tries to evolve, but you don't let it.
Testing indicates the multiplier is either x1.25 or x1.3.
Since there are currently no online sources to confirm this (that I'm aware of), here is the data I aquired through a personal test to confirm this.
I had a Lv.46 Vanillish with Lucky Egg and fought 3 Marchionesses with Lv.55 Audinos (Battle Chateau, Black Writ of Challenge active).
The first Audino yielded a little over 6000 Exp. Points, which brought Vanillish to Lv.47 (where it evolves). The next Audino yielded slightly more than 8000. After the battle, I canceled the evolution and fought the next Marchioness. Both Audinos gave ~8000 (The same number as before). I then let Vanillish evolve and fought the third Marchioness, this time both Audinos gave ~6000 again.
Really, the decision is up to you. Like Kevin Y said "Growlithe learns Flare Blitz but Arcanine doesn't" but then also if you evolve it will probably learn a move that the non-evolved will not know, but then it's all your choice how you want your pokemon to grow.
