Every increase in difficulty affects many different aspects, for example
- Less base happiness
- Barbarians being stronger, more aggressive and worth less gold
- Smaller rewards from ruins
- AI civs getting more hostile
- ... and many more
However, probably the most significant change in the higher difficulty settings is the many boosts AI civs receive, which include
- Free units and technologies as the game starts
- Cheaper purchase costs
- Faster construction speed
- Greater bonus against barbarians
- Lower unhappiness
- ... and again, many more
A more concrete example: playing in the Emperor difficulty means the AI starts with Pottery and Animal Husbandry researched; gets an extra warrior and a scout; units cost it only 75% of their worth; unhappiness is only at 85%; etc.
A full table of the affects on both players and AI is available at civfanatics.com
Personally, I feel disadvantaged the most at the early stages of the game, where you struggle desperately against every barbarian camp while they're dominating with their extra soldiers and reach new eras faster, not to mention getting all the wonders because of the production and technology boosts.
You can see a more precise listing of exact effects in the XML file <civ5 installation folder>\Assets\Gameplay\XML\GameInfo\CIV5HandicapInfos.xml.