Each system you add to your empire has two different costs:
- While it's an outpost, it has a Dust cost, labeled as "Distance to Empire".
- Once it's a colony, it receives Expansion Disapproval: between -6 and -10.5 Approval (depending on difficulty) per system in your empire above 2. There are four technologies in the Exploration & Expansion tech tree (bottom) that will reduce this by 22% each.
Note that outposts generate Expansion Disapproval for your other systems, but they don't experience the penalty themselves.
Expanding to multiple planets within a single system has a different cost: Overpopulation Disapproval. For every population point above 3 in a system, that system gets -1 to -1.6 Approval (depending on difficulty), which can be reduced up to 80% by the improvements Unlimited Information Highways and Permanent Vacation (which also have a flat Approval increase). This is relatively minor compared to Expansion Disapproval: even a heavily-colonized system won't get above -40 or so Approval this way, and by the time you have this much Overpopulation Disapproval, you have so much FIDS that you can afford to buy, build, or invent some Approval-generating improvements.
Expanding to multiple planets also has a flat penalty to Approval in that system: every planet worse than Class I has a -5 Approval penalty per Class. So Terran worlds have no approval penalty, but Tundra worlds are -5, and Gas Giants are -20.
So, to answer your question: your ability to expand is limited by your ability to build Approval-generating improvements, and your ability to pay the Dust costs to support distant outposts for the 30 turns before they become colonies. These limitations go away with sufficient technology and improvements, so bigger empires are better as long as you can handle the costs.