Actually, the size has a lot to do with the Pokémon's stats in the game. I leveled up two Pidgeys until they were both at the same point on the CP meter. One was an XL and the other regular. The XL's CP was 79, while the other's was at 73. So imagine the margin between an XL and XS Pokémon. It could be a very big CP difference. 

And to the above theory that an XL Pokémon requires more stardust and candy - it's simply not true. I have an XL and regular Venonat at the same spot on the CP meter and they both take 400 stardust and one candy to upgrade.