Pokemiles are calculated by the distance between your home location and the other player's home location. So by Streetpass, you're mostly getting people from...basically 0 miles away.
Streetpass is generally pretty useless for Pokemon XY for this reason, and I would personally suggest ignoring it. If you want more, you could make sure to walk/drive past Streetpass relays, but your time would be much better spent simply wondertrading a few pokemon to get the Pokemiles. Since a good percentage of wondertrades are from Japan and the US, which are of course a good distance away, wondertrading will very often yield a fair number of miles.
