Compared to, for example, completing 5 bounties in each of 5 different games.
No, I've tried this twice. You just get the standard rewards from clearing the bounties in the act.
If you talk to Tyrael after completing the last act, he can even incorrectly suggest that some other town needs your help.
Do note that two acts grant double the usual number of blood shards when you complete the bounties for those acts (marked with "Bonus" on the world map). So it's worth at least doing two acts worth of bounties before making a new game.