After a ladder match, the only way I can invite that person seems to be by adding them as a friend or inviting them to my party. Is there no way to invite them to a game by using the match history / score summary screen?
Unfortunately not after a ladder match.
A ladder match is a random opponent whose MMR rating is similar to your's. It's how the system determines if you should be moving up or down in your ladder. A second game with that person would not be random and would have no effect (which is probably your intention) on the ladder system.
Like you said, the only way is to either friend them and then invite them to a match or invite them to a party after the game.