I'd assume Splatfest Battle (Open) is just standard matchmaking. The game will try to find people of similar skill level with relatively good connection to play with you.

Splatfest Battle (Pro) on the other hand is what other games consider "Ranked Matches", i.e. these will influence a matchmaking rating putting you into class/categories/tiers/leagues. This also tries to find people of similar skill level to play with, but will also keep them from the same general tier/class you're in.

After a quick look into it, you could differentiate them this way:

Both do matchmaking trying to put you into games with players of similar skill level and relatively good connection to each other.

However, in Open, you're able to bring your friends with you (so this way you might bring a way weaker or way stronger player with you). In Pro you're restricted to queue solo and the game will strictly try to find something it considers fair and with you not being able to influence who you're playing with.