It is mostly random. The army usually chooses the closest high development province, but can also move very far away. If a nation is rather small, the capital province is very likely to be chosen.
From the wiki
If an army loses a battle while having low enough morale to be disorganized, they will be forced to retreat to a controlled province (owned, allied in war, or occupied by player or allies). This province can be very far away from where the battle took place. They will usually retreat to one with high development, a fort, and no adjacent enemies. While retreating, it cannot be engaged in combat or be controlled until it reaches the safer province (or in extreme circumstances if it recovers to 100% morale before reaching the destination). The army also moves slightly faster, and will recover morale at a normal rate during the retreat. If there are no available controlled provinces to retreat to within a large range, the army will shattered retreat to one province away. The army can then be immediately re-engaged, often with 0.00 morale (assuming a monthly tick has not yet completed). This can be devastating as an offensive tactic, but can also affect a player as well.
Offensively you pretty much have no choice but to follow the defeated army to find out where it goes. Defensively it is often a good idea to manually retreat (move your army away from the battle, retreat is only possible after 12 days). If you do this while your armies morale is still above 0.5 you can choose which province it retreats to.