Your assumption is correct - 5* artifacts with 3 initial substats has less potential than 5* with 4 substats.
How it works (based on community research):
There is a set of predefined values for each substats - 4 different values (they are called "tiers" in the table below), they are also different for artifacts rarity - 5* Tier 1 is better than 4* Tier 1. Initially, your 5* artifact receives either 3 or 4 substats. Then it receives additional substat on +4, +8, +12 +16 and +20 upgrades, 5 upgrades total. For any 3 substats artifact the first upgrade will always add missing 4th substat, then each next will add additional value to one of the 4 substats. Additional values could be of any tier, so T1 substat could receive T4 upgrade or vice versa.
So, 5* 3 substat artifacts will have 3 + 5 = 8 substat values total, while 5* 4 will have 4 + 5 = 9.
This is the table with all possible substat values:
Source - this reddit post
P.S. this game is not some highly competitive PvP game, so trying to get the best possible artifacts just not worth it - you can clear all the content with all average teams with any average gear.