Its because you are only being shown offensive retainers (retainers that affect your units). Defensive retainers are not shown to the enemy (though I believe there is a retainer that reveals your opponents defensive retainers).
There is no penalty to taking max retainers. In fact, there is a penalty for NOT taking the maximum, because your general becomes more expensive to deploy with rank-ups, and so it is necessary to take the fullest advantage of your general as he becomes more expensive.
Alternatively, your opponents could have prestige'd and thereby only have one retainer slot at the moment.