It's a question of time. I want them to gather, and go to my target (a siege, or a weaker lord wandering and too slow). If I summon them while in my castle, do they go to that castle first? Or does the AI just direct them towards me?
If you're marshal and summon your forces, they will always attempt to come directly to you unless you've spoken to the lords indiviually to give them different orders. There are other scenarios where lords won't come directly to you, but these can be vaguely summarised as prioritising 'household business' over 'realm business' (for instance, they may be low on troops). Lords may also fail to attend if they're feasting, but usually the marshall's call will end active feasts.
I turned on cheats and then used Truesight to see all parties on the map. The results were inconclusive. The summons to start a war party affected 3 lords, but only 1 was far enough away to test, and he wandered quite a few times to chase caravans. Ultimately, it may not really matter, since the distance you can travel after summoning a war party is not that far.