I have since done some tests on the Spy callout, and it seems that it's only based on hitboxes in your line of sight: In other words, if you look at someone, even if that person is under full cloak, then you can call them out as a Spy.
The callout also works at range: you could be looking at a Sniper on the other side of the map and still be able to call him a Spy.
Contrary to @scarothjagaroth's and my earlier comments, It is not possible to call someone a Spy through:
- Opaque Walls
- Transparent Walls
- Other Players
Here's a video I made testing the callouts, hitboxes and the different scenarios listed above:
Now onto the specific example situation you give. In the heat of battle you might have thought you were looking at the Demoman you've ran past, but the server may have either
- Calculated hitting an Engineer that was far away (as your view swings past his hitbox)
- Called out a Spy under cloak (that was disguised as an Engineer)
In conclusion, the 'Spy' call is probably working as intended. Most good players will be extra aware of all 'friendly' classes around them anyway if people are shouting 'Spy!', not just the class or player you specifically called out. So long as you made your team aware of a Spy in the vicinity, it shouldn't matter too much who you've accused.