As of the 7 July 2016 update, this behaviour is supposed to be fixed; there should no longer be any difficulty with aiming reflections while near teammates. The original answer follows.
Okay, I think I know what is happening here. This is basically a total guess, but it seems like the kind of thing that would happen in a Valve Source game.
Figure 1: A typical reflect. (I hid the rocket to see things better.)
Basically, the game checks where you're looking and directs the rocket that way.
Figure 2: A reflect with a taunting Scout in your face.
The teammate's axis-aligned world collision box is blocking your "vision", making the rocket's target point much closer to you, and so making it go a weird direction. It's just what happens when your targetting vector (your vision) isn't coming out of the same place the attack is: it happens at every range, you just don't really notice unless it's point-blank. The Wrangler is a more extreme example of this kind of behaviour.
So what can you do about this? Not much really, other than walk away from your teammates so their larger-than-reasonable invisible wall won't interfere.
Since making this educated guess of an answer, I have found this video that proves it correct.