It's possible to break enemy troops out of an overwatch state but is there a way to stop Mutons from counterattacking when melee attacked?
Flashbangs, according to this XCOM Wiki page (emphasis mine):
Prevents use of any and all special abilities - Stun Lancers will not use their blades, Officers will not use grenades, Mutons cannot even counter-attack in melee, if Disoriented.
I have not been brave enough to confirm this for myself, however.
In addition, some Psi Operatives abilities (Insanity and Void Rift) may Disorient enemies for the same effect, but this is somewhat less reliable.
Psi-Powers. If you can disorient your opponent (There is a Psi-power that can do this however I forget it's name right now.) most of their abilities such as Dodge and Counter won't work.
This is in addition to the other answer about flashbangs.