It does work, but as you've made mention, most people that work in organized teams (guilds, Arena teams, pick up groups/raids) end up using a Ventrillo or Teamspeak server that someone has access to.
Functionality-wise, it's no different than using a voicechat server. You'll set up a push-to-talk key and utilize that for whenever you want to voice something. I'm not sure if there's an "always on" option, or a voice detection option to auto-broadcast when a decibel threshold is hit, but I don't believe so. Edit It does have auto-detection/always-on as well, thank you for the correction.
Generally speaking, the in game voice chat is very low quality compared to what you'd get on a Vent/TS server. Additionally, it's exactly the reasons you mentioned above that people tend to not use it. You can prevent certain group situations from coming through via voicechat (e.g. specify that you only want to hear it in groups, not in battlegrounds or raids), but it's still subject to the same abuse that an admin-less voicechat server would face, which causes people to inherently distrust it.
edit: And as Raven Dreamer has mentioned in his comment to your initial post, the majority of guilds are already set in their ways of using 3rd party software to do the same thing. There was little need / desire to change, especially at the relatively low cost per month of having a much higher quality and customizable option for your guild/friends.