Sometimes when I leave a group I joined from the dungeon finder mid-run, I get a debuff that prevents me from joining another one for 30 minutes. It doesn't happen all the time, though. Exactly what triggers this?
The Dungeon Deserter debuff was implemented when the Dungeon Finder first came out, and has had several revisions since. It's intended to force you to stick with a group for a certain period of time rather than dropping group based on your own personal bias (for example, maybe you really don't like Paladins and the group you just got slotted into has four of them).
As it is designed now, you should receive the Dungeon Deserter debuff if you leave the group before the last boss is defeated (the last boss is generally the boss listed on the dungeon's achievement, for instances where you go after the last boss before optional bosses).
This is all not to be confused with the hidden-ish Dungeon Cooldown debuff, which prevents you from requeueing for a random dungeon for 15 minutes after entering a dungeon. This is to prevent pre-made groups from "rolling the dice" over and over in rapid succession to get the "random" dungeon they prefer.
However, the implementation of both debuffs has left a lot to be desired, and it's fairly easy to avoid getting either debuff. It's definitely considered an exploit to intentionally circumvent them, and Blizzard is constantly working on ways to improve exploit detection, but you generally won't receive the debuffs for things outside of your control.
For example, you'll avoid getting the debuffs: