In skyrim, I've noticed that sometimes, when fighting, a cutscene occurs, and my character kills the enemy I was fighting. Is this random, or is there a specific way I can do this? Could I use this to, say, sneak up behind an enemy with a dagger and backstab them or something?
This is just a random kill cam. While it used to be restricted to just melee weapons a patch to the game has introduced it to just about every weapon in the game including spells now.
As for the triggering, it can happen when your attack would kill the target not you kill the target because it happens. So to that end, it can not be used as a way to instantly kill things... Although an 'assassin' type person can often one hit kill most things from stealth and thus will see this animation quite often.