For the PS3. Performing a finishing move can happen randomly in a fight but can be
done on command by tapping R1 and flicking the left analog stick
forward. A finisher factors in how strong the player is against its
enemy and how the player's character is positioned.
Also, after doing a bit of testing, to perform Magic/Archery finishing moves(besides the randomly occurring ones), you need to have the target moving, having less than 5% of their health, or be unaware of your presence, and your attack be strong enough to kill them.
and I think your problem is:
Since Magic/Archery finishing moves were added in patch 1.5, you will
need an Internet connection to your PS3 to install/update to the
current patch.
If you want to learn more:
http://elderscrolls.wikia.com/wiki/Finishing_Moves