I extracted the first cinematic video to analyze it from MPQ archives (it is called
Cinematic_1_Intro.ogv). I checked its properties using VLC and the video codec seems to be Xiph.org's Theora Video.
While audio codec is Vorbis Audio (vorb).
You can use MPQ Editor if you want to try extract cinematic videos and play them outside Diablo 3 client.