Combining two runes will give you a random rune of the same tier. Combining 5 runes will give you a random rune of the next highest tier. You cannot combine 5 tier 3 runes at once (as there is no tier 4).
The rune combiner is random. It doesn't matter whether you combine marks, glyphs, seals, or quintessences -- they all work with each other, and the output can be anything.
The idea is that, generally, runes themselves are priced such that it is slightly cheaper to generate a tier 3 runeset from your old runes than by buying the tier 3 runes on their own. For instance, the level 1 +AD rune is 15 ip, while the level 2 +AD rune is 80 ip -- if you bought 5 of the level 1 runes, combined them, and wound up with a level 2 +AD rune, you would have saved yourself IP.
Unfortunately, as mentioned earlier, this process is completely random. If you have tier 1 and 2 runes, by all means, combine them to higher levels; the rune combiner is just a way to possibly recoup the IP you spent on runes you no longer find useful, but it cannot be counted on to give you what you want. You might luck out and combine 2 random runes into a +Dodge quintessence (very expensive!) -- or you might get a mana/5 Mark (with a tertiary stat, and probably not worth the IP cost of your combined runes).
It's very much a gamble, and you don't get anything exclusive from it, so use it if you need to get rid of T1 / T2 runes, and safely ignore it otherwise.