Are there some common rules on which cards to discard immediately?
That's pretty hard to say because it really depends on your build and playstyle. But the cards that cost 5-10 mana should in most cases be removed without thinking. Unless they're some really powerful high quality cards in which case it could be smart to keep one but only if your other cards seem good.
I think it's not worth risking early game by keeping high cost cards unless you're sure you can pull it off.
Besides the mana cost, which factors should I take into account making this decision?
Some spell cards are more effective late game, even if they cost low amount of mana. Rather remove them unless you feel like they might come in handy early game. Keep good early game spells, especially those that cost low amount of mana and are able to destroy enemy monsters.
I'd suggest keeping good taunt cards that are up to 5 mana cost. Taunts are very useful and you never know will you get one when you'll really need him later.
Low cost minions that let you draw more cards in any way should be kept as well. Quite useful.
In the end there's no magical formula which will tell you what to keep since you can't know what you're going to get next by drawing. Sometimes you remove cards you don't need early game just to get them replaced with other un-needed cards.