Yes it is !
The % damage in those effects is doing magic damage once calculated, so let's say its 10% magic dmg on 1000hp, then the result will be 100dmg, to which you apply the % of magic resistance so to finish the example let's say 50% magic resist. You apply this 100dmg x 50% MR = 50dmg.
That's amumu Despair skill (the W), as you can see it clearly is magic damage once calculated, so resistance apply to it.
And that is vayne's W, where it is stated that it is true damage, so no resistance is applied
I just noticed you asked for botrk too, sorry. It will work exactly the same way but with armor as the % from botrk is calculated as physical damage. So once the % is calculated based on target's hp, the % damage reduction from armor will apply. So, botrk is really efficient against high hp, low armor champions, but if they start to build a lot of armor you will need a black cleaver or a last whisper to penetrate armor if you want the effect of botrk to still be important.
Yes, physical and magical damage will get reduced by armor and magic resistance.
There is also the common misconception that you "have" to buy percentage damage against Tanks. While this is true to a certain degree, it will only help if they have a lot of HP without resistances, If someone has 200 Armor and 3000 life, a Bloodthirster with some crit will often do just as well as a BotrK.
The percentage is calculated first and the damage reduction after. So if you have our favorite spin to win champion with 100 armor (50% dmg reduction) and 1000 HP, your blade will do a passive 40 physical damage.
The same goes for effects like Brand Passive and Amumu W of course just with Magical damage and magic resistance.