Resistances are a flat reduction, after armor is taken into account. So if you have 20% fire resistance, you take 20% less damage from a packet of fire damage. It works the same way for a vulnerability.
I'm playing on Hard, and matching offensive elements to enemies is definitely useful, but not so useful that I would carry around multiple sets of weapons. You can usually just power though that.
If you are having trouble with a fight, going back with the appropriate elemental resistance potion equipped, however, could definitely turn the tide.
Some of the higher-tier dragon fights might be worth speccing against. Dragons have a ton of health, and the longer the fight goes, the greater the chance that something could go poorly for you.