The stacking penalty applies to the bonus, not to the module.
If you have a thermal hardener and an EM hardener fitted, both would provide the full resist bonus. If you add omni-resist plating, its thermal and EM bonuses would be reduced because of the stacking penalty, but its explosive and kinetic bonuses would not.
The precise formula is somewhat complicated, but it can be summarized as:
- The first module provides 100% of its bonus.
- The second provides ~87% of its bonus.
- The third provides ~57% of its bonus.
- The fourth provides ~28% of its bonus.
- The fifth ensures your fit gets posted in a "fail fits""fail fits" thread on the forums.
The order you fit the modules doesn't matter; the one with the largest bonus will always be considered "first".
Some modules (notably Damage Control Units) are exempt from stacking penalties.