It is possible to get someone's magic resist/armor below zero with proper items/skills/spells. Like you said magic/armor penetration doesn't do that. Only % magic/ resist/armor reduction and flat magic resist/armor reduction.
MAGIC RESISTANCE REDUCTION
ITEMS
- Abyssal Scepter: Unique Aura: Reduces the magic resistance of nearby
enemy champions by 20. +70 ability power, +45 magic resistance. 2650
gold.
- Malady: Unique Passive: Your basic attacks reduce enemy magic
resistance by 4 for 8 seconds. Stacks up to 7 times. +25 ability
power, 50% attack speed. Unique Passive: Your physical attacks shred
your target doing 15 + 10% of your ability power as bonus magic
damage. 2035 gold.
ABILITIES
Flat magic reduction:
- Amumu's Cursed Touch passively makes his autoattacks reduce the
magic resistance of enemies hit by 15 / 25 / 35 for 3 seconds.
- Fiddlesticks's Dread passively reduces the magic resistance of
nearby enemies by 10.
- Heimerdinger's H-28G Evolution Turret, at rank 2 and above, reduces
armor and magic resistance by 1 per hit for 2 seconds (effect stacks
up to 50 times).
- Kog'Maw's Caustic Spittle reduces the magic resistance of the target
by 5 / 10 / 15 / 20 / 25 for 4 seconds.
- Morgana's Tormented Soil reduces the magic resistance of all targets
by 4 / 5 / 6 / 7 / 8 each second for 5 seconds.
- Ryze's Spell Flux reduces the magic resistance of all targets by 12
/ 15 / 18 / 21 / 24 magic resistance reduction for 5 seconds.
- Soraka's Starcall reduces the magic resistance of all targets by 8 /
9 / 10 / 11 / 12 for 8 seconds (effect stacks up to 10 times).
Percentage magic reduction:
Trundle's Agony steals magic resistance of the target by 15% / 20% /
25%, this amount is doubled over the next 6 seconds.
Jayce's Transform: Mercury Cannon grants his next auto attack to
reduce the enemy's magic resist by 10% / 15% / 20% / 25% for 5
seconds.
Karthus's Wall of Pain reduces the magic resistance of all targets
affected by 15% for 5 seconds.
Kayle's Holy Fervor allows her autoattacks against enemy champions
reduces their magic resistance by 3% for 5 seconds (effect stacks up
to 5 times).
Nidalee's Bushwhack reduces the armor and magic resistance of
enemies that step on the trap by 20% / 25% / 30% / 35% / 40% for 12
seconds.
SUMMONER SPELLS
- Exhaust, when augmented by Summoner's Wrath, reduces the target's
magic resistance by 10 for its duration.
ARMOR REDUCTION
ITEMS
- The Black Cleaver: Unique passive: Dealing physical damage to an
enemy champion reduces their Armor by 6.25% for 4 seconds. This
debuff stacks up to 4 times.+250 health, +50 attack damage, +10%
Cooldown Reduction. Unique: +10 Armor Penetration. 3000 gold.
ABILITIES
Percentage Reduction:
- Jarvan IV's Dragon Strike reduces the armor of all enemies hit by
10% / 14% / 18% / 22% / 26% for 3 seconds.
- Jayce's Transform: Mercury Cannon reduces the armor and magic
resistance of the enemy hit by his first attack after cast, by 10% /
15% / 20% / 25% for 5 seconds
- Kayle's Holy Fervor reduces the armor and magic resistance of
enemies hit by her autoattacks by 3% for 5 seconds. This debuff
stacks up to 5 times.
- Nidalee's Bushwhack reduces the armor and magic resistance of
enemies that step on the trap by 20% / 25% / 30% / 35% / 40% for 12
seconds.
- Renekton's Slice and Dice, when empowered by having more than 50
Fury, Dice will reduce the armor of all enemies hit by 15% / 20% /
25% / 30% / 35% for 4 seconds.
- Shyvana's Flame Breath reduces the armor of all targets hit by 15%
for 4 seconds.
- Trundle's Agony reduces the armor and magic resistance of his target
by 15% / 20% / 25% and gains for the same amount. This amount is
doubled over the next 6 seconds.
- Urgot's Noxian Corrosive Charge reduced the armor of all enemies hit
by 12% / 14% / 16% / 18% / 20% for 5 seconds.
- Vi's Denting Blows reduces the target's armor by 20% for 4 seconds
after every third consecutive hit.
- Wukong's Crushing Blow reduces the armor of the target hit by 30%
for 3 seconds.
- Xin Zhao's Challenge reduces the armor of targets hit by his basic
attacks or Audacious Charge by 15% for 3 seconds. Only one target
may be challenged at a given time
Flat Reduction:
Corki's Gatling Gun fires automatically every half second, reducing
enemies armor with each hit by 2 / 4 / 6 / 8 / 10 for 2 seconds.
Heimerdinger's H-28G Evolution Turret, while in ranks 2, 3, 4 and 5,
the turrets' attacks reduce armor and magic resist by 1 per hit
stacking up to 50 times.
Kog'Maw's Caustic Spittle reduces the target's armor and magic
resistance by 5 / 10 / 15 / 20 / 25 for 4 seconds.
Nasus's Spirit Fire reduces the armor of all enemies in the area by
20 / 25 / 30 / 35 / 40 for the time they are on it plus 1 second.
Rammus's Puncturing Taunt reduces the target's armor by 10 / 15 / 20
/ 25 / 30 for 1 / 1.5 / 2 / 2.5 / 3 seconds.
Taric's Shatter reduces the armor of all enemies hit by its
activation damage by 10 / 15 / 20 / 25 / 30 for 4 seconds.
SUMMONER SPELLS
- Exhaust, when augmented by Summoner's Wrath, reduces the target's
armor by 10 for its duration.
SOURCE
LOL WIKI
EXAMPLE
An easy example that everyone can try out and find out if armor can actually go negative is this:
Heimerdinger's level 1 Armor is 10. So get Corki and get exhaust with the Summoner's Wrath mastery.
Go meet each other at a lane (level 1) and exhaust him and use Gatling Gun on him for 2 seconds and watch his Armor become 10 - 10 - 8 = -8!!
