Just as their names imply, Magic Penetration allows you to ignore a certain amount of Magic Resistance, whereas Magic Resistance Reduction reduces an enemy's Magic Resistance.
Noticeably, since MR Reduction (also known as MR shred) actually reduces an enemy's Magic Resistance, an MR shred item will benefit anyone on your team who deals Magic damage. Magic Penetration, on the other hand, will benefit only yourself.
In addition, it's important to know that MR Reduction is applied before Magic Penetration. This means that if you can reduce an enemy's MR to 0, then any Magic Penetration you have becomes worthless.
Flat Magic Penetration vs. Percentage Magic Penetration
It's also important to note the difference between Flat Magic Penetration and Percentage Magic Penetration (as well as Flat Magic Resist Reduction and Percentage Magic Resist Reduction).
Flat Magic Penetration stacks additively, so if you have both Haunting Guise (+15 MPen) and Sorcerer's Shoes (+15 MPen), you get 30 total magic penetration.
Percentage Magic Penetration, however, stacks multiplicatively. Therefore, if you have both the Arcane Knowledge mastery (+8% MPen) and a Void Staff (+35% MPen), you'll have 40.2% total magic penetration. This is calculated as follows:
0.402 = 1 - (1 - 0.35)(1 - 0.08)
As of Season 3, the order in which penetration is applied is as follows:
- Flat Magic Resistance Reduction
- Percentage Magic Resistance Reduction
- Percentage Magic Penetration
- Flat Magic Penetration
Abyssal Scepter vs. Void Staff
For the purposes of this answer, I will assume that you're running Flat MPen marks (+0.87 MPen) and that you're taking the Arcane Knowledge mastery. This gives you 8% Percentage MPen and 7.83 Flat MPen.
Solely considering which item will allow you to pass through more Magic Resist:
Abyssal Scepter provides a Unique Aura that reduces the MR of nearby enemies by 20.
If x represents the MR of an enemy, then having Abyssal Scepter will set the enemy's effective MR to
(x - 20) * 0.92 - 7.83.
Void Staff provides +35% Magic Penetration to the wielder.
If x represents the MR of an enemy, then having Void Staff will set the enemy's effective MR to
0.598x - 7.83.
Applying math, we find that when the enemy has at least 58 (or more precisely, 57.14) Magic Resistance, Void Staff is more effective in terms of penetration. When the enemy has 57 Magic Resistance or less, however, Abyssal Scepter is more effective.
It should still be noted that Abyssal Scepter requires that you be very close to the enemy to apply its effect. This can be much riskier for you if you're not built as a tanky AP or if you don't have Zhonya's Hourglass; for this reason, buying Void Staff is usually preferable for AP mids, while Abyssal Scepter is usually preferred on AP junglers and AP tops.