These aren't exact numbers but let's say malphite reaches 800 armor and has 85% damage reduction from physical damage. Getting a last whisper reduces his armor to 800 - 35% = 520 armor. That is still around 75+ % damage reduction.
First we should correct this math, just for the sake of completeness. A character with 800 armor takes .11 damage for each 1 physical damage they would normally be dealt, which is 89% damage reduction. So to start with, if you have 100 AD you'll instead hit for 11 damage per shot.
So in the above example, what is the effect of a Bloodthirster versus a Last Whisper? Bloodthirster adds 80 AD, so that brings us to 180. How much difference does this make? 11% of 180 is about 20, so you spent 3500 gold to increase your damage by 9.
Last Whisper will reduce the target's armor against your physical damage to 520, as you said. To start with, this means that instead of taking 11% of damage dealt, he'll be taking about 16%. It will also increase your 100 AD character to 140, which means you'll be dealing about 22 damage per shot, or 11 more damage. Not a huge increase over Bloodthirster, but 800 is an absurd amount of armor.
Foregoing physical damage altogether with Wit's End is a reasonable option, especially since 800 armor means your opponent probably didn't build anything but Thornmails and Frozen Hearts. Increasing your damage by up to 42 ain't bad, and the defensive stats aren't awful, but consider that the rest of the enemy team likely isn't also building all armor, and will still get chunked by physical damage. Most AD carries don't deal a substantial amount of magic damage, so the MR shred isn't great, and now your abilities are less powerful against people whose builds aren't completely stupid and designed to target you specifically. Not to mention that if your team picked reasonably at all, you should have a good mix of damage, and poor Malphite with all armor and no health or MR will get absolutely destroyed by someone else while you go to town on the rest of their team.
Edit adding info from comments:
This Reddit thread details the difference in damage value between adding either Last Whisper or Blade of the Ruined King at varying AD, armor, and health levels. It found that LW will mainly be more effective than BotRK if (a) you build crit, (b) you have physical damage abilities, and/or (c) your opponent is building lots of armor, but not much health.