I've never heard of this problem, nor can I find information regarding this specific issue. However, after reading through the PS3 hardware specifications, I'd be almost certain that you have a bad RSX 'Reality Synthesizer' GPU.
The RSX chip was co-developed developed by Sony and NVidia and is proprietary to the PS3. The only way that you can get your hands on another one is to canibalize another PS3 for the component. Even that isn't realistic though. The pins are connected from the bottom of the chip, onto the system PCB. In other words, you're not going to take that component off of the board in one piece and, if you did/could, it'd be nearly impossible to get it back on another board without some top end manufacturing equipment.
This particular GPU is responsible for handling some of the floating point mathematics and shader operations for the system. I'm not exceptionally well versed on this type of hardware but if I had to venture a guess, I'd say you are having problems with the parallel vertex pipelines. I would assume that this could mean that the chip still operates but with odd results, as you are seeing.
Considering the above information, it looks like your best option is to find a replacement system on a resell site if you really want to keep an older model PS3, or upgrade to a newer model.