I know that PS2 games can only be played in a 'fat' PS3 with the right hardware to run it. But why is so difficult to emulate a PS2 game in the PS3 without that hardware? Is the processor, the architecture, both? Thanks in advance.
PS2 games were made for an older CPU. They cannot run on the CPU designed for the PS3.
PS2 games were created to work on Sony's Emotion Engine CPU and their Graphics Synthesizer GPU. This CPU/GPU combo was put in all PS2 units and in the first variants of the PS3 (the NTSC 20GB and 60GB models) to make them backward-compatible. These models can play almost all PS2 games.
The 80GB Metal Gear Solid IV bundle had the GPU as well, but replaced the Emotion Engine chip with software emulation of the chip. It can still play most PS2 games, but support isn't as good as the earlier models with the actual EE CPU.
Models since the MSG:IV bundle had neither the CPU nor the software emulation of the CPU required to play older games.
Sony has a knowledgebase article that explains which three versions can play PS2 games. They also have a compatibility tool that you can use to see if specific games can play on those specific models.
PS1, being a system simpler and older than the PS2, is easier to emulate using only software. In 2000, Sony bought a Playstation emulator called Virtual Game Station. While the first PS2 models incorporated hardware of the PS1 (like its CPU), later PS2 models completely emulated the PS1 via software (and were thinner).
In addition to Shaun's very good answer, consider the cost of emulation.
First, the PS3's with the extra hardware for running PS2 games. Basically Sony was having to pay for PS2 and PS3 hardware in manufacturing, and then selling it at the PS3's price. This was probably a losing investment for them, since the PS2 part wasn't the main attraction; it probably didn't increase PS3 sales very much. And if it had, it wouldn't have increased PS3 game sales very much -- and Sony wouldn't make any more money than they already had on the PS2 games people owned.
Second, the software emulation. Emulation is hard, especially for a sophisticated console like the PS2. As Shaun said, the reason it's hard is because the PS2 had different processors than the PS3, meaning everything had to be "translated" from PS2 instructions to PS3 instructions. While certainly more deterministic than human-language translation, this is still a hard problem to solve; especially when you only have the limited resources of a console (emulation often requires about 5x better hardware than the system being emulated did). Not to mention that the bus system (how the chips are laid out and connected) is very different between the two systems. Sony probably had to start up an entire team devoted to emulating the PS2 in software, and again it probably wasn't worth the return on investment so they shut it down.
Older gamers may recall that the PS2 could run original Playstation games seamlessly (well, mostly), and be asking themselves why the considerations listed herein didn't apply to then.
Sony essentially used a original Playstation (with it's several hardware quirks) built onto a single chip as the audio-processor for the PS2, and added bus-mastering feature that allowed the sound chip to take over the whole machine when a original game was run.
This worked out without adding undue cost because the power of the original machine matched new machine's sound processing needs.
I agree with what everyone else has said about software emulation, but I'll add that writing a game console emulator is very different from other emulation undertakings.
The PlayStation 2 was Sony's main console for well over 6 years. During that time the hardware (from the point of view of the developer) changed very little. This means that game developers had plenty of time and tons of incentive to figure out every nook and cranny of the device and take advantage of it to deliver maximum performance for their games.
Ordinarily when you write emulation software you want broad compatibility, but the precision of the emulation isn't as critical. No one cares if Wine has a glitch that causes MS Word to be slow at reformatting pages. By comparison, a PS 2 emulator has to nail nearly every aspect of the underlying system, in case some game programmer is taking advantage of some quirk.
Also, video games are more often skirting the performance limits of their systems than other software is, because video game hardware is so standardized. This means that, again, any particular weakness in the emulation will be obvious.
Another difficulty with video game emulation is testing. With the hardware emulator, Sony just needed to use the same hardware specification to ensure compatibility. To achieve the same in software isn't so simple, and testing it would be a nightmare. Testing MS Word within an emulator, by comparison, is simple: make a list of the features and commands, and walk through them in the emulator. With games, there is no way to list the full feature set beyond the basics, and walking through the features requires playing the game, which is designed to take time and skill.
All of this adds up to a much more difficult job than writing a typical emulator.
All emulators are difficult to write, even ones for older, simpler consoles. It's not really a case of difficulty - it's whether the PS3 is fast enough to emulate the PS2 at the required speed.
I don't believe it's impossible to do. The PS3 should be powerful enough to emulate the PS2 adequately. Not an easy programming task, but Sony should be able to manage it - they designed both consoles after all, and have demonstrated ability with their PS1 and PSP emulator (for minis on PS3).
Also, Sony has already done most of the work to emulate the PS2. The initial US 80Gb PS3 models (plus the early European 60Gb models) came with full emulation of the PS2's CPU in software.
All Sony would need to do is write an emulator for the GPU, which was present in the early PS3s, but removed from later ones. Sony removed the GPU to cut costs, which also allowed them to focus development and testing elsewhere. Sony were under a lot of pressure at that time to reduce the price of the PS3.
Whatever discussion Sony had internally to remove backwards-compatibility, I suspect their plan to create the new "HD collection" series also came about at the same time. (The "HD collection" contains (mostly) PS2 games re-made with up-res'd graphics.) This is Sony's new "backwards-compatibility" focus.
EDIT: There seem to be a few different interpretations of this question:
Most posts here have already answered #1 - the PS2 and PS3 CPUs are different, and incompatible. I'm answering #2.
The creators of PCSX2, a PS2 emulator for PC, have a blog. The blog contains many interesting articles about the difficulties of emulating the PS2. Of course, they are writing for the PC, not the PS3, but the same difficulties would apply.
Different hardware/architecture. Imagine trying to teach someone spanish by talking to them purely in german.
Writing an Emulator is hard but I think the real answer here is simply that Sony does not want to. Any computer that is Turing complete can be emulated on another given enough memory,speed, and processing power.
There are many talented programmers who have written emulators for various systems - look at MAME, UAE, Bleem! and many others. In the case of Bleem! (the PS1 emulator) SONY tried to sue the life out of the creators. They ultimately failed, but Bleem! went out of business. Sony are still pretty quick to lawyer up or even send the cops around, like what happenned to Graf-Chokolo.
Sony would much rather sell PS2 games to you again as downloadable games.
So, I do not think that PS2 emulation would be any more difficult on the PS3 or a PC than any of the other more challenging emulators.
Truth be told they can easily emulate PS1 and PS2 games on PS3 but they don't want to due to the fact that not a lot of people were buying newer games made for PS3 (a lot more expensive etc) and because of that game makers were also sticking to making PS2 games for a few more years.
This was noted in an article few years ago after the introduction of the PS3. But when it comes to emulating PS2 and PS3 on PC, it is a completely different ball game as both PS2 and PS3 are more graphic intensive systems which most PCs don't have graphic processors that high yet.