why did the gamepad take over from the joystick?
Many reasons!
My tuppenceworth: I guess once the NES pad came out, people making gaming hardware just copied it.
Firstly, from a business point of view, it has less plastic in it than an old style 8-way joystick, and offers the same precision of movement. Gamepads are significantly more compact and less bulky to store than old school joysticks too. Cheaper to make, smaller, better for business.
Personally, I think the main reason is one of optimal interaction within the constraints of the product - you don't need to rest it on a table, it takes up next to no space, and it can be more or less covered in controls as long as the user can reach and use them all comfortably.
IMHO gamepads represent an evolution of the old school joystick, (where the fitness function includes cost, ergonomics, and functionality available at your fingertips) and so they've naturally taken over from joysticks.
Sure it's less buttons than you'd need for a PC flight sim (every button on the keyboard anyone?), but you just don't need that many buttons for the vast majority of games.
TBH, the thumbsticks on an X360 pad offer the same angular resolution as any typical flight stick; so the only real reason for the massive flight stick still existing is that players want to have a joystick that makes the experience feel as close to how they imagine the real thing as possible - certainly that's why I bought one of those massive flight yoke things with the throttle controls and about a billion hat-switches to play Apache Gunship in 2001 :)