I just found the first cavern. However, I've read that all sorts of Fun monsters live in there. What features do caverns have that make it worth braving these critters?
Caverns are great resource areas. In a typical first level cavern you'll find fungi for food, tower caps for logs, spider silk, water, and large open areas that are muddied already and make excellent dwarven farms. My brother has taken to actually building his dwarf fort in the first cavern because all the resources you need are basically right there.
The first cavern is pretty safe, I haven't seen anything nasty come out of it. The second cavern can get a bit nasty. I have an army of 10 wrestlers I sent down there and lost 1 while they were clearing out the first wave of critters. But note that the second cavern also contains lava.
I haven't breached the third cavern yet. but you can imagine the trend. The deeper you go, the scarier it becomes.
In short, the answer to your question is resources. caverns are a large economic boost to your fort. The critters are a small nuisance in comparison to the value you get out of the caverns.
Underground water supply, large fungi that you can chop down and use as wood, possibly spider silk. You will almost certainly want to have a significant army if you plan on breaching into the cavern and not walling it off!
To add a bit about the safety of caverns: it is possible, though not guaranteed, to have hostile creatures/civilizations in the first cavern level. And just because they're not around when you first breach doesn't mean they won't show up later!