I'm not a hardcore player, so I'll offer my thoughts, but bear in mind that there may be serious theorycrafters out there who have opinions backed by a lot more math.
As far as I'm concerned, it's a trade-off, so there is no right or wrong answer, just advantages and disadvantages to consider. Your starting location will also have an impact on this decision. As a general rule, with an immediate Settler, you're giving up some immediate advantages for future advantages.
- Quick second city. Several turns down the road, your growth will start exceeding what it would have been without the Settler. Initially though, you'll be at a slight disadvantage.
- Placement of second city may not be great, since you'll only have scouted a small area.
- Allows you to diversify production faster.
Immediate Military Unit:
- Allows you to scout a broader area faster.
- Potential to find villages, which can be very nice in the early game.
- Better defense if you're unlucky enough to run into danger early on.
- Slows growth more than Settler or Worker.
- Quickly grows first city, putting you in a good position early.
- Could get you special resources very early, if you had a lucky spawn location.
- Tends to slow future growth.
- All your eggs are in one basket.
Personally, I think expansion is key in the early game. You're less likely to run into serious danger or catastrophic problems, so it makes sense to take some risks. The more you expand early, the better your position will be 20 or 30 turns down the road.
However, I do think it can be very worthwhile to start with Workers if you have useful resources nearby. If you can set up one or two huge food tiles, the massive growth of that first city can make up for the slower expansion.