There is no point in not attempting a dungeon when you first reach it. While some dungeons may be more difficult than others, it is a matter of skill (primarily twitch reactions), rather than actual level.
Story Mode should be attempted at the level it is given, and possibly even a few levels below depending on the party.
Exploration Mode should be attempted 5 levels after story mode, and is typically more difficult.
Dungeons offer a variety of rewards, including a great deal of experience and equipment. Ascalonian Catacombs, or AC, takes up to two whole hours for newer players. As a result, it's fairly frustrating and the rewards are fairly low-key. If you enjoy a challenge, or you play with a group of people, there's no reason to skip it. I found this to be the most enjoyable dungeon of them all.
Dungeons such as Caudecus Manor and Citadel of Fire, a level 70 dungeon, can be farmed on exploratory mode easily for Seals and experience/money. These runs typically take less than half an hour, depending on the chosen options, and the seals can be traded in for armor (level 70 rares, or 80 exotics).
TL:DR - Dungeons aren't designed to be impossible, and most are extremely easy.