You will probably be vulnerable in open combat against stronger enemies, but this is a perfectly reasonable build. In fact this is my build at the moment (sneaky assassin with daggers and archery) and it works a treat. I am just about able to 1-hit kill a Frost dragon and am level 33 or so.
Increase sneak as much as you can, and early. This will let you more easily a) stay hidden and alive and b) get those delicious sneak attacks we all know and love. Anything to increase sneak/backstab damage is really useful, including, as @Rodrigo Guedes said in his answer, joining the Dark Brotherhood and Thieves Guild (you get some great armour if you follow the latter through to the end).
I haven't got any smithing to speak of (20 or so) and am using Mehrunes' Razor (highly recommended as it can occasionally 1-hit kill, saving you some hassle) and the Blade of Woe, which is useful to claw some health back mid-battle when needed. Remember that when you're lower level and using unenchanted daggers you can still improve them a little bit!
I've also found archery to be incredibly useful. It can do a ton of damage if you level it up and use good arrows, you can start doing damage at enemies that are rushing at you once you've been spotted (the perk that slows down time when you draw is a lifesaver) and you can also increase the sneak attack damage to 3x instead of 2x. That's not as much as 30x but you don't need to be as close! :)
Also, don't forget your Thu'um. Slow Time is great for positioning yourself or running away, Throw Voice is great for getting targets into backstab-friendly positions, Marked For Death is great for extra damage in open combat and finally Aura Whisper is an absolute must - cast it as you walk around in a new dungeon and you will never be taken by surprise! I think I use this more than any other shout.
Happy stabbing!