Daggers are indeed faster, although I can't find a reference on how much faster. Everything (enchantments, perks, etc.) considered, I don't think there is a huge difference, so it should come down more to style.
Backstab: 6x sneak attack damage with a one-handed weapon
Bladesman: 10/15/20% chance of doing critical damage with a sword (3 ranks)
vs the dagger's
Assassin's Blade: 15x sneak attack damage with a dagger
and both have
Armsman: +20/40/60/80/100% damage with a one-handed weapon
So there's a difference here, in that the perks favor different playing styles: daggers benefit from stealth and swords benefit from more all-purpose combat. If you're dual-wielding a dagger and sword, that shouldn't really matter as you can sneak attack with the dagger and follow up with both.
As you get better weapons, the damage difference as a percentage falls. For example, an Iron Sword deals 7 base damage vs an Iron Dagger's 4, which is 75% stronger. A Daedric Sword deals 14 vs the Dagger's 11, so only 27% more. Smithed to legendary, it's 24 versus 21, so only 14% difference. I don't know how much the crits from Bladesman add.
Also, because daggers are faster, they will be able to deal out on-hit damage from enchantments more quickly. If you're focusing at all on enchanting, you may want to consider that.
An actual sneak attack
In my experience, a standing dual-wielded sneak power attack with two daggers will connect and get the sneak bonus on the first two and maybe more (most enemies are dead by then). I can't tell whether that's up to 60x normal damage (15x from Assassin's Blade, 2x from Shrouded Gloves, 2x from power attack) or if there's some cap, but suffice it to say that if you planning to be very sneaky, not much will hold up to such an attack with two Daedric Daggers.
On the other hand, daggers do not benefit from several bonuses to swords. That includes Fortify One-Handed, Bladesman, etc. That includes silly things like the Nightingale Gloves.
If you don't plan to be seen often, I personally have stuck with daggers. My strategy for dealing with non-sneaking situations is falling back on magic. If you want more damage in face-to-face combat, of course, consider a sword, just keep in mind that the ideal equipment for sneaking is quite different than that for combat.