A wizard is different from a demon hunter. I have no idea about demon hunters on this topic. IAS does not increase the damage number you see above the heads of your targets. In fact, it lowers unless it is supplemented with the same amount of actual damage "like 25-50 on a ring" but you will have faster attacks. The one thing I tested though is to see the actual stat weights of the two and I found that by dropping all of my IAS for "damage" my numbers were actually bigger by a much bigger margin than how fast my attacks were if i were to leave my IAS on.
For example I swapped out a 1 hand weapon and a source for a 2 hand hammer, also a ring, the hammer has 1107 damage on it, the 1 hand weapon has 700ish and a source with about 300 damage, this should be about the same damage in theory but because the 1 hand is faster it actually increases the "damage" stat by about 1.5k, that's just the weapons. Now the rings I swapped out were 8-16 damage, 14% IAS for 25-50 damage and 45% crit damage.
The ring with IAS is according to the damage stat a 1.3k damage increase, so altogether it's almost 3k damage gain with IAS according to the damage stat. However my attack speed when I take off all the IAS gear and go for straight damage stats only drops by .5 and I lose about 3k damage as a stat, but when I attack I'm seeing my hits go from 40k-80k hits vs the 20k-40k hits with IAS. Now granted they do hit slightly faster, but I just don't feel that the damage is all that comparable. Hope this helps, sorry for the non exact numbers but I saw your post during maintenance so I can't quote directly from the game.