The AH deposit seems to be proportional to the duration you have an item up on the block for, and apparently 15/30/60% of the vendor cost of the item(?), but how does the AH fee work when an item sells? Is it a fixed portion of the final selling price? Is it affected by the duration?
If I remember correctly, the deposit gets returned to you when something sells, but the auction house itself takes 5% out of the winning bid as it's fee. I do know that it's a fixed portion, and not affected by duration.
The Neutral Auction house takes a larger cut -- 15%. Filthy goblins.
As a reminder, you don't get your deposit back if your item doesn't sell. The cost of the deposit itself varies with duration (as you noticed) but the vendor price of the item in question as well.
Taken from WowWiki
Again, the goblin-run Auction Houses charge more: