Based on the official FAQ:
For the gold-based auction house:
In the gold-based auction house, a 15% transaction fee will be deducted from the final sale price of a successful auction.
For the real money auction house:
In the real-money auction-house, for equipment such as weapons and armor, a fixed transaction fee will be deducted from the seller for each piece of equipment successfully sold. This fee is assessed only if the item is sold. For commodities such as crafting materials, gems, gold, and other “stackable” items, a 15% transaction fee will be deducted from the total sale price.
The transaction fee for equipment varies by auction house currency (e.g. $1 for USD, £1 for GBP - see the FAQ for full table of details).
If you plan to have the funds transferred to Paypal, you should also be aware:
Note that sending proceeds to PayPal will be subject to an additional 15% transfer fee if the auction succeeds, calculated based on the amount being transferred. Note that additional fees from PayPal may apply.
Finally, note:
Players who wish to use PayPal to charge up their Battle.net Balance or to “cash out” funds earned in the real-money auction house will be required to sign up for our Mobile Services & Alerts SMS authentication service in Battle.net. (...) After this initial verification, you will be prompted to enter a new SMS code from time to time when accessing a feature that uses PayPal. Note that standard SMS messaging rates apply -- check with your cell phone carrier for details.