Bane's nightmare skill can work on both allies and enemies, and is transferred when the sleeping target is attacked, but what are the conditions for doing so? Does the attack have to be a targeted attack, or can any aoe effect cause the transfer (such as ancient apparition's ult)? Also, is it possible to deny allies with this spell, will they take damage while sleeping?
Yes, Bane Nightmare's skill can be used to deny allies.
Nightmare can damage allies (and Bane), and can thus be used to deny hero kills.
Regarding the damage required to wake up the target can be any type: AoE, target skill or auto-attack, even Radiance can do it. This is the reason why playing Bane within low pub teams can be difficult sometimes.
Transferring the Nightmare effect to the attacker only works with auto-attack instead.