The Spy must contact the Double Agent by saying "banana bread" to them, but the Sniper can hear when this happens. Is there any penalty to saying "banana bread" to everyone at the party, or even when the Spy is all alone?
Not mechanically, no, but it's risky.
Possible penalty: If the Sniper hears "Banana Bread", he can immediately lowlight everyone not in a conversation at the time. This is offset greatly if pretty much everyone in the room is already in a conversation, but if there are only 3-4 people in conversations you've suddenly trimmed down the Sniper's suspects tremendously.
Possible bonus: If you say "Banana Bread" while the Double Agent (DA) or Suspected Double Agent (SDA) is in a different conversation, the Sniper may highlight everyone in the conversation with the DA or SDA, thinking you contacted one of them. However, you don't know who the SDA is, though, so you risk this bonus not being helpful at all, as the SDA might be standing right next to you. Also, the Sniper could be on to your sneakiness and not fall for it.
I prefer not to say Banana Bread too early because that penalty can be pretty bad, but as with everything in SpyParty that's not a hard rule.