From careful experimentation (i.e. randomly shooting bugs) I've discovered that the aliens become hostile if you attack them. However, I can't quite tell if this is per-unit (i.e. that unit becomes hostile), per-nest (all units from the nest) or global (all aliens) become hostile. They certainly don't all attack at once the same as if I declared war on another civilization. I've also seen them attack first if I stray too near their nest.
I also noticed an upgrade (can't find it now) that claims to cause their hostility to go back to normal twice as fast, which suggests they have some internal aggression counter that gets increased when I attack and decreases naturally at some rate?
So, does anyone have a good summary of how alien hostility works and what I should watch out for?