In the diplomacy screen of Stellaris, an opinion score is shown, that shows how much the other empire likes you. One component of this score is "Threat". Some of my neighbours have huge negative threat values, some don't. Yet I never had a war with any of them.
That's a measure of how much they perceive you as a threat. Threat is generated by taking vassals or conquering planets from any empire.
The closer you and the conquerered are to an empire the more threat it generates. Threat is slowly reduced over time and there is no other way to eliminate it.
You can work around threat by liberating enemy planets and peacefully vassalizing them or simply not conquering.
Think of threat as a warmonger reputation and that an empire believes you will eventually come to conquer them as you did before with others.