In my first playthrough, I was able to get out of the ship without setting off the alarm. In my second playthrough, despite knocking out every guard and never triggering an alarm, when I passed the security checkpoint I was notified that the base was put on an heightened alert state due to some alarms from the ship. What did I do differently? The only thing I remember is a turret seeing me for a few seconds.
Being spotted by a turret will immediately alert it, but as long as it doesn't shoot at you and alert guards, who then trigger alarm panels, it doesn't affect whether or not you get the Smooth Operator bonus.
Your problem, however, might be that some guard that you missed was patrolling and stumbled upon one of your victims. Depending on where you are in the game, you can also trigger guards or robots to spawn, and if one of these are in the vicinity of your handiwork, they can be alerted. A camera can also pick up dead or knocked out bodies, and if it goes past "Suspicious", it will cause an alarm.
How do I avoid triggering the base alarm?
The answer to this question is simply: avoid being spotted by guards or cameras and prevent them from seeing dead or knocked out bodies.
What did I do differently?
This question calls for speculation and can't really be answered with the information you have given.