If you break it down, the method of how the passive is working is based off of the physics formula of velocity + speed in a single direction. If you change direction even slightly, some of the velocity is lost. You completely change direction, the velocity in that direction is completely gone. Check out this link for a better explained way.
Your bonus movement speed is lost when you make a sharp turn; 45 degrees, 90 degrees etc. The best way to use the passive of Aurelion's travel is to always go directly straight. Even small turns can cause you to lose a bit of movement speed.
At best, you should practice with trying to hold onto as much escape velocity as possible while maintaining turns in the game to get a better angle at viewing and maintaining it. You might even get an edge on how to maintain it during skirmishes and team-fights, and that knowledge would be very advantageous for you and your team.