The propeller starts either by jumping and holding the jump button (A or B) or pressing the jump button for the second time after jumping and holding it, so the ideal height will be achieved either by using the first method or by using the second at the top of your jump, either way, once you start ascending, you'll need to hold the button as long as possible to maximize your height.
Once you start the propeller and reach the maximum height or release the jump button, it cannot be used again for ascending until you hit the ground, you can use it to slow down you descent, though.
An exception to this rule is when you land on an enemy and hold the jump button as you do so, you'll get a small boost and you can once again use the propeller. If you hold the jump button this will happen automatically.
Also worth noting is that any way to jump higher otherwise can be linked to the propeller jump, for example, jumping from an orange trampoline/note block, a jump from a crouch position (hold L or R button until Mario grunts) or wall jumping and holding the jump button after the last jump.
As a side note, if you have a propeller hat and you have a wall nearby, you can ascend the same wall by constantly wall jumping and floating back towards it. Using this method this can help you save yourself if you accidentally drop off a pit.