Just wondering if when Nocturne casts fear, and you whimsy him as LuLu will it stop the cast or will it continue? (provided he stays in range)
Yes, as long as you are still in range, Nocturne will indeed still fear you regardless on the fact that he is a harmless squirrel. But if you time Whimsey correctly (it takes 2 seconds for the fear to come into effect), you can cast Whimsey right as you become feared so he can't attack you while you are feared.
Using Whimsey to polymorph the enemy into something harmless is a new functionality that no other champions have. Because of this, there is some gray areas of how it will affect certain skills. Nocturne's fear is not a channeled skill, as you can tell because he is able to move. Champions are not able to move while they are channeling. It can disrupt real channeled skills like Katarina's Death Lotus and Fiddlestick's Drain.
There are some situations in which if Whimsey is cast at the last possible moment on champions with teleporting type skills (like Pantheon's Grand Skyfall), it would not disrupt it but instead teleport the enemy champion in the polymorphed form. This can be quite humorous when suddenly a cupcake drops from the sky to waddle around you.