It's possible, but there are no guarantees for success. Here's a few things you can do:
- Assassinate him. The downside is that you gain the
Kinslayer trait if someone finds out (-3 to diplomacy) and it might take you a few attempts.
- If your king is old, frail and likely to die soon, you could still
imprison and execute your son, even if you have no moral right to do
so. You will get severe penalties to the opinion of vassals and lose
a bunch of piety, but you will be certain it will work. Can be used
in emergencies.
- Make him lead an army of only a handful of men, preferably not a very skilled troupe. Throw him against far superiour forces, and hope he gets himself killed. If he gets imprisoned, you could either refuse to ransom him out (in my experience, prisoners often die in custody) or bail him out and send him on more suicide missions.
- Give your preferred son a whole load of powerful duchies. If everything goes well, this will start a succession war once you die. You will then play as the imbecile. Make sure you lose.
You can also do the third option if he actually does inherit. If he dies as a ruler, his eldest brother will probably inherit, so long as your succession laws allow it and he hasn't fathered any child of his own.
None of these options are sure-fire, of course. The sooner you start working on the untimely death of your heir, the better.
EDIT: I was once in a similar situation. Even if you could imprison and even banish your son, he would still be your heir. You could execute him, but I've found it to be an unpreferable choice due to the penalties (-10 opinion from all vassals, -20 piety).