After consulting the Mario Kart Wiki, it seems that by default, Baby Mario's Traction and Handling are actually very high. The smaller characters generally have the best steering abilities, but the slowest top speeds - I'm guessing that when you say they're the fastest, you mean you're able to set faster lap times with them. This, in my experience, is true in the 150cc and 200cc classes, where handling ability becomes critical. (Time Trial Mode is 150cc, if I recall correctly.)
I suspect that your choice of kart and/or wheels is hampering Baby Mario's Handling and/or Traction stats, perhaps in pursuit of some other stat like Top Speed. Speed is nothing without control! So before you try other characters, I would suggest browsing the parts list and trying to build a kart with better Handling and Traction, and see if that makes Baby Mario more easy to control.