He does explain this - in the short story titled "The Last Wish".
In short, it's because sometimes portals misfire, and people indeed come out of them in multiple pieces. Or don't come out at all. Or come out at places completely different from their intended destination.
The relevant quote is as follows:
On the damp bath-chamber wall shone the luminous outline of a door which framed a swirling phosphorescent milky nothingness. The witcher cursed. He didn’t like magical portals, or travelling by them.
‘Do we have to . . .’ He cleared his throat. ‘It’s not far—’
‘[. . .] Don’t worry, my portals are safe.’
Geralt had once watched as only half a traveller using a safe portal flew through. The other half was never found. He knew of several cases where people had entered a portal and never been seen again.
The Last Wish, Gollancz edition, translation by Danusia Stok
There is another relevant passage from Season of Storms, the latest published book:
As to the journey, he was not sure what had happened, how and why the portal opened by Degerlund had thrown him here - in a swampy wasteland. He doubted that this was the effect of purposeful wizard's action. A simple teleport malfunction was more probable, something that he was afraid would happen for over a week now. That which he heard of many times, and even witnessed a few times - when portal instead of transporting it's user where it was supposed to go, threw them into a completely different place, an absolutely random one.
[. . .]
The worst portal malfunction, that had forever discouraged him from teleportation, Geralt had witnessed at the beginning of his witcher career. Among upstarts, rich nobles, and golden youth it was then fashionable to teleport from one place to another, and some wizards provided such entertainment for an enormous sum of gold. One day – the witcher had been at the place – a fan of teleportation had showed up from a portal cut precisely in half vertically. He looked like an open double bass case. And then everything had fallen and flowed out of him. The fashion for teleports noticeably dropped after this accident.
Season of Storms, chapter 13, the fan translation (an official English translation by David French is due 19 April 2018)
This has since become a running gag, which was popularised most of all by the third game (because there aren't that many portals in the previous games).
All gifs from this Tumblr post, even more gifs in this post.