I am currently level 10 and spending quite a bit of time finding Gwent cards. I think I've almost found all the available ones for Velen and have started to do more of the available quests - but I'm interested in whether it's feasible to go to Skellige and pay the (somewhat steep) price to get there, even though I'm under levelled and the quest still shows up with a skull on it.
I would not attempt to fight anything or complete quests there, purely try and find more Gwent cards and players.
If I am going to meet a painful end trying to do this, it doesn't seem worthwhile to attempt now - I could easily save the game and give it a try, but decided I'd tap into the knowledge of the stackexchange folk regardless - I don't even know for sure that more Gwent cards lay in wait there! I assume they do.