You can check the modifiers by hovering over the red x (or the green checkmark) next to the demand vassalization text. It's a detailed breakdown of which factors speak for and against accepting the demand. If the total number is positive, you'll see a green checkmark and they'll accept the demand, otherwise they won't.
Here are the most important factors:
Ethics. Similar ethics provide boosts to this, very important, some ethics like fanatical xenophobe will prevent a peaceful vassalization entirely. You cannot influence this.
Empire strength, being stronger, i.e. having the bigger fleet and bigger empire provides significant bonuses
Opinion, the biggest factor is opinion there are several ways to increase it: mutual rivals, embassy, treaties (border access, non-aggression, guaranteeing, research agreement), making the empire like you is the most important aspect of peaceful vassalization. Keep in mind, you CANNOT engage into an alliance with them, as an alliance locks you in as equal partners and disables vassalization.
It is generally pretty hard to get a purely peaceful vassalization, especially if your ethics diverge somewhat.
There is a form of military/peaceful vassalization though, that goes beyond forcing an enemy to submit.
If in a war, you choose to liberate enemy planets instead of conquering them, this will create a new empire with your government ethics, on top of that, you get a massive +100 opinion for being their liberator. This usually makes peaceful vassalization of the liberated empire very easy.