You have a few options. Basically: it's not too terribly difficult to identify mods that might conflict. You can then either remove conflicting mods or learn how to make your own compatibility patch.
Use someone else's mod list
As you mentioned, you could run a list of mods provided by someone else, in the order they specify, using their patches. So you would follow the STEP guide and never deviate from it. This should give you a stable game, if you're happy with their choice of mods.
Hope for the best
Try to choose mods that you think will work well together. Avoid mods that look like they edit the same thing, and avoid running more than one overhaul mod at a time unless you can find a compatibility patch or a statement that they're compatible without one. Start up your game and look for problems. Personally, if I make it to level 30 without any serious problems, I assume that my game is stable.
Check for compatibility yourself
File compatibility
Use Mod Organizer and set it to unpack all BSAs. Mod Organizer will show you which mods overwrite the files of other mods (if their BSAs are unpacked). It isn't usually a problem if a mod overwrites USLEEP, but for anything else, you might want to look at it. Sometimes overwriting mods is harmless (e.g. overwriting a couple textures), and sometimes it indicates an incompatibility. If mods are overwriting each other's script files and you don't expect them to, you might run into problems.
ESP compatibility
Run LOOT and make a bashed patch with Wrye Bash. Next, start up TES5Edit and click OK with the default selection. Wait for it to finish and then browse your entire mod list.
TES5Edit will highlight records in each file that conflict with other files. If you see tons of conflicts in a particular mod, it may not be compatible with your other mods. Some conflicts are not a problem: practically everything edits Worldspace 3C (Skyrim), for example. If you have a compatibility patch for specific mods, expect the patch to overwrite/conflict with both of the mods it's patching for.
If you see a lot of conflicts, you can either remove mods until TES5Edit doesn't highlight so many records in red, or you can trek down the road of becoming a Skyrim modder and use TES5Edit to create a compatibility patch yourself.
Script compatibility
Honestly... other than checking for overwritten script files with Mod Organizer, I don't know how you'd check for script conflicts. I just hope that nothing bad happens. In general, just try to find well-written mods and don't use too many script-heavy mods.