It's definitely possible, but don't just put your entire game folder under version control. Some advice:
Git is not optimized for storing non-text files. It cannot calculate diffs for them and has to store full copies instead. So your repo will continue to grow significantly if you for example try to save bunch of textures there. Imagine having multiple copies of the same game on disk - effect will be like this.
Sometimes games modify some files during run - some logs, configurations, saves. Don't version control them, otherwise you would have to commit these changes after each session.
As an alternative you can store you mod folders separataly from the game itself and provide symbolic link from game folder to actual mod folder. Switching mods will be as easy as updating one link.