I want to fail underground undercover (because I joined the institute) but I don't know if it is possible. If it does not affect my relationship with the institute, or if I can fail this quest please tell me!
You don't need to fail it. It won't affect your relationship with the Institute.
If you follow the Institute quest line, you will eventually fail any Railroad quests you have.
Check this question and answer for more information: When do I have to decide between the different factions?