I think I'm kinda stuck and I'd like to ask for help. Some preliminary stuff first:
I've already explored most of the Commonwealth and did most of the quests. I decided to side with the Railroad and they helped me build that teleporter thing. I promised Virgil I'd get him some FEV cure and Desdemona asked me to infiltrate them.
I entered the Institute and met Father. (that revelation got spoiled for me.) Throughout our conversation I was mostly 'unsure', neither accepting, nor denying his proposition to join the Institute. I did accept the invitation to look around though. He mentioned 4 names to speak to, which I did. The first 3 usually asked me if I was gonna join and I, again, said I wasn't sure yet.
Side note: Throughout my quest to find those 4 morons, I lockpicked my way into the FEV lab, in which I killed an Assaultron + turrets. But upon exiting that lab, nobody got hostile though. I also met up with that Patriot guy and agreed to help a solid dozen synths escape. Though again, upon going that far, nobody got hostile.
When I went to speak to the 4th guy, whenever I finish the conversation, I complete the 'Institutionalized' quest, though literally everybody gets hostile and starts shooting at me. I reloaded a few-minutes-earlier-save and exited the Institute, before speaking to the 4th guy. Then it says I've completed 'Institutionalized' as well. I went to speak with Desdemona again and my SS says he got kicked out of the institute, even though in that save file I did no harm. She then says I should ask Preston Garvey for help.
While still in the Institute, the 'Institutionalized' quest says 2 things. 'Enter the elevator', which I completed. Plus exit the institute, which seems to indicate the choice has already been made. Where did things get messed up?