Regrettably, you have likely encountered one of the many glitches in Fallout 4 - this one being more prohibitive than most. Referencing the wikia article for Mass Fusion there is a glitch where you are unable to relay into Mass Fusion when you are enemies with the Brotherhood of Steel - which you are.
If you betray the Brotherhood of Steel and are their enemy prior to beginning this quest, then you may not be able to relay out of the Institute. Fast traveling to Mass Fusion does not fix this, and you must load a save from before you were enemies with the BoS.[verified]
As the wikia article states, the only solution to this is to load a save game before becoming hostile to the Brotherhood. As lamentable as it is, this a strong lesson in keeping multiple save states in Bethesda games.