I thought "retire" was "go back to the mothership and repair" but it seems it does something else because I can't find the retired ships anywhere :/
Retiring ships commands them to be deconstructed and credit you with some RUs. It's like recycling them, so the resources can be used to create something else. This is exactly the same behavior as in the original Homeworld.
From the Homeworld one manual:
Bring up the right click menu and select the Retire command. Ships given the Retire order will return to the nearest Mothership or Carrier and be scrapped. A percentage of the construction cost (in RUs) will be recovered. OR: Press the [I] key once you have selected the ship(s) you wish to retire.
From the Homeworld 2 manual:
This is not 100% of the construction cost. I understand it's approximately 80% of the construction cost.