Has Nanaki/Red-XIII's species ever been formally named?
Nope, Nanaki's race has not been formally named or designated as anything other than a 4-legged beasts that resemble wolves.
Taken from http://finalfantasy.wikia.com/wiki/Red_XIII :
Red XIII was originally designed as a beast-type SOLDIER capable of using magic and understanding human language. He would have been a protector of the Buga tribe, to which Bugenhagen belonged, and his father Seto would have been his predecessor before having fled, making Red XIII contemptuous of him. In the final product, Red XIII's tribe is unnamed, but otherwise he is roughly the same as originally conceived. When he was first drawn, Nomura chose the name "Red XIII" as he wanted a name that did not sound like a name, and so combined a color with the number "13", which is believed to be unlucky. Other staff members later assigned Red XIII's true name as "Nanaki". During early plans for the game, Red XIII was to have the "Beast" job class.