A bit late, but I'll add more info. My source is this wikia page which in turn references Final Fantasy IX Ultimania. To quote:
Vivi is a black mage created in Dali at the behest of the kingdom of Alexandria after Queen Brahne learns of this technology from Kuja. Vivi is a prototype black mage who fell out of the cargo ship into a forest near the town of Treno. A month later he was discovered by a Qu named Quan who raised Vivi out of the hope he would grow big enough for him to eat, but after six months Vivi was still as short as before.
Vivi was oblivious to Quan's intentions and lived happily with his "grandpa" in Quan's Dwelling. During this time Quan named Vivi, and taught him many things about the world, sparking a curiosity in him to see it for himself. When Quan died Vivi was left alone. A week later he headed to the nearby town of Treno and got a ticket for a play, and the next day he hitched a ride on a nobleman's airship to Alexandria.
At this point the game starts and you already know that part.
As for the other black mages in the village - it appears that after a while the enchantment on the black mages started to wear off, or was broken. This resulted in their "awakening" and subsequent formation of their own community. For example, the story of No. 288:
No. 288 was once just another black mage soldier in service to Alexandria. On the day he gained sentience, he described it was suddenly awakening from a daze, and was horrified by the sight of a blood-covered soldier laying before him. He fled the scene and found others like him on the scene, and they wandered Gaia until they heard of the Black Mage Village and made their way there.