I do not think that the limitation was due to hardware, although I do not know which game was the first to feature high-scores with three letters. Sea Wolf (1976) seems to be the first game to feature high-scores, but not to save its initials.
The modern concept of high-score was introduced by Space Invaders Part II, which finally allowed players to save a name with their high-score, before this the concept of high-score was used to give extra playing time after you reach a certain score.
I emailed Taito support, the creators of Space Invaders Part II, about a limitation at this time and here is their reply:
Thank you for your inquiry.
The game supported name entry of up to 10 capital letters for high scores, not three.
Incidentally, in its factory state, the initial high score was set to 5000 points, with the high score name as “TAITO.”
We hope this answers your question.
Global Support Team
So we can safely assume that Space Invaders Part II was not the game to introduce three characters in the high scores and if there was a limitation in three characters being saved it was after Space Invaders Part II although this seems more unlikely.
Wikipedia credits also Star Fire from Exidy to have allowed people to enter their initials, I could not find someone to contact who worked at the now closed down Exidy at this time to ask the question.
Also, contrarily to what some people think here, the scores were not always stored permanently and kept when the machine was reboot, see this upgrade for DK to remember the top 5 scores.