The first computers were monochrome; they only knew 2 colors, as for each pixel, you only had 1 bit:
0
1
As for the Gameboy, it supported 2 bits per pixel, which gives us 4 possible colors to choose from:
00
01
10
11
How did the makers decide on four and why not three
Because they already had 4 colors to choose from. When people have 10 fingers, why not count up to 10?
or five?
Because that would've required they increase the screen's bit depth to 3 bits per pixel, which would have given us 8 colors to choose from. Why didn't they do it? Probably for performance reasons. As you increase the amount of bits per pixel, you increase the amount of data to process, which in turn requires more powerful hardware to process.
Was there a precedent to using four shades of grey in computing graphics before the Gameboy, or was this an invention of the system?
Of course there were precedents.
CGA, gray-scale early NeXTstation, color Macintoshes, Atari ST.