On hearsay I've heard it's maximum of 3 on first playthrough and 6 on NG+ but that's not really a credible source. I'm curious if it is unlimited as others say.
There is probably no limit.
On my first playthrough (female Byleth, Black Eagle class, Silver Snow route), I managed to recruit 9 out of 13 recruitable students (12, if siding with Golden Deer).
I can only guess as to why people think the recruitment limit is 3 for the initial playthrough and 6 for NG+. I believe that it's because most people focus too much on stats and skills, and forget that Support Bonds help reduce the recruitment requirement.
As a result, most players probably end up with Sylvain (free for female Byleth), Felix (because Sword and Speed will come naturally to most), and maybe Ignatz (for those who make good use of Batallions to train their Authority).
So don't forget to bond!
Honestly, there's actually no need to focus on any of the stats the students ask for. As long as you improve your Support with them to B, you'll have a much easier time recruiting them. Even if you don't manage to recruit them, students with a B Support may ask to join on their own volition, even if they refused to join the day before.
I've recruited some students that way, despite not putting any time in the skills they asked for, such as Flying for Ingrid, Riding for Marianne, and Lance for Leonie. Of the 4 I couldn't recruit, 3 only had a C Support, and Hilda was the last student I got B with, with Leonie asking to join the day before the end of Part 1. Bad timing on my part, I suppose...
If I had focused less on my own students' motivation, and a bit more on bonding with students from other houses, I believe I could've recruited all of them on my very first playthrough.
Note that if you've sided with Black Eagle, there is a point where you can lose 1 month's worth of time to recruit students depending on your choice. This extra month can make all the difference, as this is when I managed to recruit my last 4 students (all of them asked to join on that month).
Not all student are recruitable.
Obviously, the House Leaders can't be recruited at all, that is:
Their aide also cannot be recruited due to their loyalty towards their master. They are:
Hilda may seem unrecruitable at first, but can be invited to tea at some point. Starting with Month 3 (final month of Part 1), she can be given gifts, Lost Items, and finally recruited.