According to the Animal Crossing wiki, the museum will return only 3 fossils per day, but if you've sent more than 3 in a day the rest will be returned on other days:
There is no limit to the amount of fossils that can be sent to the Museum in a single day. However, only three will be delivered back to the player each day, with the rest delivered on succeeding days.
Assuming that's true, and that you can find up to 5 fossils in a day, then you can quickly build up a "backlog" of fossils. So if you send 10 fossils in two days, the museum will return only 6 in the same amount of time, then 3 more the day after and 1 final fossil after that.
There must be a maximum number of extra fossils that can be stored in that "backlog". Has anyone found out what that number is?