Minecraft worlds are almost infinite in size. You can walk millions of blocks in one direction and there's still more to come. Minecraft does a lot of things in every loaded chunk all the time and there is just no way to have such giant worlds and having all of it loaded all at once. That's the reason why Minecraft loads the worlds in chunks.
One chunk is a 16x16 area from the bottom of the world (height 0) to the maximum build height (height 255). The chunks around you are always loaded, but only up to a certain distance from you. You can control how far this distance is with the slider "Render distance" in the video settings. You can try it out and observe how the distance you can view changes with it. But unless you have a really powerful computer, you will notice a significant performance decrease when you set the limit high.
If you go farther than that distance away from the village, the chunks are no longer loaded and time will pretty much freeze for that area. Villagers won't grow up, water won't flow, redstone machines will pause.
There are ways to keep any chunk loaded that you want, but that affects performance and is complicated to do. But there is a certain area in any Minecraft world that is always loaded: The spawn chunks. Those are the chunks around the place where you originally spawned in the world. You can find the location of them for your world using tools like this one. You can then either move your villagers there or, if you're really lucky, there might even be a village there already.
This part does no longer apply to 1.14. Chunk loading is very broken in that version, so you need a player around to load chunks properly.
So, in summary:
- If your mine is closer than 512 blocks to the village, you can try turning up your render distance. But if your PC isn't powerful enough, the game could become like a slide-show or even freeze completely.
You can move the villagers to the spawn chunks.
- Or you can stay in the area.
(P.S.: No, the game can not only load the most important things. Even if you only consider 0,0001% of the world important, that's still 14062500 chunks to be loaded, that's 3328 times as much as what would currently be loaded with the most computationally intense setting.)