I have a new potato farm, so I moved the villagers to it, I also have a villager farm on the other side of my house (about 35 blocks away from the potato farm). But the farmer doesn't care about the potatoes, he keeps trying to get to the village I am using for breeding? According to the wiki, they will forget the village in like 6 seconds or so as long as they are 32 blocks away from it.
A villager taken more than 32 blocks away from its village boundary will forget the village within about 6 seconds. Whether in a village or not, a villager is never prone to despawning."