I have observed that citizens tend to switch their jobs from time to time. I am not sure what causes this. But I noticed that it sometimes causes the distances to workplaces to become less ideal than they used to be.
For that reason, you should re-optimize your workers from time to time. To do that you can use the rule that when you have open jobs and assign more workers to that profession, the laborers closest to the open job will be converted.
When you have, for example, a gatherer living near a quarry and a stonecutter living near a gatherer hut in the woods, and you would like them to swap their jobs, reduce the amount of workers in each profession until both are laborers. Then reassign the workers to the professions, one by one alternating between the two. Whenever you add a worker to a profession, the laborer closest to the open position will be picked, so this will cause these two workers to get the job closest to them.
But instead of constantly monitoring your workers and micro-optimizing them, you should instead regularly (every few years) re-optimize the whole village by unassigning all professions and reassigning them in order of priority.