There are ways to detect if a player is far from a command block or villager but how do you give an effect to a player who leaves a certain radius from another player?
You're gonna need execute for this one. Execute a command on the first player that selects the nearest player outside your radius and gives them the effect.
execute player_who_gives_others_the_effect ~0 ~0 ~0 effect @p[rm=5,c=1] minecraft:nausea
rm = minimum radius
This command gives the player outside the radius nausea. Find the other effects here: http://minecraft.gamepedia.com/Data_values#Status_effects