While it may or may not be feasible for a player to make the lengthy trek to your area, the simple fact of the matter is, they don't have to, to grief you. Destroying your stuff isn't the only way to grief your server
As has already been mentioned in the comments, there are things they could do to slow the whole server down; massive animal farms are an easy way, as would be a large redstone contraption that mostly just triggers a vast amount of block updates every tick. And the worst part is, if someone does try to make something like this, you'll have to actually remove it somehow; just banning the player responsible will leave the source of the lag lying around, with its chunks unloaded, and once another new player shows up and loads those chunks, the griefing will be happening all over again.
Admittedly, griefers who will do things like this are rarer than people who will just wander over to your place and trash it. And there are ways to prevent it, and/or deal with it after the fact.
- You can use something like MCEdit to directly clean up the situation, even if the server is utterly crippled.
- You can increase the spawn-protection value in your server.properties file. Crank it up enough, and all new players will have to wander far enough away before they can do anything that a brand new player won't be loading potentially-compromised chunks.
- You could also turn this around; leave the default spawn at (0,0), crank up the spawn protection enough to cover your place and a large surrounding area, and non-op players won't be able to damage things inside that radius.
- You could even go more draconian, and create a wall around the default spawn, inside the spawn protection radius, so new players can't get out of it, then require an op to teleport all new players out after having a brief chat. Most griefers will give up quickly when they can't do anything, but you'll probably also lose a lot of good players if there isn't an op on pretty constantly to greet new people.
Ultimately, while isolating yourself from new players might be useful in preventing griefing, it's not a solution on its own. At the very least, you'd be leaving new players at the mercy of the same griefers you're trying to keep out, which could very well end up discouraging the exact sort of players you're hoping to keep around.