In server.properties there is a spawn-monsters setting that you can set to false. It stops them from spawning, but does it remove existing monsters like setting the difficulty to peaceful? In other words, can I allow monsters for a little while and then safely turn them back off, or will creepers that spawned while it was on linger, awaiting unsuspecting victims?
As of 1.8, setting
BEGIN OLD ANSWER:
In order for the changed setting to take effect, the server must be restarted. until restarted, monsters will still spawn. After a restart, there should be no monsters.