In server.properties there is a spawn-monsters setting that you can set to false. It stops them from spawning, but does it remove existing monsters like setting the difficulty to peaceful? In other words, can I allow monsters for a little while and then safely turn them back off, or will creepers that spawned while it was on linger, awaiting unsuspecting victims?
As of 1.8, setting
difficulty=0 is the new equivalent to peaceful.
spawn-monsters=false will still get rid of the nightly mobs, but unless
difficulty=0, spawners will still spawn, and your health and food will not regenerate.
BEGIN OLD ANSWER:
In order for the changed setting to take effect, the server must be restarted. until restarted, monsters will still spawn. After a restart, there should be no monsters.