You need to analyze the problem and work your way towards a solution by eliminating possible culprits. Here are the main reasons for lag:
- lack of memory
- HD reading
- CPU load
- software bugs.
All of them are possible to be fixed by throwing hardware at them, but depending on the size of your server, you will not be able to fix those permanently. They will come back once the bottleneck is reached.
First of all, you need to know where the bottle neck is.
For the memory, you need one of the plugins that let you monitor the memory usage. Essentials and Commandbook allow that AFAIk. You also need to make sure that your server does not allocate all the memory from the start since this will disable you from seeing how the memory usage changes. I have the following startup command for a Quad core 16GB:
java -Xmx10G -Xmn512M -XX:ParallelGCThreads=4 -jar craftbukkit.jar nogui
This defines 512MB For faster moving stuff and helped me a lot. I have a max usage of 10GB. When I start the server, it uses pretty much only 2GB and I can see with the /mem command from essentials where I am as it grows slowly over the day. If you realize you are running out of memory, a restart is often the only solution. There are a lot of plugins that have memory leaks and will not give memory back after they have used it. So a scheduled restart (I do it once ever 24 hours, when it's 4am in the USA) does the trick. It takes only 30-60 seconds anyhow. You need to read this page to get a better idea for the parameters.
This one is more tricky to detect. It's mostly affected by people joining/leaving/teleporting around, large viewing distance and a general spread of users over large spaces. This can be easiest fixed with a RAM Disk. If you are running low on RAM, remember that in case you have multiple worlds, you do not need to have ALL your worlds in the RAM disk. Make a Ramdisk and only move your most frequently moved world into it, symlink it and you will be fine.
This one is tricky since there are several things that you will have trouble finding. Mob spawners, redstone, animals & mobs in general, can cause a lot of lag. You will have to monitor your CPU level. Check if it fluctuates if certain users are joining or if people go to certain areas. One of my users had a large ice field with torches on it. The torches would permanently melt the ice and it re-froze. This massively lagged the server. There are also plugins that limit mob spawning. Sheep eating grass that grows back are an issue there. If you have users that think they need a sheep farm with 2k sheep, you are in trouble. Make rules and check or limit with plugins. Redstone is bad if it runs permanently without anyone pushing a start button. You need to make a rule that redstone contraptions should not run by themselves longer than 2-3 minutes or so. Get a plugin like Nolag and use the /nl examine command to see if you can pinpoint lag to a special plugin.
As I mentioned before, there are lots of plugins that use a lot of CPU or Memory. Minecraft itself, Bukkit etc included. You should try to always run the latest bukkit release and get updated on the plugins. Minecraft itself causes massive lag for example when you have lava or water flowing over a surface in large quantities. Falling water and lava is fine, but a 200 block wide flow of lava over steps is not good. Also, if you have a world where you do not need caves, fill them. This will prevent mobs from spawning and rendering will speed up dramatically.