I run my own Minecraft server on our local network. It has a quadcore AMD Phenom II 955 CPU with 12 gigs of ram. This shouldn't be a bottleneck. The operating system is Windows Server 2012 Enterprise. Our internal network is 1 Gb/s and outgoing is 100 Mb/s down and 10 Mb/s up and around 20-30 ms ping with players.
It is fine with one player. No block lag (when mined blocks don't appear back), lag when walking or anything. When a second player joins, on home network or from somewhere else, block lag and movement lag starts.
I have allocated Java 4 GB of memory and I've set its priority to real time in Windows Task Manager. In case of one or five players, CPU usage stays at somewhat stable 20 % for Java. Why doesn't it take more power to help with the issues? What optimizations could I do?