Whether you'll need to switch server depends on what mods you want to use. Mods are developed for certain "delivery means" - either they plug in to Bukkit, Forge, or other mod manager, or they are supposed to be installed manually.
Manual installation means that you open minecraft_server.jar as a ZIP archive, take mod's files and replace similar files inside the game. This is least convenient, and there aren't much mods designed for that anymore.
Forge is a "plugin interface" for vanilla game. It means it is a mod that is installed manually (or via a "patcher" progrem that essentially does that for you), and whose responsibility is to "plug in" other mods into the game. With Forge installed, adding Forge mods is as easy as dropping mod's archive into
/mods folder and restarting a server.
Bukkit, Spigot and others are custom servers, built from the ground up by people separate from Mojang. They are, of course, supposed to run the game exactly as vanilla game does, so official clients can connect and play normally. There are small changes, of course, like dropped items behaving differently, but world format is the same, so worlds can be transferred between them (well, at least from vanilla to custom :-). Those servers have plugin interfaces built into them, so, too, just drop the plugin into a folder and restart.
Whether you will lose your world also depends on mods you choose, but in a different way. Some mods just change existing rules of the game or add monsters or items (like minimaps or airships) - those are fine. Mods that you will need to watch out for are mods that change world generation. If you play with default world generator, and then swap it for another, the game will run okay, but as soon as you reach the edge of explored land, the landscape will be torn apart and a whole new world will start beyond - the game has no way to "stitch" together old and new landscape.
On the other hand, this may not be the case. If a mod just adds new ores, without changing how the landmass is generated, there won't be any problems, you will just have to travel to new places to find those ores (because explored chunks are already generated and saved, no new ores there).