I just want to know of it is possible to install mods into my own server that I have already played in. This is for me and my friends to play in.
Yes, (usually) you can do that.
Not being able to would be pretty pointless, since no one sets up a whole server with all plugins and mods without ever joining it to test things before it's set up completely.
Some mods might require a new map (usually their own one), though. But I've never seen a mod requiring a brand new server.
So the simply answer is - it depends. Not so simple..
So there is two things to consider.
Depending on the mod you might need a specific version of Minecraft (as some mods only support specific version of Minecraft) - so if you have let's say version 1.9, and the plugin only supports 1.10 +, then you would need to update the server (the .jar file) to a version the mod supports. The supported versions should all be listed on the mod-page (where you download it from).
Some mods require forge (or some "mod-loader"). In these cases, you'd need to use a completely different .jar file all together.
Also it's probably worth pointing out - if you are playing on a Minecraft server that you created, it's more then likely Bukkit/Spigot. If you're trying to download forge mods and put them into your Bukkit/Spigot folder - it won't work. Forge has their own launcher for that.
Hope it helps. If you want to add more information I can help further.