I want to play modded survival with my friends and I have the mods, do they have to download all of those or do I just have to have them?
You need to install the mods onto the server, make sure you download the server version of the mod, if you can't find one then the mod probably doesn't have one.
BOTH of you need the mods. The way it works is the server checks to see if your client is compatible. If it doesn't have everything then it just simply won't let you join because your client would most likely crash anyway.
You can try to find a server that already has mods in it. Tekkit is a wildly used one.
There are also things called plugins, with those, neither of you need to download anything but plugins are only available on bukkit servers.