When making a fabric server and adding mods to it, do you have to download the mods the server has in order to even join your Minecraft server?
Most of the time
There are 2 main types of mods used in minecraft server. Plugins and your typical mods, "plugin" is just another way to say "server side only mod" That means server plugins dont need to be installed on the client. Some examples of a server plugin are WorldEdit and other tools used on the server. Those arent the same as normal mods, like rlcraft or galacticraft that add new things to the game(blocks, items, ui, settings, textures, ect) those must be installed on the client and server.