I removed my follower's bow, arrows, weapons and for some time they're just using their fists to fight the enemies but a bit later on as the fight goes on, they suddenly now have a bow or a knife that they seem to have gotten out of thin air. Is there a way/mod to stop this from happening? I want them to only use what's on their inventory and not pull a weapon out of thin air.
Followers can pick up dropped weapons if they don't have any to use so its likely that they picked up something from the enemies you have killed.
There is one possible way of forcing a follower to stay unarmed and that is by fortifying their unarmed to the point where it does more damage than whatever weapon they pick up. This requires using an exploit or console to achieve as the basic unarmed enchantment is not stronger than most weapons. I'm not sure if this will work but it's worth trying.
Finally there are various mods that add brawler type followers that will stay unarmed.