Follower Commentary Overhaul has got you covered.
From the mod's FAQ:
Q) I keep an unsafe amount of followers in my house and it's a
cacophony of madness!! How do I stop it
A) Don't fret, I've got you
covered! FCO includes an MCM where you can adjust the frequency at
which comments are made as well as which voicetypes are able to make
them! 100% is the default frequency, and is balanced best for vanilla
play or small groups of followers, but if you roll with a large
following in combat or at home, you can slide it on all the way down
to 10% frequency. If you only want particular voicetypes chiming in,
or just can't stand one in particular, you can switch them on and off
at will as well.
You can just set frequency of all comments to 0, which will effectively disable them.