To anyone else who may be wondering, it appears to be a hardcoded variable in the ActionScript code for the interface.
See line 24 of InventoryDefines.as (courtesy of the SkyUI code repo):
static var QUANTITY_MENU_COUNT_LIMIT: Number = 5;
I haven't found a config setting that matches. It is possible to recompile the code as a mod though...
UPDATE: It works! If you really want to change this you'll need Adobe Flash CS5 (ActionScript 2.0). I have zero experience with Flash so did this mostly out of curiosity. Figured it would be too minor a change to put on SkymrimNexus.
Thanks to the hard work of the SkyUI mod team, you can republish directly from the
.fla files. They went through all the flash files, decompiled them with Trillix (my guess) and cleaned /fixed the code as a working basis for the SkyUI mod. You can find the "un-modded" version of the game interface code in the vanillasky branch.
For this particular interface tweak, you'll want to edit InventoryDefines.as as described above. Then open itemcard.fla in Flash, and republish. Note that you'll have to provide the fonts (search on SkyrimNexus). Place the file in the Skyrim Data folder:
<Skyrim folder>/Data/Interface/inventory components/itemcard.swf
I've tested in game successfully with a limit of 10 instead of 5, trading with an innkeeper, my follower and interacting with a chest. Can't guarantee that it is 100% stable as that would require much more thorough testing ("ItemCard" appears during enchanting, alchemy, and a lot of other things).