Without looking at the game code it is hard to say. I did some experimentation and learned the following:
The reset time seems to be mostly dependent on time.
The items will not restock while you remain in town.
Teleporting back and forth repeatedly is not usually enough time to cause a restock.
The reset times appear to be separate for each merchant. I purchased an item from the blacksmith and then one from the gem seller some time apart (less than one minute). I then teleported in and out until the blacksmith reset. I then discovered that the gem seller had not reset.
Reset time is not dependent on number of loading screens. The merchants did not appear restock when I gated back to the dungeon right next to the stairs down, proceeded down one floor and teleported back immediately after arriving on the lower floor. The gem seller did restock if I waited about 3 minutes in the dungeon and then teleported back. The gem seller also restocked when I waited about 3 minutes in town, teleported to the dungeon and then back immediately.
After I examined these observations, I believe that the restock is primarily dependent on time. I believe the restock interval is less than 4 minutes but longer than 1 minute. The exact time is hard to establish using my method. My educated guess is that when you return to town, the game checks the time since last restock for each merchant and restocks if more time has passed than some set value. If the game restocks every x minutes while you are out of town is impossible to tell since you only get to see the latest restock.
By teleporting I mean I used town scrolls. I tested all this on my level 51 vanquisher teleporting to and from the 42th floor of the shadow vault. I am using the following mods which affects Merchants: Better Balanced Sales Prices, Awful Arthur the fishmonger, Additional Gem Availability, Spell Dealer. None of their description say that they affect restock times.