It does not seem to matter how fast you beat on them to determine the quality of the overall drops. The most important thing is to actually kill them before they open a portal and escape. The last drops upon death (and possibly right before death) are magic items, as opposed to gold, so it is well worth making certain you kill them.
There are different types of treasure goblins, and they seem to have different qualities of loot.
I encountered a Treasure Pygmy during Act 3 of normal, and, despite not killing him particularly quickly (certainly not as fast as I had killed some early treasure goblins), I had the extreme fortune of having 3 rare items drop from him. I'm not certain if this is simply an extremely lucky coincidence, or if the quality of the drops was influenced by the particular monster title.
Edit: I just encountered a Treasure Goblin much later in Act 3, and he dropped 4 regular magic items (no rares), even though I beat him down much faster than the Treasure Pygmy. This seems to corroborate the idea that the Treasure Pygmy may simply be a rarer, better version of the Treasure Goblin.
Edit: After quite a few more tests, ranging from chasing the Treasure Goblin (of a number of varieties) around, to beating them down before they could move more than a few yards, to killing them moments before they step through a portal, there is absolutely no discernible difference in the quantity or quality of items they drop. They always drop 1-4 magic items, plus gold, gems, and crafting pages/tomes (if you are high enough level for gems and/or crafting pages/tomes to drop).