From the info you provided, your GPU is the more likely bottleneck. A lot of things could cause lag, but these are the most likely scenarios:
The game needs more than 512 MB worth of buffers to run at higher resolutions. If you're running at 1680x1050 for example, two screen-sized buffers with a 4-byte encoding will take 13 MB. But that's just the minimum; more buffers are required for textures, reflections, modern lighting techniques and so forth. As you increase the resolution, GTA might be trying to upload higher-quality textures to your GPU. Some buffers might end of up in system memory, which tends to be much slower for the GPU to access.
Your GPU just doesn't have the processing throughput required to run GTA's fragment shaders at a high resolution.
If you want to be certain about where the bottleneck is, try some GPU profiling software.
