I have tried different settings in the game but nothing solves the problem.
The game runs perfectly at high settings when I'm not in combat but when I fight with more than 2 enemies, the game no longer runs smooth. I'm 90% certain it's not from the spell effects, fog, blur, etc. I tried reinstalling it but nothing changed.
How can I make it work smooth again?
Seeing as you said the fan no longer spins properly
The more the video card has to render, the hotter it will become. The hotter it becomes, the more it relies on its heatsink and fan to disperse heat. If the fan is not spinning properly, you will have all sorts of issues trying to render graphic-intensive scenes - as you have described in your comments above.
Lubericating the fan may not solve your issue as you may have a problem with the bearings in the fan.
I would recommend replacing the fan on the graphics card or replacing the entire graphics card altogether.
On the software side, you could install the Streamline mod. It has a feature (part of its "Streamsmooth" component) that can temporarily lower graphical settings (causing higher FPS) when you are in combat. It also has other features that can improve FPS too.
[An] optional smoothing feature is the FPS combat booster. If you want slightly higher FPS during combat, you can simply set the boost variable in the INI file a bit higher, and the smoother will keep FPS nearer to the top of your regular FPS range automatically. Say you set your FPS range to be 20-30, and the combat boost to be 0.8. During combat, the smoother will attempt to get FPS 80% of the way from 20 to 30 (right around 28). When combat ends, it will go back to normal adjustments. If you set it above 1.0 (which is 100%), you can effectively boost FPS during combat -over- your normal range.
From: Streamline's 'SL Tome of Knowledge.txt' file
I'd also recommend installing Oblivion Stutter Remover (OSR). It should also significantly help with your framerate. Note that OSR has an FPS cap, by default. If you're using OSR, make sure you set MaximumFPS = 0 in sr_Oblivion_Stutter_Remover.ini to avoid OSR putting an artificial cap in your framerate (this is set to 30, by default). I recommend setting MinimumFPS to 20 or 25.
The vanilla Oblivion heap is unoptimized. OSR can replace the vanilla Oblivion heap with a different heap to improve performance. Set bReplaceHeap = 1 and change iHeapAlgorithm in sr_Oblivion_Stutter_Remover.ini to 1, 2 or 5. 1 usually works fast enough. Read sr_Oblivion_Stutter_Remover.txt for more info.
