When calculating chances to hit, how do you properly setup their velocity vectors for two ships trying to orbit each other? And one ship trying to orbit the other ship who is approaching?
You should be using the transverse velocity of the ship that is orbiting you and make the vector for that speed perpendicular to the ship that is firing. Leave the ship that is firing stationary.
The math gets more complicated when you make an elliptical orbit, as you need to change the direction of the orbiting ship's velocity; it is no longer perfectly perpendicular. The best model for this could be the speed and direction each ship is moving in at the exact moment of firing, but that isn't exactly what you want, is it?
Full disclaimer: I always screwed up vector assignments in my physics class. I'd suggest looking up orbital state vectors for a better description. What follows are my best guesses as to how you'd go about this.
One approximation I think that will get the job done for you is to treat the slower ship as stationary and the orbital velocity of the faster object reduced by the speed of the slower ship. So an interceptor going 4k orbiting a rifter going 2k would be the equivalent of the interceptor going 2k orbiting a stationary rifter.
In all orbits, both bodies are actually in orbit. If the solar system consisted of the Earth and the Sun, the Sun would still have an orbit, though the radius of the orbit would be well inside the radius of the sun itself. In the case of two ships whose speeds are comparable, the effect you will see is roughly equivalent to a double helix, which is basically two circles being translated along another axis.