Once I have a worker assigned to a building, how do I remove the worker so I can remove the structure? You can't seem to fire workers and you can't delete a structure that has them. In this case I'd rather not have a stadium but an athlete is already in place.
As far as I know, unless there is another place your worker can move to, you simply can't. A good alternative would be to move the stadium as far away from the rest of your stuff as possible, but I believe that is the best you can do.
I thought I'd share this here, to to answer half of the question in case people are still looking at this:
Also, here's a screenshot from my game, showing that a message "Cannot delete the only building left" appears at the top when when I try to delete a building, of which I only have one:
So, while copies of a building can be deleted, the original cannot.
This is still true as of update 1.60 (Oct 23, 2013)