Once I have a worker assigned to a building, how do I remove the worker so I can remove the structure? You can't seem to fire workers and you can't delete a structure that has them. In this case I'd rather not have a stadium but an athlete is already in place.
As far as I know, unless there is another place your worker can move to, you simply can't. A good alternative would be to move the stadium as far away from the rest of your stuff as possible, but I believe that is the best you can do.