I was in the jarls palace in whiterun and "accidentally" took an item called "unusual gem." Now every time I go to whiterun then I get massacred by a huge group of guards and if I kill anybody I'll just be digging myself into a bigger hole. :(
Just taking that item (Which is a quest item) shouldn't cause them to attack you.. However. Draw your weapon then put it away, if it really is just a minor infraction over the limit they should also put their weapons away and then confront you to let you pay off the debt or go to jail.