When checking my trap lines for rabbits, I keep finding traps that are not shaking, but are in the "down" position; they look kind of squarish instead of rounded as well. The interaction option is "check trap", but there's never a rabbit inside in that case. Is this a bug or glitch of some kind, or is this a new feature added to the game since the last time I played?
No, this is not a bug. I'm unsure when the last time you played was, but when I started playing in May 2013, I experienced this as well. It means that whatever you were trying to trap triggered it, but got away.
As per the wiki article on Trapping, where you can also see a picture to verify if you scroll down:
Sometimes, a trap may fall but not catch an animal that has moved out of the way too quickly. After a while, it'll appear to shake with a creature inside, but it'll lose durability if picked up in either state.
To reduce the chance of this happening, I place my traps right on top of the rabbit holes. This makes it so that they are trapped immediately right when they exit. I'd suggest using this method, as the durability loss on traps without catching anything will hurt your efficiency in food gathering.