These doors are grey and usually (always?) open into a "weapon" room, i.e. there is a weapon on the minimap for them. I've gotten in two, but it was very non-obvious what I did to open the doors. For other doors, I've killed every mob on the floor, put out every fire, and dropped a bomb on the door and it still didn't open. Does anybody know what the rule is for these things?
The doors are tied to your health. The door will only be open if you have full health. However, the temporary HP boosts (blue hearts) can serve as a equivalent of the missing red hearts.
For example: If you're missing one red heart but have one blue heart the door will open. But if you're missing two red hearts and only have one blue heart the door will still be locked.
The doors of Challenge Rooms are tied to your health:
The normal doors will only open if the sum of your current HP (counting red, blue, and black hearts) is at least equal to the number of red-hearts-containers you have.
The boss doors (recognizable from the skull they have on the door head) require the player to have only one red heart at most, but the amount of blue and black hearts are not taken into account. It is best to enter them with a Full Health pill, or with plenty of blue/black hearts (the health gambling machines can be used to reduce the amount of filled red-heart containers while keeping the blue and black hearts intact).
If you get items such as Transcendence or Lord of the Pit, or the Hanged Man tarot card, you can actually fly through/over the door, regardless of your health level.
EDIT: I believe this applied to the original Flash game; it does not apply to Rebirth and newer. Hence my answer is valid given the time at which the question was asked.