In the 4th world, right before the final castle, there was a black question mark box world. I think it costs coins to enter the world. What is the significance of this black question mark box?
It's simply another level. Once you spend the coins, it will unlock, and you can play through it just like any other level. It's better to thing of these levels as having a "minimum" coin requirement, rather than a coin cost, as you keep your coins after unlocking them.
Eventually, the castle levels start requiring coins to unlock, which means you are unable to complete the world (and move on to the next one) without unlocking (and then beating) the level. But in other instances (like World 4), you lose nothing by skipping them (or by unlocking them, in fact).