I've removed many player-made patches of dirt (from removed projects), but I'm wondering if there are un-removable patches of dirt in my town. Are there unremovable patches of dirt?
All dirt can host grass on top of it unless it's under a building. All visible dirt will, given the correct methods, be able to regrow grass.
So no, there are no permanent dirt patches.