Short answer:
Not really, there are three secret chests in total that require you do destroy/interact with certain object to appear. But later in the game you get Hoardseeker and Fury's Embrace that reveal the location of the chests on the map, so you can just safely leave the environment intact until the map says there is a chest that you can't seem to find.
Long answer:
There are three secret chests:
The Crossroads
Location
Main Plaza
Interaction
Break all six fire hydrants
Choking Grounds
Location
Graveyard
Interaction
Destroy all the big grave stones
Twilight Cathedral
Location
Basement
Interaction
Light all the torches
(source)
Bonus info:
There is a chest in Drowned Pass, but it is quest specific, so don't waste time searching for anything to break like I did. I don't remember what it was, it's a long time ago, but either it is a glitch and there is no chest (source) or you can only do it while you are doing the quest
for the rock door guy and you have to go into those 4 orange portals to get the door clear.
If you don't get it then, you can never get it, but it's only a chest of souls so it doesn't matter.(source)