Do they care if you cut down a city tree in one of the set pieces?
Bonus question, feel free to ignore: Do the guards care if you mine the rocks next to the rock cutters?
I tested this out with a couple short games.
The guards will get argo'd with the first hit of the axe to the tree in a city set piece. So they aren't worth chopping down.
The rocks in the middle of the rock cutters area will also agro the guards but only on the final hit that destroys the pile. They don't care at all about the trees and rocks outside of the fixed set-pieces. And I think anything you plant is fair game.