Does this game use the New Leaf rules?
No, it's not quite like New Leaf but back to that in a sec.
Yes, you need space all around the tree/sapling, as in it can't be up against a building, right next to the slope to the beach, or even on that peninsula jutting out to the right on the island. A tree that cannot grow in that spot will just remain a stunted sapling.
A good way to test whether a tree can grow in that spot is to dig up a sapling from elsewhere and try to plant it in the target spot (rather than planting the seed or fruit). The hole won't let you plant the sapling if it's a bad location.
But back to that New Leaf stuff, the weird issue of tree spacing (in which the tree in the middle of a group of trees wouldn't grow) has been resolved: