Why are three/four of the shapes on Brick's skill tree hexagonal while the rest are square? Aesthetics?
The skills on the tree with a hexagon shape are skills that have some sort of time limit on how long they last. The fist for example increases melee damage for a few seconds. Juggernaut reduces damage for a few seconds, etc.
The other characters have similar skills that last for a limited amount of time.
The more you upgrade these skills the longer that skill lasts.
Hexagonally outlined skills are what are referred to as "on-kill skills". These skills trigger upon killing an enemy. There are other skills that can be considered on-kill skills, but seem to be disqualified based on being instantaneous.
Unlike Doozer Blake's answer and CodexArcanum's comment, hexagons do not outline all "skills that have some sort of time limit on how long they last" and it is not the case that "hexes are powers that trigger on something (hitting, being hit, using active skill) and then last for a while after." Examples to the contrary include Lilith's Silent Resolve skill and Brick's Payback skill.