An enemy counts as flanked when you attack from a direction their cover does not block. This can be spotty at times and it is also based on line of sight of your soldiers and the enemy. You can check how flanked something might be by rotating the camera to the direction the enemy is coming from and seeing how much of your soldier is out in the open.
For example an alien was counted as flanked for me once when he rested against a gas station counter on one side but had the little thin arm that blocks you from moving on the other side. The arm was so thin that it didn't count as cover and he was considered flanked.
Finally enemies such as chrysalids, muton berserkers or any other enemy that does not use cover cannot be flanked. Only enemies who actively rest against an object to use as cover can be flanked.