You can't use a simple command in macros to check this condition. The only conditional check for whether something is in combat or not is
[nocombat] and this only checks whether the player is in or out of combat.
You can however call scripts with a macro but of course you'll have an annoying character limit.
/script allows you to execute commands in the WoW scripting language through a macro.
I'm not very familiar with the language itself (maybe you'll find more information on GameDev.SE) but it certainly is possible to achieve this through a macro although it might be more effective to write a small add-on.
Now as I said. I don't know a lot about the WoW scripting language but I could imagine that the macro looks something like this:
/run if UnitAffectingCombat("nearestenemy") == true then TargetNearestEnemy(); end
Here you can find the article of WoWwiki about using scripts in macros.