The reason why
/execute if entity @e[limit=10] does not work for this is that you're essentially telling the game this: "Make a list of all entities and limit that list to 10 entities. Then do something if there is something on that list." Limiting the list does not help you in figuring out whether there are more or less than 10 entities.
AMJ's answer is a method of counting entities that worked for a long time already (updated to 1.13+ command syntax), but since 1.13 you can also do it with better performance by using the length of Minecraft's internal entity lists instead of iterating your command over each of them.
You need a scoreboard objective for this as well:
/scoreboard objectives add entities dummy
Now you can efficiently count all entities with this slightly weird looking command:
/execute store result score @s entities if entity @e
You actually do not need
run and a following command here. This is one of the few use cases of the fact that the return value of
/execute if behaves similar to the old
/testfor command in 1.12 and before: It returns the number of matching entities, which you can then store in a scoreboard using
You can then use the score like in AMJ's answer, for example doing something if there are at least 5 and also at most 15 entities is done like this:
/execute if score @s entities matches 10..15 run …
This method is very efficient, because Minecraft already has a list of all entities and executing this command causes it to take a shortcut: Instead of iterating over all entities and adding 1 every time, it just gives back the length of that list, which is already in memory. This will be very noticeable if you use such a command for large amounts of entities or a lot of times, like in looping functions.
Of course you'll want to limit the selector
@e somehow when you actually use this command. If you for example use
@e[type=zombie], Minecraft does not go through all entities in the world and checks each for being a zombie, but also takes a shortcut, because for efficiency, Minecraft stores a list of entities per type, so in this case it can just return the length of the list of all zombies.
Because these lists per entity type exist, it is actually more efficient to write
@e[type=armor_stand,tag=foo] instead of just
@e[tag=foo], even if you know for sure that nothing else ever gets tagged with
foo, because Minecraft only needs to check all armour stands for the tag and not all entities.
Source for efficiency explanation: https://minecraftcommands.github.io/commanders-handbook/selector-argument-order (archive)