As your commenters have pointed out, there are a couple of different level caps that you're running into.

In Warcraft 3's (and its expansion's) campaigns, your heroes will often have a level cap that grows as you progress through the campaign. In multiplayer games, your heroes can not advance beyond level 5 without fighting an opponent's units. Killing creeps will still reward gold, but you will get no further experience.

As you've already realized, you can tell when you've hit a level cap by the fact that your experience bar will be empty and will not fill when killing a creep. This is also an easy way to test whether or not you should spend money on an experience tome. First, kill a creep. If you gained experience, a purchased tome will have an effect. If no experience is gained, a tome will have no effect, since you are at a level cap.

As an addendum, in multiplayer games, I believe that a "Tome of Experience" will always have an effect, so long as your character is below level 10. The limit in multiplayer games is whether or not you will gain experience from creeps, since the hero level cap is always 10.