As your commenters have pointed out, there are a couple of different level caps that you're running into.
In Warcraft 3's (and its expansion's) campaigns, your heroes will often have a level cap that grows as you progress through the campaign. In multiplayer games, your heroes can not advance beyond level 5 without fighting an opponent's units. Killing creeps will still reward gold, but you will get no further experience.
As you've already realized, you can tell when you've hit a level cap by the fact that your experience bar will be empty and will not fill when killing a creep. This is also an easy way to test whether or not you should spend money on an experience tome. First, kill a creep. If you gained experience, a purchased tome will have an effect. If no experience as gained, a tome will have no effect, since you are at a level cap.