What is and how does World of Warcraft's Custom Lag Tolerance setting work?
Whenever you want to cast a spell, there's a delay between the time you press the button and the time you actually start casting. That delay is determined by how long it takes for you to talk to the server: you need to tell it you want to cast this spell, and the server has to say "OK, casting starts now". This delay is called latency, or more informally, lag.
You can take advantage of this delay to reduce the time you actually spend between spellcasts: just before you finish casting one spell, you start casting another. If timed correctly, you will finish the first cast and immediately start the second cast, due to the time it takes to talk to the server.
Of course, in order to do that, you need to know exactly how long it took to talk to the server, and since we're generally talking a few hundrer milliseconds, it's not easy to measure on your own.
Custom Lag Tolerance is intended to help you with that: once you know the approximate latency you're playing with, you can set it up and the cast bar display will take this value into account so you can chain spells more effectively.
Instead of using the built-in functionality, there are also addons which actively measure the latency for each spell cast and adjusts the cast bar based on that. If you're using PitBull Unit Frames, it includes a module for this which you can enable. Before I switched unit frames, I used Quartz as a cast bar replacement, which also contains this functionality.