The short answer to your specific question is that component/composite cables MAY help but most of the delay is probably elsewhere in your system. Read further for more details....
First, try to separate what's simply audio delay from "display lag." Most of the answers here address display lag, which is a video issue.
For audio lag:
- Here the biggest concern is the time between when you play and when you hear the sound. Play while the game is paused. If it's still bad, it's audio lag.
- Do not run audio through your TV in any fashion. This includes using HDMI for both audio and video between your PS3 and TV and then using the TV's audio output (digital or analog -- doesn't matter) to send to your speakers or receiver. This usually causes a lot of delay, because very few TVs are designed to handle this splitting/pass-through quickly. If you have no receiver or speaker system and are using the TV's speakers you have little choice. Sorry.
- There's an inherent delay in the Rocksmith RealTone cable to do some ADC (analog-digital conversion), frequency analysis, etc of your guitar's output and for the Rocksmith software to interpret it, run it through its software mixer (the pedals, amps, etc you have set up), and then send the audio out. Some of this delay is purely in detection for the purposes of checking your play against the notes coming down the "highway," and I think Rocksmith goes ahead and passes through the audio before getting feedback on this, but regardless some audio delay is just unavoidable because you're running your guitar's sound through a lot of equipment.
- (Info) An SPDIF out (or digital out) such as your optical setup is often considered better because it transmits the audio in a digital version. Your receiver then converts it back. Digital transmission is less prone to signal degradation and interference over the cable itself. Optical (tos) cables are even better protected from electromagnetic interference because it's light transmitting the data and not electric current. However, optical cables are not inherently faster. The fact that "light travels faster than sound" is irrelevant here because it's not sound being transmitted in the case of a coaxial (copper) cable for digital audio. It's electric current, which moves essentially at the speed of light (close enough anyway - no less than half the speed of light, even through a crappy cable)
- Digital out (whether optical or coaxial) may be better than RCA, but it does induce some delay. It takes extra digital signal processing time for your PS3 to output the data and for your receiver to convert it into a signal for the speakers. In most cases this delay is small enough that it's not noticeable, but it's certainly possible that a bad receiver or one which doesn't support the PS3's output (see next point) might add noticeable delay, in which case a composite or component cable will help.
- Be sure your PS3 is outputting a signal your receiver supports. Extra delay may come about from the receiver having to downgrade the signal. When in doubt, use stereo with the best sample rate your receiver says it can handle.
For video/display lag:
- Here the biggest concern is that you see a note coming down the highway at the right time so when you play at the right time, Rocksmith agrees. Note that this is probably completely different from audio lag, so fix that first. There's a lot less you can do here but also a lot less you can screw up.
- As others have suggested, toy with your TV's video settings. The less "clean-up" it's trying to do with the signal the quicker you'll get the video.
- You can try composite/component cables. They may be faster to display on your TV, but these days it's still handled digitally, so chances are it won't make a big difference. (This is why some old NES games and such are nearly impossible to play now. Pretty much all modern TVs add about one frame of latency that wasn't there with CRT displays).
- After reducing the delay as much as you can through these means, use the Rocksmith setting to change the delay.
With a half-way decent system and the right connections, you shouldn't have much audio or video lag. If it's very noticeable, something's probably wrong with your setup.
Good luck....