I own the Guitar Hero instruments. Can I buy a Rock Band game and use the instruments from Guitar Hero to play the game? And vice versa?
It depends on the instruments.
Here is a really nice compatibility chart that I found when I had the same question.
As JustinP8 pointed out, you can also check here:
But there's no need to read that, as long as we're talking about a game released in the last 2 years or so, all the plastic instruments are broadly cross-compatible and work in either game — microphone, drums, guitar. Pick Rock Band instruments, pick Guitar Hero instruments, everything "just works" in any rhythm band game.
You do need to be careful on drums, because the 3 pad / 2 cymbal arrangement of the Guitar Hero drums makes 4 pad Rock Band songs all but unplayable.
Remember that GH supports a 4-lane drum pad mode, but RB does not support a 5 lane mode, nor will RB3 "pro mode" 4 pad / 3 cymbal drums support GH's oddball 3 pad / 2 cymbal arrangement.
So if you plan to play drums in both games, I strongly recommend you get a Rock Band drumset only.
An additional way to do this is with very cheap adapters:
I have GH 1 for the PS2, GH 2 and 3 for PS3, and RB 1 and 2 for PS3 and I find the worst guitar of the lot is the Les Paul (it just seems to suffer interference) so while I use the Rock Band Stratocaster for Lead, I use a GH1 guitar for bass - through an Ant Commando PS2-PS3 adapter. Only cost me £15.
If you want a larger compatibility chart, check out this PDF from the Guitar Hero website. It's got a lot more games.
For the record, you might be surprised to know that PS2 games started being cross-compatible with PS3 and 360 instruments starting with GH: Aerosmith. I have personally tested a 360 Rock Band drum kit with the full-band GH games on the PS2, and they work fine, it goes into 4-lane mode and everything.