Currently, you are able to gain ~125% of the initial value of the given science. Each time you perform science, you reduce the value you will receive in the future. Regardless of how you acquire it, you will always end up maxing out at ~125% the initial value for a particular area and method of doing science. It doesn't matter if you transmit your science or return it to Kerbin. The actual maximum is dependant on the type of science, and varies between 110% and 180% of the initial value.
Also, note that I mentioned the area that science is performed. There's a difference between the surface of Kerbin, the upper atmosphere of Kerbin, and the orbit of the Mun, and each of the places, along with others have their own maximums of science that can be performed. Also note that the science you get from these different areas can be scaled differently. The KSP wiki article on science explains all of these parameters (or in some cases, at least tries to) much better than I could in this limited space.
Now, there is good reason for choosing transmission over return, or vice versa. Early in the career, you don't have access to RTG's, solar panels, or even batteries. This makes transmitting science impractical at best, as you need electricity from your engines in order to transmit again, so you need to return your science. Conversely, if you're sending a robotic mission to Jool, it's impractical return a dozen Science Jr. pods and Mystery Goo containers if you want to gather science from as many of the moons as possible. You're better off sticking a bunch of solar panels on it and transmitting often. As of the 0.23 update, you also have to weigh the effect of the transmission cap on science: you'll only ever get 40% of the maximum science from transmission alone; and without a lab attached, some of the experiments are single use only. Sooner or later, you'll need to return the samples.
Basically, if the cost (in terms of electricity) isn't too high to transmit and you're able to reacquire the same science easily, transmit it, but always hold on to the last bit of science you have if you're returning the craft to Kerbin.
Basically, prefer return over transmission due to transmission cap and single use experiments, but always include the possibility of transmission.