It's very situational. If you want to get more influence from a city-state, return the worker. If you want to get to friendlier terms with the civilization, return the worker. Pretty straightforward. I do have some more specific insights about that, though:
- Not returning a worker has no bad influence. A civilization won't frown upon you if you just don't return it.
- In the early game workers are extremely important. I don't think it's a good idea to return unless you have to (e.g. have to placate a potential enemy).
- The benefit from returning a worker to a city-state is 30 friendship points. That's the same as a diplomatic mission and is slightly less than fulfilling a quest. It's pretty nice and can turn a neutral city-state into a friend or a friend into an ally.
- The influence gained from returning a worker to another civilization is noticeable but not very large. It affects them slightly less than if you denounce one of their enemies. Its level of positive effect is comparable to the level of negative effect you get by breaking a promise (e.g. promising not to expand close to their borders, and then breaking it). Also keep in mind that unlike city-states, other civilizations are in direct competition with you so you want to be wary of giving them stuff.
My personal approach - except for early game or if I really want one, in which case I don't return it - is to only return to other city-states.