Throw a regular ping onto him so your mid/bot (maybe top) know that he is there. Do not engage the buff monster (btw -- always fight these monsters in the bush so you get a heads up invades, sounds like you know this already but just in case).
This plays out in one of a few ways:
1) He overextends and tries to fight you and your allies come and kill him with you.
2) He overextends and tries to fight you and you 1v1 him because you're stronger, he leaves and has to base and is now behind potentially.
3) He fights you but you're weaker and no allies respond to your pings because you're in silver -- disengage. If he attempts to steal your buff, you fight him and it's a 2v1 and you win. Under no scenario do you let him take the buff for free, even as a weaker jungler if it's JUST him, you go for the 2v1 + smite steal attempt.
4) He attempts to take buff, you counter him as in #3, then he leaves, then you start the buff, then he comes back. This is the trickiest of them all -- I would highly recommend not starting the buff UNLESS your mid came and scared him off for good. If not, you should time your smite + most powerful ability to finish the killing blow, so he cannot simply smite steal if he has or gets vision
5) You're doing a buff, and he gets the jump on you and kills you or takes your buff. You either were unlucky (low %), or you failed to time your smite+killing blow ability properly, OR, most critically bad of all, you didn't pull a buff monster into a bush.
6) If you're a jungler with a ward (i.e. Aatrox), you can make this trivial by warding a path to your buff / his escape route and know for certainty if you can take the buff or not. If he doesn't see your ward, you can use this to spam assist pings and get your allies to come secure the kill for you.