Personal experience:
The screen needs to be on and Pokemon GOGo needs to be active.
It will vibrate when a Pokemon is near.
When you get the accessory called Pokemon GOGo Plus it will notify when Pokemon are near even when the screen is not active.
Pokemon Go has a power save mode, when you turn it on it can be turned up side down in your pocket, the screen will go blackish with the Pokemon Go logo on it, it will vibrate if a Pokemon is near