What is the max bonus damage bristleback can deal with his "quill spray" and "bristleback" abilities?
Bristleback's Quill Spray stacks. The stacks are shown in your status bar where you can see how many stacks are on you and when the next stack is going to run out.
These stacks are individual and non-refreshing which means once a stack is applied it will run out after its duration even when new stacks are applied during that time.
Each stack lasts for 14 seconds and the cooldown of the ability is 3 seconds.
Each stack that's currently active on an enemy adds 30 bonus damage to the Quill Spray.
Which means if you only use Quill Spray you can do a maximum of (14 / 3) * 30 bonus damage which is 120 bonus damage.
But that's where Bristleback's passive comes into play.
Bristleback: Bristleback takes less damage if hit on the sides or rear. If Bristleback takes 250 damage from the rear, he releases a Quill Spray of the current level.
This means if people hit you enough you can even get additional stacks to the ones you're casting yourself. These stacks also count towards the bonus damage. There is a cap to the bonus damage though which is 400 at most.
A bit of theorycrafting:
At level 4 Quill Spray you need 11 stacks to reach the 400 bonus damage which means you need 7 additional stacks from your passive (you get 4 stacks from always using your Quill Spray). Which is 2000 Damage you need to take in the course of 14 seconds (otherwise the first stack will run out again).
TLDR: Max Bonus damage is 400 per Quill Spray but this is only achieveable if you take a lot of damage from the rear and you skilled your passive called Bristleback.