Now that the game has been out for a few weeks, it looks like the general consensus is that the Cool Bonus is based on how many hits you took during an Act.
Users in this thread all suggest that this is the case:
The less you get hit the higher it is, getting killed instantly scores you a 0.
The highest cool bonus you can get is 10,000 points and I think you lose 1,000 every time you get hit.
It was found out in the first few days of the console release back in mid-August. You can easily test it yourself; just finish Green Hill Act 1 without getting hit once. Then restart, go back and grab some rings, get hit by anything in the level or the boss once, then finish it.
This is also backed up on the Sonic Mania wiki page:
Upon completing an Act, the player is graded based on their time, and their score; an additional "Cool Bonus" is added if the player clears an Act without sustaining damage.