I have found two viable solutions:
Bi-directional Pistonless Counter
Signal strength memory

The second one is faster, more compact and outputs directly as signal strength while the other has 15 different binary outputs.

I have found a viable solution: Signal strength memory

It does exactly what the description says, except the "button" has to be exactly 2 ticks or it will increase/decrease too much. A simple pulse shortener should sort that out.

Here is what it looks like as well as a layer breakdown:

I found a partial solution to my problem:
Sequential Counter

The issue here is that this design uses items with droppers and hoppers, which makes it quite slow. It can also only count up. I would still love it if someone has a better plan.

I have found two viable solutions:
Bi-directional Pistonless Counter
Signal strength memory

The second one is faster, more compact and outputs directly as signal strength while the other has 15 different binary outputs.