When you have a fast car it does not mean you can get anywhere in the world in the same amount of time. It depends on many factors such as traffic, road condition, or how far the place is. It is the same for the Internet. When you connect somewhere, you never have a direct connection (unless you threw a fibre to the server). You connect through several intermediate points and your connection quality depends on each of them.
Network connection for video streaming is quite different from your average video game. Video streams are based on UDP protocol, which is faster than TCP, that is common in games. Your experience with video streaming is based on whether you can pull data fast enough and the server can give it in the same rate. When you deal with ping, you need that all of your packets reach the destination and return to you fast. The bandwidth is not so important here.
To find out why you have the problem would require you to learn about networking quite a lot. But even then you rarely can do something about it except for change ISP maybe. You can begin the investigation by pinging the problem server and a good one over a long time. With that you can find what's happening when you see a spike: either it's a packet loss or just delay. Then you can use tracert to find out at which point in route the problem occurs. If it the first step, then the problem is in your router if you use it or ISP otherwise. If it is the last step, then the problem is with the target server. If it is somewhere in the middle then god help you. Even if you change ISP there is no guarantee the problem will be solved. You might try a server in a different region if you can.
The block above is for a fibre connection. If you connect to the internet with a wireless connection then there could be a freaking lot more different issues. Wireless connection is not an option if you want a good ping.
