Ask everyone to open the ports in their router, GT5 Prologue reports:
Gran Turismo 5 Prologue online uses the following ports for communication with game servers.
If you experience problems connecting to online races, you may want to check the status of these ports in your router or firewall to see if they are being blocked or filtered. Your router instruction manual or manufacturer will be able to assist you with opening these ports up.
TCP Ports: 80, 443, 5223, and 10070 - 10080
UDP Ports: 3478, 3479, 3658, 10070, and 50100
Note: Make sure to enable these TCP/UDP ports in BOTH directions and are entered into the TCP and UDP port fields in your router / modem.
— http://eu.gran-turismo.com/gb/support/d4219c13.html
Googling reports that the ports for GT5 itself are:
TCP: 80, 443, 5223
UDP: 3478, 3479, 3658, 5658
So, I would suggest to just open the ports for Prologue and add TCP 5223 and UDP 5658 to be sure.
See How are Gran Turismo online games physically hosted? for more details on how it works; in short, the best connection is chosen as the host.