If you have 5 people in a party and create a custom game this happens.
What you need to do is cancel the party and then start a custom game, select 4v4 mode and add people manually using the Add Player button (Near Add AI button).
Note: not verified, i do not have any friends online now.
I will describe the steps to create a 4v4 custom game:
Select Multi Player at the top left part of the SC2 screen
Select Create Custom Game on the bottom right part of the SC2 screen
Select a map for 8 players (for example Lava Flow)
At the right side of the screen, select 4v4 from a drop box
Add players as you like and play
This works fine for me and i have never had any problems.