Is there any way to invite specific people to your game if they are not listed on the social tab (Friend/Recent lists)?
Based on the available commands and key bindings there doesn't seem to be a way to do this.
There are some command line switches referenced here that might have allowed something like that but most of them appear to have been removed from the retail version.
If you are talking to someone in one of the chat rooms in game you can right click their name, you will get a drop down list, with the option to invite to party. They can also do the same to you to send you an invite to join theirs or your game.