I have been wondering this for a while now. I have played dozens of games where people want to report me because I was playing poorly, but how many reports does it take to get into the Tribunal?
There is no set number of reports that automatically determines whether or not you will come before the Tribunal.
In section 14 of the Tribunal FAQ, you'll find the information that addresses your question.
As you can see, there are a number of factors that go into Tribunal reports, and this is not the full list, as Riot does not disclose that. The linked FAQ should be able to answer any other questions that you have about the Tribunal process.
I can't tell you exact number but this is how it seems to work. Let's say you were mad and started feeding so 3 members of your team and 2 of the other reported you. That wouldn't do much, but get you headed on your way to the tribunal. Now if you did it again and people reported you again then you'd probably be sent to the tribunal where the user base decides on if you should be punished or not. Depending on the chat logs and what the users said that reported you. You could receive a temp ban, perma ban, or just a warning. If this keeps going it will lead to a perma ban.
One exception to the reporting system is the "Bad Player/Noob" Report option. Riot has said that they will not punish anyone for being bad. Reporting anyone with that option does nothing. So if they are reporting you for being bad, don't worry about account bans. But do try to get better as it's more fun if you are winning. Good luck on the rift!