In order to reconcile the Geth-Quarian War and gain the most out of the situation you need to score at least 5 points on the following list
Mass Effect 2 Decisions
- +2 for Tali is exonerated at her trial (eg NOT exiled)
- +2 for destroyed the heretics
- +1 for resolving the Tali/Legon conflict with a renegade or paragon solution
As noted in this answer, if you did not import a game and started a new mission in ME3 you will have saved Tali, Tali will have been exiled by her people and Legion was never activated and thus the heretic mission would have have appeared. Ergo this avenue is closed if you started a new game file.
Mass Effect 3 Decisions
- +1 for Rannoch: Destroy the Geth Squadron on Rannoch (aka fighter base)
- +1 for Rannoch: Save the Admiral (not his men, the admiral)
You must also complete Legion's Mission in ME3 or the peace options will not be available. You need a high reputation to get the Paragon/Renegade options to appear.
If you rewrote the heretics in ME2 instead of destroying them, and still manage to achieve peace, you'll get more war assets but there will be less Quarians.