Looking strictly at the QB position (ignoring rings, teams, schemes, etc.) Brees pretty easily takes the #1 spot and the only reason anybody could disagree is because everybody wants rings to count as a QB achievement instead of a team achievement.
How? The only argument is rings. He'll only have the most yards and TDs by virtue of attrition (and those records may not last with a 17 game season). He's never been extraordinarily accurate. If you evaluate the QB position the same way we evaluate every other position in football then Brady simply isn't the best. Manning and Brees were both definitively better QBs.
Randy Moss. Aaron Hernandez. Gronk. Ben Watson when he was still in his prime. Wes Welker. Julian Edelman. Brady had plenty of really, really good receiving options in NE. And in TB his top three receivers would all be WR1 on basically every other roster in the NFL.
The best receivers Brees ever threw to were Marques Colston and Jimmy Graham until Michael Thomas showed up.