Brees finished second in the MVP voting four times--2006, 2009, 2011, and 2018.
I don't think the voting was affected by Brees playing for a "small market" team. In those years, the Saints were still a "feel-good" story because of Katrina and were a team people wanted to watch on television because of how explosive our offense was.
In 2011 and 2018, there were other quarterbacks who had incredible years--Rodgers in 2011 and Mahomes in 2018.
In 2009, Indianapolis, which had lost talent on offense, won its first 14 games (though we won our first 13), and the Peyton Manning brand prevailed.
But Brees's problem was that he always had to battle the perception that as an athlete and quarterback he was not quite on the same level as Manning and Rodgers. The reasons were that he was not a first-round pick, his height, his lacking the absurd arm talent of Rodgers and Mahomes, and in New Orleans Payton receiving the credit for the team's offense whereas in Indianapolis Manning was considered the de facto coordinator and received the credit.