Reason, the Saints job in 2006 was incredibly unattractive. Payton wanted to go to Green Bay, but the Packers picked McCarthy. As far as 2023, I think the Denver job was attractive because of ownership, some talent, and Denver's status as a storied franchise.
The city was what was unattractive in 2006. Nobody knew if the population would return or what the condition of education, housing, etc. would be. The job itself wasn't too bad.