The Saints were a sacrifice to appease the class action lawsuit. It was a personnel vendetta that Roger Goodell was visualizing against the Saints, and specifically Sean Payton. It was never impartial to begin with which is why he ended up recusing himself for Tagliabue.
The problem is the majority of NFL fans never had the attention span to follow the story beyond the initial headlines. Also, hearing the reality of the scandal didn't jive with the whole "derp, the Saints are good so **** them!" mindset many had developed. You don't get haters until you're good, after all.