It's mostly perception. An offensive lineman could be having the game of his life but if he holds on a 80 yard bomb for a TD, that is what the masses will remember.
Out of sight, out of mind. For every miss tackle we see (which you graciously pointed out isn't many) there are ten plays we don't see where he saved TDs. It is what it is with some Saints fans.