We're writing him off because he's already failed before. This isn't a too soon situation. It was a gift wrapped playoff caliber team that he knew the ins and outs of. It lacked a franchise QB but it had 2 QB's Payton went 5-2 with and 4-1 with.
He completely mishandled the 5-2 QB to the point where I'm sure the relationship is broken despite him being a leader in the locker room. He moved the 4-1 QB to TE despite him having under 40 career catches. He goes with a backup journeyman over both and loses the season with him in one of the lowest scoring offenses in the 2nd half of the season.
His game management has been epically abysmal and cost us a playoff spot in a terrible division. He's the one that wants to go conservative on offense, not Carmichael. He's the one that wanted Dalton. We can improve on Carmichael and get a new QB but at the end of the day it's still dressing up a pig. Allen is an ultra conservative coach, it'll take a complete makeover of his coaching style for an explosive offense to be in place here because it simply doesn't mesh with the kind of team he wants.