It's detailed in the OP how well he did with the Panthers his first year there and how Matt Rhule may have been the issue. Considering QB's always have career years under Brady (except 2021), its more probable Matt Rhule was the issue in 2021.
Also, Allen is playing very similar this season to how he did once Brady took over the offense last season. Allen was not playing well at all last season until Brady took over.