Agreed. This is why I never get people who want to just blow it up. It very occasionally works but most of the time teams who rebound from top 5 picks who are teams that just had a terrible year due to injuries and already had a solid base roster. On the other hand there are teams like the Bears, Panthers, Jets, and Browns that have been picking at the top of the draft for years and have never gone anywhere despite taking several QBs that were supposed to be the best or one of the best in their classes.
Blowing it up is never a guarantee that you build back and more times than not a team stays blown up for a decade or two. I'd much prefer to have a mediocre team with a solid base roster that is one or two good drafts or one great QB away from contending than blow it up and then take 3 or 4 years to hope to have a mediocre team again.