I totally agree with you. This has been my philosophy as well. Don’t reward a team for winning a crappy division. Getting into the playoffs and hanging that division banner should be enough.
It's always bothered me. Has a 7-9 or 8-8 division winner EVER won a playoff game against a team with a better record? OTOH, I can think of 2 or 3 wild card 9-7 or 10-6 teams who ran the table to the Superbowl, and several more who made an appearance in the championship game.
Besides addressing basic fairness, and fixing the contradiction of "better records = higher seed, except for weak division winners", this would go a LONG way toward eliminating final weekend "coast" games. As the rules are currently, an 11-5 or 12-4 team could be the 2nd best team in their division, and the 2nd best team in the conference, but "locked in" to the #5 seed with 1-3 games remaining in the season; so they start sitting injured and star players. I'll bet that doesn't happen anymore if they have a shot at the #2 seed as a wild card team, and home games against lower seeds. So it would be fair to the teams, and more exciting and interesting for the fans.