The passing lane is for passing. I don't care how fast you are going, unless you are passing someone move to the right lane. If you don't you're doing it wrong.
But since it's illegal to drive above the speed limit, by design, it would mean people driving the speed limit would be the ones getting in the left lane to pass slower vehicles.
If a car driving the speed limit is slowing you down, the driver is actually doing you a favor, and helping you avoid a ticket.