To help frame the debate a little better, I did some quick back of the envelope cypherin' based on the 2009 federal tax brackets (the difference between 2009 and 2010 is probably minimal) and I estimated that the income needed to be paying 40K to the feds is between 165,000-185,000, depending on filing status.
Of course, that's assuming that the entirety of that income is such that would be taxed at standard rates (like a paycheck) and doesn't take into account any of the myriad credits and deductions. Realistically, you could be making 250,000+ before you have to honest to goodness cough up 40K+ to Uncle Sam.
Assuming you took the super easy route and didn't claim anything, that income puts you in the top 5% of income.
Of course, if you live in San Francisco, you'll need every dime of it and you can still claim to being "middle class" but being "middle class" doesn't necessarily make you an "average" earner.