I can absolutely relate to this. I lived in New Orleans right up to college (N.O. East, to be exact) and I don't have a very discernible accent. I HAVE had people in northern Virginia/Maryland/Pennsylvania claim they can tell I'm from N.O. or tell I have some kind of accent, and I've even had people at LSU claim they can tell I'm from N.O., but I can't really tell, and I know for a fact I don't have a yat accent.
Where's my point? People who grew up around the city, but not necessarily in it, are more likely to have a "New Orleans" accent. There are many reasons for this phenomenon, but it boils down to the fact that a lot of these accents are simply disappearing. Since I'm a 1st generation New Orleanian (like Peyton and Eli), I think my accent is likely to be influenced by my parents and friends. Well, most of my fiends have my "accent" and growing up, I didn't encounter that many "yat" accents. In my experience, most of the people with heavy accents lived outside of Orleans parish. St. Bernard, St. Tammany, and Jefferson parishes seem to be the actual home to these accents. Post-Katrina, I think this trend has greatly accelerated, and I'm afraid you really have to look to the older folks to find that many left in the city.
Now, with all of that said, I think I'd be very likely to sort of "pick up" or "put on" portions of the accent if you put me in the middle of a conversation with people that have it and give me a few beers. Like I said, I'm sure I have a little bit of one, but no more than the average New Orleans native, which isn't much. And that doesn't make me any less New Orleans. The real New Orleanians know this; the posers are the ones who claim otherwise.