I'd suggest going back. I go down there about once a month or so to go out and never see the things you claim to have seen.
People love beating up on the FQ. There are troublemakers everywhere. But from my experience on Bourbon, which is pretty extensive, it's rare that I don't have a good time and usually the crowds are fine.