This attitude would get you banned here and should get you thrown out of any stadium anywhere. It is not a free for all fight or cursing match. Anyone can attend a game anytime in any stadium and should not have to worry about some drunken idiot wanting to get in a fight or cursing at you.
And I agree, the op should have said something to the idiot Saints fans acting like that. I can tell you that it has happened in my section.
By the way, I bring my son to every game and since when were kids banned from attending a football game because of some idiots.