I don't see how protecting the face is a "learned" anxiety rather than an inherent one. That being said, it's not inherent in me to go for the groin kick, so I need to "learn" that because I don't know karate and I don't have a CCW.
Sounds like he is attempting to gaslight you into believing that your face doesn't need to be protected and you are being irrational for wanting to protect your face from random punches on the streets of New York.