This is an important part of the discussion that should be had, in a good way. Many of us don't understand unconscious bias, of which you should never feel guilty (it's environmentally installed and behaviorally reinforced.
But we also all have a duty to do a little more "standing in the shoes of the other person" because we don't realize the effects of even largely well-meant behavior. Part of the justice and equality we all deep down want to achieve will require us to undo some of this wiring so many of us have.