The only ones who are agressive are yellowjackets. The sound of a lawn mower will send them into attack mode.
Red wasps are gentle. They won't attack unless you physically threaten their nest. They are also known as a
gardeners best friend. Leave them be. They prey on insects that damage yours plants