As a man who enjoys the company of women and who also likes Pumpkin Spice, I've wrestled with this apparent incongruity. But I think this summed it up best.
Pumpkin Spice isn't the end-all flavor. It's fine. But it only comes around during the undisputed best time of year by every objective metric. And that association makes it awesome.