If all you're watching is Netflix or Hulu, I think the Chromecast is faster than the Firestick. Navigating on your phone and casting it is faster than the Firestick. And it's not even really close.
Which is why I'm disappointed still with the Chromecast. Newer TVs these days come with Netflix and Hulu preinstalled. The Chromecast sucks at everything else, especially Sling.