Easy peasy. Hulk gets out of control at some point in Avengers 2, going off mission or causing some serious damage. The World Council (the shadowy figures that bossed Fury around) decide Hulk is too dangerous to stay so they send him away, perhaps with Fury's knowledge/help. If Thanos is the villain, the team may end up in space. Fury could ditch him while they're out there and lie about it to the rest. Planet Hulk happens, then he comes back for the council.