Easy peasy. Hulk gets out of control at some point in Avengers 2, going off mission or causing some serious damage. The World Council (the shadowy figures that bossed Fury around) decide Hulk is too dangerous to stay so they send him away, perhaps with Fury's knowledge/help. If Thanos is the villain, the team may end up in space. Fury could ditch him while they're out there and lie about it to the rest. Planet Hulk happens, then he comes back for the council.
Want to make extra sure Hulk feels betrayed? Have Fury ask Tony to design a plan to get rid of Hulk as a fail-safe in case he ever got out of control, then have Fury use the plan behind Tony's back. Banner/Tony would see Stark tech and think everyone else knew.