The problem with this is is totally against canon, as none of the Istari (wizards) arrive until 1000 years into the Third Age (which begans with the initial defeat of Sauron by Isildur / Elrond)
And it could be that at least some Elves took an oath to not return to Valinor until Gil-Galad granted them leave.