Sorry to hear about your problems. Is there no way to file for full custody? Seems if you have the best interest of the kids, and the means to take care of the. You should have full custody of them.
Best of luck man. Being a father of 2 young ones I miss them when I am a work, so I can barely understand how much this must all be on you.