Oh I agree with you in spirit. The fact that the state can pluck you and make you fight in a war is ridiculous (except maybe in the extreme case of a mass homeland invasion but even then idk) and is very akin to slavery. Unfortunately, all of this reasoning has been tried and failed on the highest levels of the court. I just don't see that changing with all the precedent that exists.
U.S. Constitution - Article 1 Section 8
Seems to me that's all they need to justify it.