There's just a greater chance for abuse. Some nameless desk jockey flying a drone around that people may or may not see...
It goes back to how much power you are willing to give to the state to ensure your safety. And how much abuse of that power you're willing to tolerate.