I agree that if collateral damage is avoidable, it should be avoided. Depends on the danger presented by the terrorist. If he has a backpack full of anthrax or a portable nuclear device, taking him by other methods would be preferred, but having a way to take him down from stand-off distance, shouldn't we have the option in our arsenal?
Let me ask you a question. If the second airliner that was headed for the second tower of the WTC could have been crippled or downed, would it have been been better to shoot it down once its hostile intent was clear or let it take out 1500 people in the building?
These are all hypotheticals and drones would not be the first weapon of choice, but under closely defined circumstances they probably have a place in our quiver. The problem is that anyone can imagine abuse of it, just like automatic weapons, so it is easy to fearmonger them to people that are afraid of the government in the first place.