The problem lies with the idiot who asked you that question. Not with your non-verbal answer and others interpretation of that answer.