Personally, if the person they are replying to was making a joke, I leave it. If they weren't, I generally remove.