I think that there is a slight process of "guilt by association" going on, people still remember Falwell and Robertson post 9/11 speeches when they blamed the calamity on homosexual and other "sinful" living of some of the population. Their speeches, then, were indeed hateful, so hateful that they had to come up with retractment.