It's only libel if it's not true. It has nothing to do with whether or not evidence is offered at the time of the allegation.
But, seriously, you're right. We shouldn't all rush to judgment based on countless first-hand reports and material evidence over many years now. I mean, look what happened with O.J. Simpson. We all thought he was guilty--and it turns out he didn't do it!