From what I've read, he hasn't been all that controversial. Biographies of him note that he approved Sarepta's drug for use against muscular dystrophy that had failed its stage III trials. There was a reaction to that.
But there is no question that the knows a great deal about vaccines, and that's why RFK Jr wanted to be rid of him.