When it comes to folks like Castro, Kim Il-Jung, etc...there's a lot that goes on we don't know about, particularly when it comes to their personal lives. Take a good look at Pol Pot's life. Or Saddam's life (prior to the UN embargo after the first Persian Gulf War). Or Nicolae Ceausescu---whose son was a notorious playboy.
Maintaining a "cult of personality" requires the leader to ensure every effort is spent hiding every detail about your personal life from the masses (in order to maintain the illusion you are "really representing the average citizen").
Many oppressors give the outside world the impression they are poor, when in reality they are extremely wealthy.