I believe part of the motivation on the part of those who want to have a banning feature is to be able to say publicly that which cannot be rebutted publicly.
If this is not the case there would be no objection to a private board being visible only to those invited to post there.
My suggestion is, I believe, the reverse of the current test. Rather than allow the mods to ban on a so-called private board allow them to invite. In other words they have to individually select each member allowed to post on that board. Those not listed don't see it.
Obviously this would constitute a new kind of board here and would have to be selected that way when the board was created.