One way it can happen is if you sweep your cursor across the Favorites page and accidentally click on one of the "Remove" buttons. That's happened to me before. There's no "are you sure you want to remove it?" message so you are not notified if you remove a board.
Another way is if somebody maliciously posts a URL that causes you to remove a board if you click on it.