People don't leave Darien or Greenwich Connecticut because of taxes. Connecticut is a very expensive state to live in.
In Darien and Greenwich, people's wealth ticks up and down via their portfolios, and those in the nested yacht set (as well as the mere "very wealthy" people) have only a vague idea of their actual net worth.
By all means though, let's give them a tax cut. They worked so hard for their lucre.