For the most part, that is a correct way of stating it. A patent that WILFULLY omits prior art can be invalidated when challenged, irrespective of whether inclusion of the prior art actually would have led to the rejection of any given claim. To be a basis for invalidating such an issued patent, or rejection of a patent application, however, the omission must be intentional.