Easy. It is not a contract, and the law that creates SPAs specifically gives the FDA the ability to do what they did:
US Code of Federal Regulations. 21 USC 355:
So given this one must argue that the decision was made improperly, and not just that it is wrong, that is a tough road to hoe. And certainly there is no prima-facie argument that the FDA breached a contract.