Biotech Values

[iwfal]

Setting the p value discussion on the right track (but note that there are other factors important to patients that affect N):

Recently it has become common to make the following logical argument:

a) the threshold at which a p-value is called stat sig is arbitrary (and was picked by Fisher)

b) Therefore we can change what we call 'stat sig' if there is need.


Although that is what Fisher (the father of statistics but also a renowned god-complex individual) would probably like us to believe it is mathematically incorrect - because in reality the aggregate p value is driven by the base rate of false hypotheses (in the case of biotech that is the number of INDs for which the original drug/protocol did not work (generally meaning Adverse Events worse than benefit))

See here for a good, first order, numerical example. But note that he assumes only 90% Base Rate of bad drug/protocol at IND. The reality is that it is probably closer to 99% either fail or have substantive changes in their protocol after IND in order to provide measurable overall benefit.

So lets run some numbers:

a) Assume, generously, 90% of drugs in their original IND protocol are actually meaningfully bad for QOL, and another 9% are close to 'neutral'. And 1% are good in their original protocol.

b) If approval is one trial p<0.05 - then when you take your next drug you should expect 2.5x greater chance of harm than of benefit.

c) If approval is one trial p<0.15 - then when you take your next drug you should expect 8x greater chance of harm than of benefit.

d) If approval is per FDA guidelines (2 trials p<0.05) - then when you take your next drug you should expect 14x greater chance of benefit than harm.

I.e. the knee in the curve is probably somewhere between 1 and 2 stat sig (0.05) trials.


So with this base rate approving drugs on lesser evidence because some patient set is in more need is very likely to cause those "in need" patients additional harm.

The only way around this is if there are some classes of drugs for which the base rate of worthlessness/harm is a lot lower. Clearly not all diseases/drug-classes are the same (e.g. psychiatry is clearly worse, cancer better - Nature Biotech did a survey a while ago in this although I no longer remember the details).

Lets assume that a particular disease/drug class has only a 50% rate of harm, 30% neutral, and 20% beneficial (other than me-toos I can't think of any drug/diseases that would be better than this).

1) If approval is 1 ITT trial p<0.05 on prespec primary endpoint - then when you take a drug of this class you should expect 13x greater chance of benefit than harm.

2) If approval is 1 ITT trial of p<0.15 on prespec primary endpoint - then when you take a drug of this class you should expect 4x better chance of benefit than harm.

3) If approval is 1 ITT trial of p<0.3 on prespec primary endpoint - then when you take a drug of this class you should expect 50:50 chance of benefit vs harm.

And note that there probably are diseases/drug-class pairs where the base rate approximates this. E.g. monogenic diseases where the disease protein is fairly static and the drug makes a meaningful change in protein levels.

But, finally, as I noted in the beginning - there are other factors besides p value that influence the necessary trial size. In particular small trials cannot detect uncommon but extremely severe adverse events, and because small n = huge uncertainty it is, essentially, impossible to figure out which drug is meaningfully best.