Replies to post #128882 on Biotech Values

[DewDiligence]

Re: Fallacy of ignoring program-survival bias

…you have a small biotech that had what appear to be good Phase II results -- which are hardly conclusive but certainly encouraging -- and then you have enrollment data, Phase III trial parameters, and a good (if not great) sense of likely survival for the control arm. Then you assume that the drug is performing consistent with its Phase II performance, and you get modeling outcomes that are consistent with the trial progress to date (e.g. with guidance from the company on when data should be available or when an interim trigger will be met or simply vis-a-vis how much time has gone by). In such a case, I think you have reasons to become bullish…

The fallacy in the above is the text I highlighted in bold. Because of program-survival bias, investors should absolutely not assume that phase-3 performance will be consistent with phase-2.

The modeling error from neglecting to take into account program-survival bias is essentially what iwfal’s exercise was about. Yet, over and over, many biotech investors continue to neglect program-survival bias to their detriment. Regards, Dew

[iwfal]

But there seems to be a pat attitude on this board that this particular DD tool is flatly invalid -- "the numbers game never works" --

While I agree that there are some who say that the numbers game (modelling enrollment and comparing to "comparable" historical trials and then inferring it is going long due to excess treatment survival) I am more undecided. But that said, I have yet to find an example where the trial went so much longer than historicals that it significantly reduced risk of investment.

Take, for example, your recent CSLN example - you have two trials (or studies) with protocols that differ at least somewhat from each other and from the CSLN protocol and have a WILD disparity in Median PFS. Statistics says that two samples (of medians) that far different implies wild uncertainty in what the placebo median is likely to be for your trial. The p value you get from that kind of analysis is very poor - like 0.5 - until the blended survival of the CSLN trial gets to somewhere over 100 months or so. Yes, you can better match the protocols - but the uncertainty in that too is very large (as demonstrated by the large numbers case-matched trials that failed in ph iii).

Even ONTY, with 5 or 6 arguably comparable trials, has significant disparity in medians - at best any such inferring would have to be considered to have a p value of, at best 0.2 or worse. And worse than that if you note that the more comparable trials tend to have the longer medians.

Finally note that each of the above WAG p values is before correcting for PSB (or if it makes you feel better to think of it in more conventionally - the Base Rate Fallacy). After such correction I think it can pretty definitely be said that the analysis of blended survival going long isn't adding much.


The closest I have yet to see of a trial looking long is AVEO - but they'll have to go a bit longer before I'd say it meaningfully increases the odds of success.