Replies to post #153867 on Biotech Values

[mcbio]

I would disagree with this.

I would think that such small numbers increase your chance of seeing a result that is not representative of the "true" outcome as would be revealed by a large trial. In a sense, the result here is driven by the 4 control patients... if by chance they're a tightly knit group of 4 with very similar (but somewhat atypical) outcomes (ie. low st. dev.), then they could be the driving reason for the statistical significance.

Put it this way: if these results were stat sig from 4 patients on drug and 12 on the control, there would be significant skepticism (rightly in my mind) about a company touting a stat sig result. I don't see why having 12 patients on drug and 4 on control makes it any more comforting.

I thought p-values took into account small numbers in a trial and there would thus be a need for more robust results to attain stat sig results. And I thought the definition of stat sig means that results, whether they occur in a few patients or in a large number, reflect a pattern and are not just due to chance. I'll defer to others on here who are much more knowledgeable than me on statistical matters (admittedly not my forte) but that's my impression.

[iwfal]

In a sense, the result here is driven by the 4 control patients... if by chance they're a tightly knit group of 4 with very similar (but somewhat atypical) outcomes (ie. low st. dev.), then they could be the driving reason for the statistical significance.

Finally figured out what bothered me about this sentence. It is incorrect in the sense that it matters not at all whether the 4 are 'tightly knit' since, given that they were chosen randomly then 95% of the time they would be no more tightly knit than the remaining 12. I realize this is counter-intuitive, but you'll just have to trust me (unless you try the intuition building suggestion below). And the odds of you picking 4 patients who happened to be skewed from the remaining 12 is 0.05 (or whatever the p value was).

You are making the same mistake as biomaven made - that the distribution matters. It doesn't - any possible distribution is enveloped in the p value calculation. I would strongly recommend to everyone that they build a spreadsheet and play with some of the simpler non-parametric tests - e.g. it is possible to gin up an MC spreadsheet on the statistics used for ORR p value calculation in less than an hour. And no matter what distribution you assume for responses (bimodal, tri-modal, heavily skewed, flat, ...) as long as the drug is a null then the p value will get less than p=0.05 only about 5% of the time in an MC run of thousands. {Note that if the FDA allowed distribution assumptive calculations then for ORR (or HCV viral levels or ...) everyone would probably do a difference of the means test. But instead they do a responder test because it is non-parametric.}