P-values: a (not very) brief discussion . . . .
Long-time long (and lurker), first-time poster, owing thanks to too many of you to mention. As a small and hopefully helpful contribution, am chiming in with a very amateur clarification of p-values, in an attempt to convince myself that I do remember something from grad school. Am doing this because p-values get mentioned quite a bit here, and sometimes it seems as though there might be some confusion around them. I'll begin with some terminology and concepts, and try to tie it all together at the end. Smarter people, feel free to jump in.
P-values are calculated for various statistics, but within the context of an experiment such as a clinical trial, p-values indicate the degree to which we can have confidence that an observed change in a dependent variable (ORR, days of survival) can be attributed to the independent variable (a drug such as Bavi). Scientific studies often consider a p-value of .05 or smaller to be the holy grail, though there is really nothing magical or sacred about .05. Depending on what's being measured/studied, and whatever's at stake, a larger or smaller p-value might be more appropriate.
NULL HYPOTHESIS: In a clinical trial (since that's what we're talking about on this board), suppose the researcher has a "hypothesis" that the independent variable (a drug such as Bavi) has a positive effect on a dependent variable (ORR, days of survival, etc.). The treatment group will get Bavi, and suppose that the control group gets a placebo. The "null hypothesis" is that, in the end, there will be no meaningful (or "statistically significant") difference in the dependent variable for the two groups (we'll get back to null hypothesis later).
EFFECT SIZE: Suppose that the dependent variable is the mean length of survival after the start of treatment with Bavi, measured in months. Also suppose that the trial's observed mean survival turns out to be 4.0 for the control group, and 11.9 for the Bavi group (this is just for discussion, not intended to be realistic). The difference between those two means, 7.9 months, is known as the "effect size."
INFLUENCE OF SAMPLE SIZE ON P-VALUES: A p-value is influenced by sample size and effect size. First, let's talk about p-values and sample size. Using our example above (and holding everything except sample size constant), a p-value for the effect size of 7.9 months will be smaller with a sample of 1000 than it would with a sample size of 25. In other words, for a given effect size, statistical confidence gets higher as sample size gets higher. This makes sense when you think about it.
INFLUENCE OF EFFECT SIZE ON P-VALUES: Now for p-values and effect size . . . . Using the same clinical trial example (and holding everything except effect size constant), a p-value for an effect size of 7.9 months will be smaller than it would with an effect size of 1 month. Statistical confidence gets higher as the effect size gets larger. This also makes sense.
Sample size and effect size work together to influence p-values:
*A small sample size needs a big effect size.
*A small effect size needs a big sample size.
*Big sample size and big effect size leads to smaller p-value.
*Small sample size and small effect size means "back to the drawing board".
DRUMROLL, PLEASE: The best stats professor I ever had described p-values this way, and it has stuck with me for 20 years . . . . If you could repeat the exact same experiment 100 times, the p-value indicates the number of times that you would expect to observe an effect size larger than the one you already observed (7.9 months), if the null hypothesis is true (i.e., no meaningful difference in outcomes for the treatment group vs. control group). In plain English, using our fictional clinical trial as an example, suppose that we observe an effect size of 7.9 months, and a p-value of .03. The .03 value means that if we could re-run the identical experiment 100 times, we would expect an effect size bigger than 7.9 months only 3 times out of the 100 repeated trials.
Hope this wasn't too scrambled, and that others can build on it. Again, thanks to all.