Power, effect size and clinical significance

Transcript

RyanHere's a story from an open-access paper by Sullivan and Feinn. The Physicians' Health Study followed more than 22,000 people for an average of five years, to see whether aspirin prevents heart attacks.

SoheeAnd the result was highly statistically significant. A p-value smaller than one in a hundred thousand. The study was stopped early, and aspirin was recommended for general prevention.

RyanBut the effect size was very small. A risk difference of 0.77 percent. The paper says many people were advised to take aspirin who would not benefit, yet were at risk for side effects. And the recommendation has since been modified.

SoheeSo a result can be extremely significant, and still small. That's today's episode.

RyanThis is EPPP Prep, the podcast. I'm Ryan.

SoheeI'm Sohee. We're covering effect size, power, confidence intervals, and the difference between statistical and clinical significance. The exam loves the space between those ideas.

RyanLet's start with the problem. Sullivan and Feinn say a p-value can tell you whether an effect exists, but it won't reveal the size of the effect.

SoheeAnd here's why. Statistical significance depends on both sample size and effect size. With a big enough sample, a statistical test will almost always find a significant difference, unless the effect is exactly zero.

RyanTheir example: with a sample of ten thousand, a significant p-value is likely even when the difference between groups is negligible. They put it memorably. Sometimes a statistically significant result means only that a huge sample size was used.

SoheeEffect size, by contrast, is independent of sample size. It tells you how big the difference is.

RyanThe simplest version is the absolute effect size, the raw difference between group means. Useful when the measure has intrinsic meaning, like hours of sleep.

SoheeBut what if the measure is a rating scale? Is a 0.4 point change on a five-point scale a lot? That's where standardized effect sizes come in.

RyanFor two groups, you take the difference between the means and express it in standard deviation units. Cohen's d is the classic example.

SoheeAnd Cohen offered benchmarks. Small is about 0.2. Medium about 0.5. Large is 0.8 or more.

RyanCohen described a medium effect as visible to the naked eye of a careful observer. I love that.

SoheeAnd to make 0.8 concrete: the paper says that at that size, the average person in one group scores higher than 79 percent of the other group.

RyanBecause standardized effect sizes put different studies on one scale, they're commonly used in meta-analyses. Hold that thought for a later episode.

SoheeAnd effect sizes come in two families, the paper explains: differences between groups, like Cohen's d, and measures of association between variables.

RyanAn association measure you already know is r squared. In that aspirin study, r squared was .001, which the paper calls an extremely small effect size. Significant, and tiny.

SoheeThe paper's advice is to report both: the effect size and the p-value. Together they tell the whole story.

RyanNow, power. Sullivan and Feinn define statistical power as the probability that your study will find a statistically significant difference when an actual difference does exist.

SoheeAnd from our hypothesis testing episode: power is one minus beta, where beta is the probability of a Type II error, missing a real effect.

RyanLike significance, power depends on effect size and sample size. A large effect can be detected with a smaller sample. A smaller effect needs a larger sample.

SoheeThe paper lists ways to increase power. Use a more potent intervention with a bigger effect. Increase the sample size. Reduce measurement error with highly valid outcome measures. And raise the alpha level, but only if a Type I error is highly unlikely.

RyanThat list is pure exam material. And the reverse is worth knowing too. Shrinking the sample, or using a noisy measure, lowers power.

SoheeWhen do you calculate power? Before the study. The paper says post hoc power calculations have limited value, because they wrongly assume the sample effect size is the population effect size.

RyanAnd where does the common target of 80 percent power come from? The paper says Cohen chose a beta of 0.2, reasoning that a Type I error is more serious than a Type II error, so beta could be four times alpha. Four times .05 is .20.

SoheeThe paper even works an example. A pilot study finds a p-value of .06, which looks non-significant. But the effect size is 0.5, a medium effect. To reach 80 percent power, they'd need roughly sixty people in each group.

RyanSo that pilot didn't show the effect was absent. It was too small to be confident either way.

SoheeThe Mishra article on statistical methods makes the same point: if the sample is smaller than needed, the study will be underpowered to detect the difference, so a real difference can come out statistically insignificant.

RyanOkay, pause and think. A small pilot study of a new group therapy gets a p-value of .09 and a medium-to-large effect size. The authors conclude the therapy doesn't work. What's wrong with that?

SoheeNot rejecting the null isn't proof of no effect. With a small sample and a sizable effect, low power is a real possibility, and the risk is a Type II error. The fair conclusion is that a larger, adequately powered study is needed.

RyanNow confidence intervals, which tell you about precision. The OpenStax statistics textbook gives the wording: we estimate with 95 percent confidence that the true population mean is between two values.

SoheeAnd it's careful about what 95 percent means. It's most accurate to say the confidence level is the percent of confidence intervals that would contain the true value if we took repeated samples.

RyanTwo rules of thumb from the text. Increasing the sample size makes the interval narrower. And increasing the confidence level makes it wider. To be more confident, you need a wider net.

SoheeNow the distinction this episode is named for. Ranganathan and colleagues, in an open-access article, say the term statistical significance is often misinterpreted as a clinically important result.

RyanPartly because in everyday English, significant means important. In statistics, it has a far more restrictive meaning.

SoheeClinical significance, they explain, is about the magnitude of the actual treatment effect, and whether it's likely to change practice.

RyanTheir example is striking. A trial in 569 patients with advanced pancreatic cancer compared two drug regimens. Median survival was 6.24 months versus 5.91 months, with a p-value of .038. Statistically significant.

SoheeAnd the difference was about ten days. The authors say most oncologists would agree that's a clinically irrelevant improvement, especially given the added toxicity and cost.

RyanSo what makes a result clinically significant? The article quotes LeFort: the extent of change, whether it makes a real difference to people's lives, how long the effects last, acceptability, cost-effectiveness, and ease of implementation.

SoheeAnd note this: there are traditional cutoffs for statistical significance, but the article says there aren't established values for clinical significance. Often it's the judgment of the clinician and the patient.

RyanThe article also mentions that most journals endorse the CONSORT statement for reporting randomized trials, which emphasizes reporting the effect size and its precision, such as a 95 percent confidence interval.

SoheePause and think, last one. A study of ten thousand clients finds a new intake procedure lowers no-show rates with a p-value below .001, but the effect size is tiny. What's the best interpretation?

RyanThe result is statistically significant, likely because of the huge sample, but the effect is small. Whether it's worth adopting is a separate, clinical judgment about size, cost and real-world benefit.

SoheeSo how does the exam test this? Expect a study with a tiny p-value and a small effect, or a non-significant result in a small sample, and you choose the most accurate interpretation.

RyanThe tempting wrong answers: a small p-value means a large or important effect; non-significance proves no effect; and power can be fixed after the fact.

SoheeRecap. The p-value doesn't show size. Effect size does, and it doesn't depend on sample size.

RyanCohen's d expresses a difference in standard deviation units: about 0.2 small, 0.5 medium, 0.8 or more large.

SoheePower is the chance of detecting a real effect. Raise it with larger samples, bigger effects, less measurement error, or a higher alpha. Plan it before the study, often for 80 percent.

RyanConfidence intervals show precision. Bigger samples narrow them, higher confidence widens them.

SoheeAnd statistical significance is not clinical significance. Ask: how big is the effect, and does it matter to real people's lives?

RyanHonestly, that last question is one you already ask as a clinician every day. You're bringing real wisdom to this topic.

SoheeTry a few interpretation questions next, and for each one ask, how big, and how sure? Thanks for listening to EPPP Prep.

Loading the read-along timings…