Hypothesis testing and the two kinds of error

Transcript

RyanThink about the smoke alarm in your kitchen. It can make two kinds of mistakes. It can shriek when you're only making toast. Or it can stay silent while something is really burning.

SoheeAnd those two mistakes don't feel the same at all. One is annoying. The other is dangerous.

RyanHold on to that alarm, because it's the whole idea behind today's topic.

SoheeWelcome to EPPP Prep, the podcast. I'm Sohee.

RyanI'm Ryan. Today, hypothesis testing: the null hypothesis, the p-value, alpha, and the two kinds of error. It's a core idea in the research domain, and once the logic clicks, it stays clicked.

SoheeEvery test starts with two competing statements. The OpenStax statistics textbook describes the null hypothesis as a statement of no difference between the variables, that they're not related.

RyanAnd the alternative hypothesis is the claim that contradicts the null. It's what we conclude if we reject the null, and usually what the researcher is trying to show.

SoheeSo in a therapy study, the null says the therapy makes no difference. The alternative says it does.

RyanThen there are only two possible decisions. Reject the null, or do not reject it. And here's a phrase worth tattooing on your memory.

SoheeThe text says that when you do not reject the null, it does not mean you should believe the null is true. It means the sample didn't provide enough evidence to cast serious doubt on it.

RyanSo you never prove the null. That's why the textbook's wording is do not reject, or decline to reject, rather than accept.

SoheeNow, how do we decide? The textbook's logic is about rare events. You assume the null is true. You gather data. If your sample would be very unlikely under that assumption, you take that as evidence against the assumption.

RyanIts story is lovely. At a birthday party there's a basket with two hundred plastic bubbles, and two friends, Didi and Ali, are told only one holds a hundred-dollar bill. Didi reaches in first and pulls out the hundred-dollar bill.

SoheeIf there really were only one, the chance of that was one in two hundred. So her friend Ali starts to doubt what they were told, and hopes there are more bills in the basket.

RyanThat's hypothesis testing in a nutshell. A surprising result makes you doubt the assumption.

SoheeAnd the p-value puts a number on how surprising. The text defines it as the probability that, if the null hypothesis is true, another random sample would give results as extreme or more extreme than the ones you got.

RyanNotice the if. The p-value is calculated assuming the null is true. The smaller it is, the more unlikely your result would be under the null, and the stronger the evidence against the null.

SoheeThen we compare it with alpha, the significance level, which is set in advance. The textbook defines alpha as the probability of a Type I error, rejecting the null when the null is true.

RyanThe rule: if the p-value is less than alpha, reject the null, and the results are called significant. If not, don't reject. The textbook even offers a memory aid: when the p is low compared with alpha, the null is rejected.

SoheeAnd what alpha do psychologists usually use? The OpenStax psychology textbook says psychologists generally consider differences statistically significant if there's less than a five percent chance of observing them when the groups don't actually differ.

RyanPut another way, the text says psychologists want to limit the chance of false positive claims to five percent or less. That's the chance of a false alarm when there's truly no difference.

SoheeNow the heart of it, the four possible outcomes. The statistics text lays them out. Two are correct decisions, two are errors.

RyanCorrect decision one: don't reject the null when it's true. Correct decision two: reject the null when it's false. The probability of that second one has a special name, the power of the test.

SoheeError one, the Type I error: rejecting the null when it's actually true. You claim an effect that doesn't exist. That's the smoke alarm screaming over toast. A false alarm.

RyanError two, the Type II error: not rejecting the null when it's actually false. A real effect goes undetected. That's the alarm staying silent during a real fire. A miss.

SoheeAlpha is the probability of a Type I error. Beta is the probability of a Type II error. The text says both should be as small as possible, because they're probabilities of errors, but they're rarely zero.

RyanAnd power is one minus beta. So power is the chance of catching a real effect when there is one. The text says we want power as close to one as possible, and that increasing the sample size can increase power.

SoheeHere's the method that makes these questions reliable. It's the same advice in the study notes for this topic on our site: ask two things. What did the researcher decide? And what is really true?

RyanResearcher rejected the null, but there's truly no effect: Type I. Researcher kept the null, but there's truly an effect: Type II. Rejected the null and the effect is real: a correct decision. That's the power side.

SoheeOkay, pause and think. A team concludes that a parenting workshop reduces children's tantrums. Later, several large, well-designed trials show the workshop has no effect at all. If those later trials are right, what happened in the first study?

RyanThey rejected the null, but the null was true. No effect existed. That's a Type I error.

SoheeNow flip it. A small study of a new sleep therapy finds no significant difference from usual care. But the therapy truly does help. What happened?

RyanThey failed to reject a false null. A real effect was missed. Type II. And a small sample may be part of the reason, because smaller samples have less power.

SoheeNow, which error is worse? The textbook's answer is: it depends on the situation, and its examples show both directions.

RyanIn one, the null is that Navah's rock climbing equipment is safe. A Type II error means she thinks it's safe when it isn't. And she climbs. The text says the Type II error has the greater consequence there.

SoheeIn another, the null is that an accident victim arriving at the emergency room is alive. A Type I error means the crew thinks the victim is dead when they're alive, and won't treat them. There, the Type I error is worse.

RyanAnd that's why researchers think carefully about alpha. Sullivan and Feinn, in an open-access paper on effect size, list raising the alpha level as one way to increase power, but only if making a Type I error is highly unlikely.

SoheeSo there's a trade-off. A stricter alpha, like one percent, guards harder against false alarms, but makes real effects harder to detect.

RyanPause and think, one more. A researcher sets alpha at .05 and gets a p-value of .03. What's the decision, and if it's wrong, which error is it?

SoheeThe p-value is below alpha, so reject the null. If that decision is wrong, the null was really true, so it would be a Type I error. A Type II error isn't possible here, because the researcher didn't keep the null.

RyanThat last point is a favorite trap. Once you know the decision, only one type of error is even possible.

SoheeSo how does the exam test this? Mostly with scenarios. A study's conclusion, then later information about the truth, and you label the outcome.

RyanThe tempting wrong answers: mixing up which error is which, calling a rejection a Type II error, and treating do not reject as proof that the null is true.

SoheeLet's recap. The null hypothesis says no difference or no relationship. The alternative is what the researcher usually hopes to show.

RyanThe p-value is how likely results this extreme, or more extreme, would be if the null were true. If it's below alpha, reject the null.

SoheeType I error: rejecting a true null, a false alarm, with probability alpha. Type II error: failing to reject a false null, a miss, with probability beta.

RyanPower is one minus beta, the chance of detecting a real effect, and bigger samples raise it.

SoheeAnd the two-question method: what did they decide, and what's really true?

RyanIf Type I and Type II have ever blurred together for you, you're far from alone. They blur for almost everyone at first. Picture the smoke alarm, ask the two questions, and they separate.

SoheeGo try a few error questions now, slowly, out loud if you can. Thanks for listening to EPPP Prep.

Loading the read-along timings…