Sensitivity and specificity: what a screening result can and cannot tell you

Transcript

RyanQuick question to start us off, and go with your gut. A depression screener catches 90 percent of the people who truly have depression. A patient screens positive. What's the chance that this patient has depression?

SoheeMost people say 90 percent.

RyanAnd the real answer is: you can't tell from that number alone. By the end of this episode, you'll know exactly why, and you'll never mix these terms up again.

SoheeWelcome to EPPP Prep, the podcast. I'm Sohee.

RyanI'm Ryan. Today, sensitivity and specificity, and their cousins, the predictive values. This is one of the most testable ideas in the assessment domain, and one that researchers and clinicians themselves can misunderstand.

SoheeLet's build the picture first. To study a screening test, you need something to check it against. Traditionally that's called the gold standard. Trevethan's review in Frontiers in Public Health notes that because the validity of these gold standards can be questioned, they've increasingly been called reference standards instead.

RyanSo, say, a full structured diagnostic interview is the reference standard, and a two-question screener is the test we're judging.

SoheeEvery person lands in one of four groups. They have the condition and screen positive: a true positive. They have it and screen negative: a false negative. They don't have it but screen positive: a false positive. They don't have it and screen negative: a true negative.

RyanNow the definitions. Sensitivity is the proportion of people with a condition who are correctly identified by a screening test as having that condition.

SoheeSpecificity is the proportion of people without the condition who are correctly identified as not having it. The federal epidemiology glossary from the Centers for Disease Control and Prevention words them almost the same way.

RyanHere's the technique that makes these questions much more manageable. Find the starting group. Sensitivity starts with the people who truly have the condition. Specificity starts with the people who truly don't.

SoheeAnd the predictive values start somewhere else entirely: with the test result. Positive predictive value is the probability that people with a positive screening result really do have the condition. Negative predictive value is the probability that people with a negative result really don't.

RyanLet's put numbers on it, using a worked example from Parikh and colleagues. Picture 100 people who truly have the condition, and 100 who truly don't. The test comes back positive for 75 of the 100 who have it.

SoheeSo sensitivity is 75 out of 100. Seventy-five percent.

RyanAnd it comes back negative for 85 of the 100 who don't have it. Specificity, 85 percent. Which also means 15 healthy people got false positive results.

SoheeNow switch to the predictive values, where the starting group is the result. Everyone who tested positive is 75 true positives plus those 15 false positives. That's 90 people, and 75 out of 90 is about 83 percent. That's the positive predictive value.

RyanAnd everyone who tested negative is 85 true negatives plus the 25 people the test missed. Eighty-five out of 110, about 77 percent. Negative predictive value.

SoheeNotice one sneaky detail about that example. Exactly half the people in it had the condition. Keep that in your back pocket.

RyanLet's pause and think. A primary care psychologist evaluates a two-question depression screener. Of the 200 patients whom a structured interview confirms have major depression, 180 screened positive. That's 90 percent. What is it?

SoheeStart with the starting group: 200 people who truly have depression. The share of them the screener caught is sensitivity. Ninety percent sensitivity.

RyanAnd the tempting wrong answer is positive predictive value, because the word positive is floating around. But positive predictive value would start from everyone who screened positive, not from everyone who has the condition.

SoheeSame move works in a school. A psychologist checks a brief reading screener against full diagnostic evaluations. Among the 50 students the evaluations identified as having a reading disability, the screener had flagged 40. Starting group: students who have the disability. So 80 percent is sensitivity.

RyanAnd specificity would only enter the picture if the stem started with the students who don't have a reading disability, and asked how many the screener correctly cleared.

SoheeSo back to our opening question. Ninety percent sensitivity tells you about the people who have depression. It says nothing about how many of the positive screens are false alarms, because the false positives never enter that calculation.

RyanTrevethan makes this point bluntly. A positive result on a test with high sensitivity is not, by itself, useful for definitely concluding the condition is present in a particular person.

SoheeAnd there's a name for the mistake. It's called confusion of the inverse. Here's the example from the article, and it's a good one to say out loud. This animal is a dog, therefore it probably has four legs. That's sound.

RyanBut flip it. This animal has four legs, therefore it's probably a dog. That's the error. Lots of four-legged animals are not dogs.

SoheeA highly sensitive test is like four legs. Nearly every dog has them. But having them doesn't make you a dog.

RyanSo when are sensitivity and specificity useful for an individual? The article says: only when they're extremely high, and in a specific direction. That's where the mnemonics come in.

SoheeSnout. Sensitive, negative, out. A highly sensitive test rarely misses true cases, so if it comes back negative, you can fairly confidently rule the condition out.

RyanAnd spin. Specific, positive, in. A highly specific test rarely flags people who don't have the condition, so a positive result helps rule it in.

SoheeNotice the pairing, because it's backwards from intuition. Sensitivity helps you trust a negative. Specificity helps you trust a positive.

RyanNow, the tradeoff. Parikh and colleagues explain it with eye pressure and glaucoma. If you set a high cutoff for calling a test positive, very few healthy people score above it, so specificity is very high. But a significant number of people with glaucoma fall below it, so sensitivity is low.

SoheeLower the cutoff, and you catch more true cases, but you flag more healthy people too. Trevethan describes a study where a test reached 100 percent specificity but only about 45 percent sensitivity, suggesting the cutoff was so stringent that sensitivity had been sacrificed.

RyanAnd that's a deliberate design choice in real instruments. In Yu and colleagues' review of autism assessment, one brief observation measure sets its cutoff for autism risk to ensure around 90 percent sensitivity while keeping specificity adequate.

SoheeSo that cutoff is set to miss few true cases. And for screening tools, the same review is clear about what a positive result leads to: a child found at risk should be referred for a more in-depth diagnostic evaluation. A positive screen leads to a closer look, not to a diagnosis.

RyanThat's also why many screeners come in stages. The same review describes a widely used toddler autism checklist that is a two-stage screener, with a structured follow-up interview for children whose first-stage score suggests moderate risk.

SoheeAnd Parikh notes that repeating a test can raise the positive predictive value. Their example comes from medicine: repeating a different version of a test in the group that already tested positive increases both specificity and positive predictive value.

RyanOne more warning from Trevethan, and it sets up our next episode. These four numbers are not fixed properties of a test. They depend on how stringent the test is, and on the prevalence of the condition in the people you studied.

SoheeHold onto that word, prevalence. Next time we'll see how a rare condition can make an excellent test produce mostly false alarms.

RyanSecond pause and think. Researchers report that of all the veterans who scored above a PTSD checklist's cutoff, a certain share actually had PTSD on the clinical interview. Which statistic is that?

SoheeThe starting group is everyone who screened positive. So that's positive predictive value, not sensitivity.

RyanLet's recap. Sensitivity: of the people who truly have it, how many did the test catch? Specificity: of the people who truly don't, how many did it clear?

SoheePositive predictive value: of the positive results, how many are real? Negative predictive value: of the negative results, how many are truly negative?

RyanSnout: a highly sensitive test that's negative helps rule out. Spin: a highly specific test that's positive helps rule in.

SoheeRaising or lowering a cutoff trades sensitivity against specificity. And none of these numbers is a fixed property of the test.

RyanAnd whenever you're stuck, ask one question: who's the starting group?

SoheeIf these four terms have ever blurred together for you under exam pressure, you're far from alone. Our opening question fools plenty of experienced people. You worked through it carefully today, and that counts.

RyanTry a few sensitivity questions next, and before you look at the options, name the starting group out loud. We'll pick up the story with base rates next time on EPPP Prep.

Loading the read-along timings…