Critical appraisal: validity threats and levels of evidence

Transcript

RyanA health system picks out its highest-risk patients, the ones with the most extreme numbers on cost or clinical measures, and enrolls them in a new coaching program. A year later, their numbers have come down.

SoheeThe program director is thrilled. But there's a statistical ghost in that room, and today we're going to name it.

RyanThis is EPPP Prep, the podcast. I'm Ryan.

SoheeI'm Sohee. Today is critical appraisal: deciding how much to trust a study. Internal and external validity, the classic threats, and levels of evidence.

RyanTwo kinds of validity first. Patino and Ferreira, in a short open-access article, define internal validity as the extent to which the observed results represent the truth in the population being studied, and aren't due to methodological errors.

SoheeAnd external validity as the extent to which results can be generalized beyond the study, to the people and settings we actually work with.

RyanThey add an important ordering. If a study isn't internally valid, its results deviate from the truth, and external validity is irrelevant. You can't generalize a wrong answer.

SoheeSo let's go through the classic threats to internal validity. The Harris review on quasi-experiments lists nine, adapted from Shadish, Cook and Campbell. Each one is an alternative explanation for an apparent effect.

RyanAmbiguous temporal precedence. It isn't clear whether the intervention came before the outcome. Cause has to come first.

SoheeSelection. Systematic differences in the people in each condition could cause the effect. That's the volunteers-versus-nonvolunteers problem.

RyanHistory. Events happening at the same time as the intervention could cause the effect. A new school policy, a news event, a change in staffing.

SoheeMaturation. Naturally occurring changes over time get mistaken for a treatment effect. Children grow and develop over a school year whether or not a program is running.

RyanRegression. When people are selected for their extreme scores, they'll often have less extreme scores later, and that can be confused with an intervention effect.

SoheeAttrition. Losing participants can produce artificial effects if who drops out is related to the intervention.

RyanTesting. Taking a test can change scores the next time you take that test. Practice, familiarity, remembering items.

SoheeInstrumentation. The measure itself changes over time or across conditions. New raters, a revised scale, a recalibrated machine.

RyanAnd the ninth, additive and interactive effects of threats. An open-access catalog of validity threats by Matthay and Glymour explains that several biases can add up, offset one another, or interact in a single study.

SoheeNow back to our coaching program. That's the regression threat. Linden's open-access paper describes exactly this situation.

RyanPeople invited because of outlier values will likely have lower values when measured again, even without any intervention. That's regression to the mean, and Linden says it often leads to the inaccurate conclusion that the intervention caused the effect.

SoheeWhy does it happen? The Harris review explains that a rise can be an extreme observation that's still within the normal range. If the true average hasn't shifted, regression to the mean predicts the elevated numbers will tend to decline even without an intervention.

RyanThe phenomenon was first described by Francis Galton in 1886, measuring heights. Very tall parents tended to have children shorter than themselves, and very short parents tended to have taller children.

SoheeSo what protects you? Linden says regression to the mean can easily be mistaken for a program effect when there's no equivalent comparison group. And the Harris review says adding a second pretest before the intervention helps refute regression to the mean as an explanation.

RyanLinden also notes that analysis of covariance is the most common analytic approach in the literature for adjusting for regression to the mean.

SoheeLinden even notes that commercial disease management organizations have long advocated evaluating their programs without a control group, despite recognizing that the intervention group will show better outcomes because of regression to the mean.

RyanWhich is exactly why a critical reader always asks: who was chosen for this program, and compared with whom?

SoheeOne more threat from outside the list, the Hawthorne effect. The OpenStax psychology text traces it to studies at Western Electric's Hawthorne Works plant near Chicago.

RyanThe text says any change, even returning to poorer lighting, improved productivity, and the effect faded when the attention faded. The Hawthorne effect describes better performance by people who know they're being observed.

SoheeAnd the text says it remains a factor that has to be controlled for. A treatment may produce an effect only because participants are getting more attention.

RyanOkay, pause and think. A school starts a reading program in September and reading scores are clearly higher by May. There's no comparison group. Name two threats.

SoheeMaturation: children's reading naturally improves over a school year. And history: anything else that happened that year, like a new curriculum. Testing could also play a role if the same test was given twice.

RyanNow external validity, generalizability. The psychology text points out that the vast majority of psychological research has historically used college students, who tend to be younger, more educated, more liberal and less diverse than the general population.

SoheeAnd that makes it difficult to generalize findings to the larger population. Samples matter for external validity.

RyanPatino and Ferreira say external validity can be increased by broad inclusion criteria, so the study population looks more like real-life patients, and by choosing interventions that are feasible to apply.

SoheeThe OpenStax statistics text adds a practical checklist for critical evaluation. Problems with samples, self-selected samples, samples that are too small, undue influence in how questions are asked, non-response, claims of causality from relationships, self-funded studies, misleading use of data, and confounding.

RyanOn self-funded studies, its advice is balanced. Don't automatically assume the study is good, and don't automatically assume it's bad. Evaluate it on its merits.

SoheeNow levels of evidence. Burns, Rohrich and Chung, in an open-access review, describe it as a hierarchical system of classifying evidence, and a cornerstone of evidence-based medicine.

RyanThe levels were originally described by the Canadian Task Force on the Periodic Health Examination in 1979, and further expanded by Sackett in 1989.

SoheeBoth systems place randomized controlled trials at the highest level, and case series or expert opinion at the lowest. And the hierarchies rank studies according to the probability of bias.

RyanWhy are randomized trials on top? The review says they're designed to be unbiased and have less risk of systematic error. Random allocation also randomizes confounding factors.

SoheeBut here's the nuance that separates good answers from tempting ones. The right design depends on the question. For a question about prognosis, what happens if we do nothing, the review says the highest evidence would come from a cohort study, not a randomized trial.

RyanAnd a level doesn't guarantee quality. The review says a poorly conducted randomized trial may report a negative result because of low power, when a real difference exists.

SoheeIt lists what to check in a trial: randomization, blinding, a description of how both were done, how many people withdrew or dropped out, confidence intervals around the estimates, and a power analysis.

RyanAnd lower levels still have value. The review says case series and case reports are important for generating hypotheses, which can lead to more controlled studies.

SoheePause and think, second one. A clinician wants the strongest evidence on whether a therapy works. Rank these: a single case report, a well-designed randomized controlled trial, and expert opinion.

RyanThe randomized trial is highest. The case report and expert opinion are at the lowest levels. But I'd still check that trial's randomization, blinding, dropouts and power before trusting it.

SoheeSo how does the exam test this? A study description, and you name the threat, or choose the design change that would fix it, or identify the highest level of evidence.

RyanThe tempting wrong answers: confusing maturation with history, missing regression to the mean when people were chosen for extreme scores, and assuming any randomized trial is automatically sound.

SoheeRecap. Internal validity asks whether the result is true for this study. External validity asks whether it generalizes. Without internal validity, external validity doesn't matter.

RyanThe threats: temporal precedence, selection, history, maturation, regression, attrition, testing, instrumentation, and the way threats add up or interact. Plus the Hawthorne effect, improvement from being observed.

SoheePeople chosen for extreme scores drift back toward the mean. Comparison groups and multiple pretests help.

RyanLevels of evidence rank designs by risk of bias, randomized trials high, case reports and expert opinion low. But the best design depends on the question, and every study still needs appraisal.

SoheeYou know, critical appraisal is a skill you'll use for your whole career, long after this exam. Every minute you spend on it now is an investment in your future clients.

RyanTry a few validity-threat questions next, and for each one ask, what else could explain this result? Thanks for listening to EPPP Prep.

Loading the read-along timings…