Efficacy, effectiveness and the Dodo bird

Transcript

RyanIn 1936, Rosenzweig published a paper about some implicit common factors in different methods of psychotherapy. And his subtitle quoted a Dodo.

SoheeAt last the Dodo said, everybody has won and all must have prizes. Hold on to that line. It's the image behind one of the longest debates in psychotherapy research.

RyanWelcome to EPPP Prep, the podcast. I'm Ryan.

SoheeI'm Sohee. Comparative efficacy is one of the two heaviest topics in Domain 6, five blueprint spots. So we're giving it two episodes. Today: how to read the evidence, and the famous Dodo bird debate. Next time: head-to-head comparisons.

RyanFirst, two words the exam will absolutely test. Efficacy and effectiveness.

SoheeA primer by Singal and colleagues defines them cleanly. Efficacy is the performance of an intervention under ideal and controlled circumstances. Effectiveness is its performance under real-world conditions.

RyanAnd they add that the line between them is a continuum, not a dichotomy. It's likely impossible to run a pure efficacy or a pure effectiveness study.

SoheeEfficacy trials. The placebo-controlled randomized trial is the ideal design, because it minimizes bias through standardization and blinding. That gives high internal validity.

RyanBut it costs something. The primer says efficacy trials require substantial deviations from clinical practice. Restrictions on who can enroll, control over the provider's skills and actions. And strict inclusion criteria can limit how well results generalize to real patients.

SoheeEffectiveness studies, also called pragmatic studies, use more heterogeneous patients, less standardized protocols, and routine clinical settings. They're more often compared with usual care rather than placebo.

RyanThe trade, in one sentence from the primer: effectiveness studies sacrifice some internal validity, but they have higher external validity than efficacy studies.

SoheePause and think. A study tests a manualized therapy in a community clinic, with whoever walks in, including people with several conditions, compared with the clinic's usual care. Efficacy or effectiveness?

RyanEffectiveness. Heterogeneous patients, routine setting, usual-care comparison. If instead it recruited carefully screened patients with one diagnosis, trained and monitored therapists, and a placebo or tightly matched control, that would lean efficacy.

SoheeNow the Dodo bird. The question underneath it is simple to ask and fiercely debated: are some psychotherapies more effective than others?

RyanBruce Wampold's 2015 review in World Psychiatry lays out both camps. Today, some claim certain treatments, in general or for specific disorders, are more effective than others. Others claim there are no differences among psychotherapies in their outcomes.

SoheeAnd the OpenStax psychology textbook takes a position in its section on evaluating psychotherapy. It says no studies have found one psychotherapeutic approach more effective than another.

RyanWampold's framework for the no-big-differences side is called the contextual model. It says psychotherapy works through three pathways: the real relationship, the creation of expectations through explaining the disorder and the treatment, and the enactment of health-promoting actions.

SoheeAnd before those pathways can work, an initial therapeutic relationship has to be established. So the relationship is the doorway.

RyanThe expectations pathway has a lovely idea behind it, from Frank, author of Persuasion and Healing. People often arrive demoralized, not only by their distress, but because they've tried many times to fix the problem and failed. Wampold says participating in therapy appears to be a form of remoralization.

SoheeAnd there's a striking experiment on warmth. Patients with irritable bowel syndrome got sham acupuncture, either with a brief, silent practitioner or with a warm, attentive conversation using active listening and confident, positive expectations.

RyanBoth sham conditions beat the waiting list. But the warm, augmented relationship condition beat the limited one, particularly for quality of life. Same fake needles. Different relationship.

SoheeA powerful reminder that the relationship isn't decoration. Okay, let's get to the numbers.

RyanNow the evidence. Start with the alliance, which he calls the most researched common factor. It has three components: the bond, agreement about the goals of therapy, and agreement about the tasks.

SoheeThe most recent meta-analysis he cites included nearly 200 studies and over 14,000 patients. The correlation between alliance and outcome was about 0.27, equivalent to a Cohen's d of 0.57, which passes the threshold for a medium effect.

RyanAnd he takes the obvious objection seriously. Maybe clients who improve early then rate their therapist more warmly. He says that to address it, early progress must be statistically controlled, and the criticisms he reviews were found not to reduce the alliance's importance.

SoheeA related factor, goal consensus and collaboration, showed a strong effect, d equals 0.72, across fifteen studies. And expectations showed a relatively small but significant relationship, d equals 0.24.

RyanThen therapist effects. Some therapists consistently get better outcomes than others, regardless of the patients or the treatment.

SoheeIn clinical trials, therapist effects were modest, d equals 0.35. In naturalistic settings, larger, d equals 0.55. Which makes sense, since trial therapists are selected, trained, supervised, and monitored, and trial patients are more uniform.

RyanSo what do the more effective therapists do? Wampold summarizes research showing they form stronger alliances across a range of patients, have stronger facilitative interpersonal skills, express more professional self-doubt, and spend more time outside therapy practicing their skills.

SoheeI love that self-doubt is on that list. Questioning yourself, in a healthy way, is part of getting better.

RyanNow the specific side of the ledger. Wampold says that when meta-analyses do find differences among treatment types, they typically find differences of at most about d equals 0.20.

SoheeDismantling studies take a treatment apart. They remove a supposedly key ingredient and compare. Two meta-analyses found minimal differences, d equals 0.01. Though the more recent one found that adding an ingredient increased effects on targeted variables by a small amount, d equals 0.28.

RyanAnd adherence and competence. You'd expect that following the protocol closely, and doing it skillfully, would predict outcome. The meta-analysis found small effects: d equals 0.04 for adherence and 0.14 for competence.

SoheeWampold notes evidence that rigid adherence to a protocol can weaken the alliance and increase resistance, and that flexibility in adherence relates to better outcomes.

RyanAnd he explains the competence puzzle. In those trials, competence was rated for the specific treatment protocol, not for common factors like empathy and alliance, which are the parts that do predict outcome.

SoheePause and think. An answer choice says: the strongest predictor of outcome across psychotherapy research is the therapist's adherence to a treatment manual. Based on Wampold's review, is that defensible?

RyanNo. In his figures, adherence had a tiny effect, d equals 0.04, while the alliance, empathy, and goal consensus had medium to large ones. The tempting part is that manuals sound rigorous. But rigor of delivery isn't the same as what predicts outcome in these studies.

SoheeOne more piece of the debate. Frost, Baskin, and Wampold asked whether trials showing large differences between therapies have been replicated.

RyanThey found twelve trials with large effects within ten meta-analyses. They found no direct replication attempts, and four conceptual replications, of which only two partially corroborated the original results.

SoheeTheir conclusion: since they found no evidence that trials with such large effects are directly replicable, treatment superiority conclusions from those reviews are highly questionable.

RyanNow, fairness. Wampold is an author on both papers, and his review concludes that the common factors are important. So this is one side of the debate, argued with evidence. And next episode we'll hear from meta-analyses that do find some differences, for example for particular disorders. The exam isn't asking you to pick a team. It's asking you to know what each body of evidence says.

SoheeAnd to remember one structural fact from the National Institute of Mental Health. There's no formal approval process for psychotherapies, like there is for medications through the Food and Drug Administration. Evidence comes from research, not a license.

RyanRecap. Efficacy: ideal, controlled conditions, high internal validity. Effectiveness: real-world conditions, higher external validity. It's a continuum.

SoheeThe Dodo bird idea goes back to Rosenzweig in 1936. Wampold's contextual model predicts that structured therapies given by empathic, caring therapists, which get clients doing healthy things, will have approximately equal effects. Differences, when found, are typically at most about d equals 0.20.

RyanAlliance: bond, goals, tasks. Correlation with outcome about 0.27, the most researched common factor. Empathy and goal consensus also relate to outcome. Therapist effects are larger in naturalistic settings than in trials.

SoheeDismantling and adherence effects are small. And large superiority effects haven't been directly replicated in the trials Frost and colleagues examined.

RyanHere's a thought to take with you. These studies keep finding that people change inside a relationship where they feel understood and have a shared plan. That applies to studying too. A study partner, a coach, a clear plan. You don't have to do this alone.

SoheeThanks for listening to EPPP Prep.

Loading the read-along timings…