Experiments and quasi-experiments

Transcript

RyanA counseling center launches a new group for social anxiety. Students can sign up for the new group, or stay with the usual individual sessions. At the end of the term, the new group's members are doing noticeably better.

SoheeSo the director wants to announce that the new group works.

RyanAnd today's whole episode lives inside the question: what else could explain that difference?

SoheeWelcome back to EPPP Prep, the podcast. I'm Sohee.

RyanI'm Ryan. We're on experiments and quasi-experiments, the designs that let researchers talk about cause, and the ones that try to.

SoheeLet's build an experiment from the ground up, the way the OpenStax psychology textbook does. The most basic design has two groups, an experimental group and a control group, designed to be the same except for one difference: the experimental manipulation.

RyanThe thing you manipulate is the independent variable. The thing you measure, to see how much effect it had, is the dependent variable. The dependent variable depends on the independent variable.

SoheeAnd you need an operational definition, a precise description of how you'll measure each variable, so anyone reading your study knows exactly what you meant, and could repeat it.

RyanNow the step that makes or breaks causal claims. Random assignment. The textbook says that with random assignment, all participants have an equal chance of being assigned to either group.

SoheeAnd with large enough samples, that makes systematic differences between the groups unlikely. If the groups weren't different before, any difference afterward can be attributed to the independent variable.

RyanThe OpenStax statistics textbook says it nicely. When treatments are assigned randomly, all the potential lurking variables are spread equally among the groups.

SoheeNow look back at our counseling center. Students chose their own group. Maybe the ones who signed up for something new were more motivated, or less severely anxious, to begin with.

RyanExactly. Without random assignment, the groups could differ before the therapy ever starts.

SoheeAnd here's a distinction the exam loves. Random assignment is not the same as random sampling.

RyanRandom sampling is about who gets into the study. The psychology text defines a random sample as one where every member of the population has an equal chance of being selected, which helps the sample represent the population.

SoheeSo random sampling supports generalizing to the population. Random assignment supports concluding that the independent variable caused the difference. Different jobs, and you can have one without the other.

RyanNext, protecting the experiment from expectations. The placebo effect, in the textbook's words, occurs when people's expectations or beliefs influence or determine their experience.

SoheeSo the control group gets a placebo, like a sugar pill. The statistics text calls a placebo a treatment that cannot directly influence the response variable.

RyanAnd then blinding. In a single-blind study, participants don't know which group they're in. In a double-blind study, neither the participants nor the researchers working with them know.

SoheeWhy blind the researchers too? Experimenter bias. The textbook describes it as the possibility that a researcher's expectations might skew the results. If scorers know who got the new method, they might read a sloppy answer more generously.

RyanNow, what about questions where you can't randomly assign? The textbook's example: the effect of sex on spatial memory. You can compare groups, but you can't assign someone's sex.

SoheeAnd it calls that kind of research quasi-experimental, and says we cannot make cause-and-effect claims in these circumstances.

RyanEthics creates the same limit. You couldn't run an experiment on whether childhood abuse lowers adult self-esteem, because you'd have to randomly assign children to abuse.

SoheeAn open-access review by Harris and colleagues on quasi-experimental studies gives a crisp definition. Quasi-experiments aim to evaluate interventions but don't use randomization.

RyanThey list the practical reasons: ethics, difficulty randomizing individuals or locations, and small sample sizes. And they're blunt that the lack of random assignment is the major weakness of the design.

SoheeBut not all quasi-experiments are equally weak, and that's worth knowing. The review describes a hierarchy.

RyanAt the bottom, a single group measured only after the intervention. The review calls it the weakest design they discuss, because without a pretest or a control group, there are multiple threats to internal validity.

SoheeOne step up, the one-group pretest-posttest design. Measure, intervene, measure again. That first measurement gives some idea of what things looked like without the intervention.

RyanBetter still, add a comparison group and pretests for both groups. If the groups look similar at the pretest, it's less likely that important confounding variables differ between them.

SoheeBut they're still nonequivalent groups, so selection bias may exist. The review defines selection bias as differences in the characteristics of the groups that may be related to differences in outcomes.

RyanAnd at the top of their hierarchy, the interrupted time-series design. A string of observations equally spaced in time, interrupted by the intervention.

SoheeWith many measurements before and after, it's easier to address confounding and regression to the mean. The review adds a lovely detail: a sudden jump right after the intervention suggests an immediate effect, while a change in the slope suggests a gradual effect.

RyanThe review also mentions the counterbalanced design, sometimes called a Latin square. Everyone receives all the interventions, but the order varies, so one group gets A then B, and another gets B then A.

SoheeIt's a within-participants design, and varying the order lets researchers study whether the sequence itself matters.

RyanOkay, pause and think. A researcher compares depression scores between people who chose to attend a community support group and people who didn't. The attenders score lower. What kind of design is this, and what's the main problem?

SoheeThere's no random assignment, so it's not a true experiment. People selected themselves into the groups. The attenders may have differed before they ever walked in, so the researcher can't conclude the group caused the lower scores.

RyanLast design for today, because it's a favorite on exams. The factorial design.

SoheeAn open-access paper on factorial trials by Montgomery, Peters and Little describes the simplest version. With two interventions, participants receive neither, one or the other, or both. That's the classic two-by-two.

RyanThe big payoff is the interaction. The paper describes no interaction as the effect of intervention A not differing depending on whether participants also receive intervention B.

SoheeSo an interaction is when it does differ. Picture a study where a new therapy helps only when people are also taking medication. The effect of therapy depends on the level of medication.

RyanAnd the paper notes factorial trials are the only design that allows such effects to be investigated.

SoheeThere's a cost, though. Detecting an interaction usually needs a bigger sample. The paper says detecting an interaction the same size as the main effects, with the same power, takes a fourfold increase in sample size.

RyanPause and think, second one. A two-by-two study tests exercise and a sleep program on mood. Exercise improves mood a lot for people in the sleep program, and hardly at all for people who aren't. What do you call that?

SoheeAn interaction. The effect of one factor depends on the level of the other.

RyanBefore we wrap, one honest limitation of true experiments. The psychology text notes they're often run in artificial settings, which raises questions about how the findings apply in the real world.

SoheeSo experiments are strongest on cause, and sometimes weaker on real-world fit. That tension comes back when we talk about internal and external validity.

RyanHow does the exam test this? A design described in a few sentences, and you name it, or name its biggest weakness.

SoheeWatch for the word random. Random selection of who's in the study doesn't make it an experiment. Ask: did the researcher randomly assign people to conditions?

RyanAnd watch for groups defined by something the researcher can't assign, like sex, diagnosis, or which school someone attends. That points to quasi-experimental.

SoheeLet's recap. A true experiment manipulates an independent variable and randomly assigns participants to conditions. That's what supports a causal conclusion.

RyanRandom sampling helps generalization. Random assignment helps causation.

SoheePlacebos and double-blinding guard against participant expectations and experimenter bias.

RyanQuasi-experiments lack random assignment. Adding pretests, comparison groups and many measurements over time makes them stronger, with interrupted time series at the top.

SoheeCounterbalancing varies the order of conditions. And factorial designs let you see interactions, when one factor's effect depends on another.

RyanAnd back to our counseling center. The honest announcement is: students who chose the new group did better, and a randomized study could tell us whether the group itself is the reason.

SoheeIf you're studying after a hard result, please know these design questions reward calm, step-by-step thinking more than memorizing. Ask what was manipulated and how people got into each group, and you're most of the way there.

RyanTry a few design questions next, and say the design's name out loud before you read the choices. Thanks for listening to EPPP Prep.

Loading the read-along timings…