Qualitative analysis and meta-analysis

Transcript

SoheeAn open-access article on phenomenology lists the kinds of questions this research can answer. One of them is: what is the medical learner's experience of failure on high-stakes exams?

RyanIf that question lands close to home today, you're in good company. And notice what kind of question it is. Not how many people fail, or what predicts it. What is it like.

SoheeThat's the doorway into qualitative research. And later we'll swing to the other end of the spectrum: combining dozens of studies into one number.

RyanThis is EPPP Prep, the podcast. I'm Ryan.

SoheeI'm Sohee. Today: thematic analysis, phenomenology, and meta-analysis, three analytic methods the knowledge statement names.

RyanLet's start with why qualitative analysis exists. Coates and colleagues, in an open-access guide, say a rigorous approach to qualitative data can yield insights into the intricacies of human behavior and answer how and why questions.

SoheeAnd they mention that there are several qualitative traditions, like grounded theory, ethnography and phenomenology, chosen to fit the question.

RyanThey also ask researchers to decide their point of view before choosing an approach. A positivist approach views data as absolute, and it's the most common approach in hypothesis-testing research.

SoheeA constructivist approach, by contrast, rests on the idea that reality is a social construct, arising from different points of view on shared experiences. Their worked example uses a constructivist lens, and that fits naturally with qualitative work.

RyanThe data usually come from interviews, focus groups, or free-response surveys. Then the real work begins. Coding.

SoheeThe guide defines a code as a short representation that captures the salient features of a main idea expressed by participants.

RyanEarly on, analysts do open coding, identifying main ideas rather than fine details. And they build a codebook, a guide to what each code means, what belongs in it, and examples from the data.

SoheeWhy a codebook? The guide says it can improve consistency among analysts when several researchers are coding.

RyanIn their approach, coders work independently first, then meet to reach consensus on the coding scheme. Later they can calculate and report their percentage of agreement.

SoheeAfter coding, they look for ideas that are similar and group them into themes, sometimes with subthemes. That's the heart of thematic analysis.

RyanTheir worked example is lovely. Residents from other specialties rotating through an emergency department. Several said they were treated differently from the emergency medicine residents. That became a theme, specialty identity.

SoheeBut some experienced it as a disadvantage, being passed over for interesting cases, and others as an advantage, a lighter load. So those became two subthemes.

RyanTwo more terms worth knowing. Saturation. The guide describes it as the point where the ideas in each new transcript are fairly repetitive, and not much new information is coming in. That's when it's safe to stop collecting new data.

SoheeAnd trustworthiness. The guide names member checking, triangulation with other methods, and an external audit trail as ways to support it.

RyanThe guide notes member checking can happen during the interviews or after the analysis, and triangulation draws on other investigational methods. Both are ways of checking that the themes hold up.

SoheeNow phenomenology. Neubauer, Witkop and Varpio describe it as a form of qualitative research that focuses on the study of an individual's lived experiences within the world.

RyanAnd its goal, in their words, is to describe the essence of a phenomenon by exploring it from the perspective of those who have experienced it. Both what was experienced, and how.

SoheeThere are two major approaches, and the exam may contrast them. Transcendental, or descriptive, phenomenology, and hermeneutic, or interpretive, phenomenology.

RyanTranscendental phenomenology traces to Edmund Husserl, whom most historians credit with defining phenomenology in the early twentieth century.

SoheeIts signature move is bracketing, also called epoche. The researchers set aside their previous understandings, past knowledge and assumptions about the phenomenon, so their own subjectivity doesn't bias the analysis.

RyanHermeneutic phenomenology comes from Martin Heidegger. Here the researcher is seen as part of the world and not bias free. So instead of bracketing, the researcher reflects on their own experience while interpreting the participants' accounts.

SoheeThe article puts it this way: hermeneutic phenomenology must go beyond description of the phenomenon, to the interpretation of the phenomenon.

RyanOkay, pause and think. A researcher interviews caregivers about what it's like to care for a parent with dementia. Before and during analysis, she deliberately sets aside her own experiences and assumptions about caregiving. Which approach is this?

SoheePhenomenology, because it's about lived experience. And because she brackets her assumptions to describe the essence, it's the transcendental, descriptive approach associated with Husserl.

RyanNow the other end. Meta-analysis. The OpenStax psychology text defines it plainly, as a study that combines the results of many related studies.

SoheeAhn and Kang's open-access introduction separates it from a systematic review. A systematic review collects all available studies related to a question and reviews and analyzes their results, using clear, reproducible methods.

RyanA meta-analysis goes further, using statistical methods on the estimates from two or more studies to form a pooled estimate. You can have a systematic review without a meta-analysis, if the studies can't be combined.

SoheeAnd it doesn't take dozens of studies. Ahn and Kang note that if the data can be combined, a meta-analysis can even be done with only two studies.

RyanThey also mention the reporting standard published in 2009, called PRISMA, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses, which greatly helped standardize and improve the quality of these reviews.

SoheeGood practice includes registering the plan in advance, and having at least two investigators select studies independently, so the process is transparent and reproducible.

RyanThe pooled estimate is usually shown in a forest plot. Each study gets a square with its result and confidence interval, and the area of the square shows the weight that study carries.

SoheeAnd here's a classic exam distinction. Fixed-effect versus random-effects models.

RyanA fixed-effect model assumes the treatment effect is the same in every study, and that differences between study results are due to random error. A random-effects model assumes the size of the effect differs among studies.

SoheeThe practical consequence: in a fixed-effect model, large studies are weighted more heavily. In a random-effects model, studies get more similar weights regardless of size.

RyanAnd the article shows the choice can matter. In its example, a result that was significant under the fixed-effect model was no longer significant under the random-effects model.

SoheeNow, the biggest threat to a meta-analysis. If the studies going in are biased, the pooled result will be too.

RyanMlinarić and colleagues, in an open-access review, explain publication bias. Studies with positive results are greatly more represented in the literature than studies with negative results. And underreporting negative results introduces bias into meta-analysis.

SoheeHow do you check for it? A funnel plot. Each study is a point, with effect size on one axis and sample size on the other.

RyanAhn and Kang say that if the points form an upside-down funnel, broad at the base and narrowing toward the top, that suggests no publication bias. If the plot is asymmetric, with no points on one side, publication bias can be suspected.

SoheePause and think, second one. A meta-analysis funnel plot shows plenty of small studies with large positive effects, and almost no small studies near zero or negative. What does that suggest?

RyanPossible publication bias. Small studies with null or negative results may never have been published, so the pooled effect could be overstated.

SoheeAnd one more caution from Ahn and Kang: if the quality of evidence in the included studies is low, the quality of the meta-analysis decreases and incorrect results can follow.

RyanSo how does the exam test this? Expect a short description of a study, and you name the method: interviews coded into themes, a study of lived experience with bracketing, or a pooled estimate across studies.

SoheeThe tempting wrong answers: treating a systematic review and a meta-analysis as the same thing, mixing up descriptive and interpretive phenomenology, and trusting a pooled estimate without asking about publication bias.

RyanRecap. Thematic analysis codes the data, then groups codes into themes and subthemes. Codebooks support consistency, saturation tells you when to stop, and member checking, triangulation and audit trails support trustworthiness.

SoheePhenomenology studies lived experience. Husserl's transcendental approach brackets the researcher's assumptions to describe the essence. Heidegger's hermeneutic approach interprets, with the researcher inside the world.

RyanMeta-analysis statistically pools results into one estimate. Fixed-effect assumes one true effect and favors big studies. Random-effects assumes effects vary.

SoheeAnd funnel plot asymmetry suggests publication bias.

RyanThinking back to that opening question, about the experience of failing a high-stakes exam. Researchers study that experience because it's real, and it matters, and it doesn't define anyone.

SoheeYou're still here, still learning, and that says a lot about you. Try a few items on these methods next. Thanks for listening to EPPP Prep.

Loading the read-along timings…