Program evaluation, community partnership and sharing results

Transcript

SoheeA county wants to support teenagers' mental health, and someone proposes a peer mentoring program. Before it even exists, someone at the meeting asks: how will we know if it works?

RyanAnd that one question opens three knowledge statements at once. How programs get evaluated, how communities take part in research, and how findings get shared.

SoheeWelcome to EPPP Prep, the podcast. I'm Sohee.

RyanI'm Ryan. Each of these three has one spot on the exam blueprint, so we'll move briskly. And we'll lean on the Centers for Disease Control and Prevention, the CDC, for evaluation.

SoheeThe CDC says program evaluation lets you determine how effective and efficient your programs are in reaching their outcomes. And it separates evaluation from research.

RyanResearch aims to contribute to generalizable knowledge. Evaluation aims to continuously improve organizations, and to inform decisions. That difference in purpose shapes everything that follows.

SoheeLet's follow our mentoring program through its life. Before it starts: is it needed? That's a needs assessment.

RyanAn open-access BMJ article by Wright, Williams and Wilkinson describes health needs assessment as the systematic approach to ensuring services use their resources to improve the health of the population in the most efficient way.

SoheeAnd they make a distinction worth remembering. Need, demand and supply aren't the same. Need is commonly defined as the capacity to benefit. Demand is what people ask for. Supply is the care actually provided.

RyanWhy does that matter? Because the article notes that some groups who may need care don't demand it, including homeless people and people with chronic mental illness. As the article puts it, the needs of patients coming through the consulting room door may not reflect the wider needs of the community.

SoheeNext, as the program is being designed: formative evaluation. The CDC says it assesses whether a program is feasible, appropriate and acceptable before it's fully implemented, and it focuses on learning and improvement.

RyanIts usefulness, in the CDC's words, is that it allows modifications before full implementation. So you might pilot the mentoring sessions with a few teens and ask, does this fit their lives?

SoheeOnce it's running: process, or implementation, evaluation. The CDC says it assesses how well implementation followed the original plan, including the content, quality and quantity of what's delivered.

RyanAre mentors meeting teens as often as planned? Is the training happening? The CDC says this gives early identification of gaps that can be addressed early.

SoheeAfter it's been implemented: outcome evaluation, which measures how well the program achieved its intended outcomes. And here's a detail the exam can use. The CDC says outcome evaluation cannot determine what caused the outcomes, only whether they were achieved.

RyanFor cause, you need impact evaluation. The CDC says it compares the outcomes with estimates of what the outcomes would have been without the program, and usually seeks to determine whether the activities caused them.

SoheeAnd the big pair from the knowledge statement: formative versus summative. An open-access paper by Parry and colleagues puts it in two sentences. Summative evaluation assesses program impact and determines the degree to which the program was successful. Formative evaluation aims to improve programs as they evolve.

RyanSo formative is for improving while you go. Summative is for judging success. A participatory evaluation paper by Vat and colleagues adds the timing: summative evaluation is typically conducted once a program is established, to examine its effects or outcomes.

SoheeThen the money questions. The CDC lists economic evaluation, including cost analysis, cost-benefit analysis and cost-utility analysis.

RyanCost-benefit analysis expresses both costs and benefits in money. Its output is the net benefit, benefits minus costs. The CDC's example is reducing trans fats in the food supply: about 140 billion dollars in benefits, against 6 billion in costs, for 134 billion in net benefits.

SoheeBecause everything's in dollars, the CDC notes you can even compare health and non-health interventions, like a health program and an environmental one.

RyanCost-effectiveness analysis is different. It estimates how much it costs to gain one unit of a health outcome, like a life year gained or a death prevented. And it's comparative. An intervention is only cost effective compared with something else.

SoheeOkay, pause and think. A clinic pilots a new intake process with a handful of families to see whether it's acceptable and workable before rolling it out. Which kind of evaluation?

RyanFormative. It's before full implementation, aimed at feasibility, acceptability and improvement. If they were instead checking whether staff delivered the intake as designed, that would be process evaluation.

SoheeNow community involvement. Wallerstein and Duran, in an open-access article, gather definitions of community-based participatory research, often shortened to CBPR.

RyanOne describes it as inquiry with the participation of those affected by an issue, for education and action for change. Another says it equitably involves all partners in research on a topic important to the community, combining knowledge and action to improve community health and eliminate health disparities.

SoheeAnd they stress it's an orientation, an overall approach that equalizes power between academic and community partners. It isn't a specific qualitative or quantitative method. And it is not community outreach for one-way transmission of information.

RyanWhy does that matter so much? Trust. The article describes a lack of trust between researchers and underrepresented communities, with historical resonance from the Tuskegee study.

SoheeThe OpenStax psychology text tells that history. In 1932, the U.S. Public Health Service recruited 600 Black men in Tuskegee, Alabama, for a study of syphilis. Men with syphilis were never told they had the disease.

RyanBy 1947 penicillin was recognized as an effective treatment, yet none was given. The study ended in 1972 when the national press discovered it, and the outrage led directly to the National Research Act of 1974.

SoheeA set of nine CBPR principles from Israel and colleagues is summarized in an open-access paper by Winterbauer and colleagues. Let's walk through most of them.

RyanCBPR recognizes community as a unit of identity. It facilitates collaborative, equitable partnership in all phases of the research. It balances research and action for the mutual benefit of all partners.

SoheeIt promotes co-learning and capacity building among all partners, and builds on strengths and resources within the community. It involves a long-term process and commitment.

RyanAnd principle eight bridges to our last topic. CBPR disseminates findings and knowledge gained to all partners, and involves all partners in the dissemination process.

SoheeWallerstein and Duran add that collaborative data analysis and dissemination can strengthen community ownership and use of the data.

RyanNow dissemination more broadly. The OpenStax psychology text describes peer review. Several scientists with relevant expertise, generally anonymous, read the paper and look for a strong rationale, clear methods, ethical conduct, and flaws in design and statistics.

SoheeThe editor then decides whether to publish it as is, which the text says is rare, publish it with revisions, or not accept it. Peer review also makes sure the study is described clearly enough for others to replicate it.

RyanAnd replication matters. The text describes a replication crisis, in which some well-known studies have failed to replicate, and notes some argue it's actually improving how science is done and shared.

SoheeSometimes a paper is retracted. The text says a publication can be rescinded when data are called into question because of falsification, fabrication, or serious design problems.

RyanThe Ethics Code speaks directly to all this. Standard 8.10: psychologists do not fabricate data. And if they discover significant errors in their published data, they take reasonable steps to correct them, in a correction, retraction, erratum, or other means.

SoheeStandard 8.11: no plagiarism, even if the other source is cited occasionally. Standard 8.12, publication credit: authorship reflects actual contribution, and holding a position like department chair doesn't justify authorship.

RyanAnd under 8.12, except in exceptional circumstances, a student is listed as principal author on a multiple-authored article substantially based on the student's doctoral dissertation.

SoheeStandard 8.13: don't publish previously published data as original data. And 8.14: after publication, don't withhold your data from competent professionals who want to verify the claims through reanalysis, as long as participants' confidentiality can be protected.

RyanOne more ethical thread. Mlinarić and colleagues argue that negative results deserve publication too, noting that participants exposed themselves to risk with the assurance that the study would benefit others.

SoheePause and think, last one. A department chair who didn't contribute to a study asks to be listed as an author. What does the Ethics Code say?

RyanNo. Under 8.12, authorship reflects actual scientific or professional contribution, and holding an institutional position alone doesn't justify it.

SoheeSo how does the exam test this? Name the evaluation type from its timing and purpose, separate cost-benefit from cost-effectiveness, recognize CBPR principles, and apply the publication standards.

RyanThe tempting wrong answers: using outcome evaluation to claim causation, calling a mid-program check summative, treating CBPR as one-way outreach, and offering authorship for status.

SoheeRecap. Needs assessment comes first; need means capacity to benefit, not demand. Formative improves before full rollout. Process checks delivery against the plan. Outcome checks whether goals were met. Impact compares with what would have happened without the program. Summative judges overall success.

RyanCost-benefit puts everything in dollars and gives net benefit. Cost-effectiveness gives cost per unit of health outcome, compared with an alternative.

SoheeCBPR is an equitable, long-term partnership in every phase, including sharing results with the community. And the Ethics Code asks for honest data, corrected errors, fair credit, no duplicate publication, and data shared for verification.

RyanThat's the end of our research methods series. If you've made it through all of these, please take a moment to notice how much ground you've covered. That's real work.

SoheeRest when you need to, come back to the episodes that felt shaky, and try a mixed set of research questions when you're ready. Thanks for listening to EPPP Prep.

Loading the read-along timings…