Measuring outcomes: needs assessment, program evaluation and reliable change

Transcript

SoheeA public health clinic's vaccination rate is well below its goal. The director calls in a psychologist and says: build us a training course for the nurses.

RyanAnd nobody has looked at why the rate is low.

SoheeMaybe the nurses need training. Or maybe it's the scheduling system, or a policy, or the hours the clinic is open. Building the course first is answering a question no one has asked yet.

RyanWelcome to EPPP Prep, the podcast. I'm Ryan.

SoheeI'm Sohee. This is our last Domain 5 episode, and we're zooming out from one test-taker to groups, programs and organizations. Then we'll zoom back in, to ask how you know whether one client really changed.

RyanLet's start with that clinic. The Centers for Disease Control and Prevention's training guidance says a needs assessment identifies gaps in knowledge or skills within the workforce, and helps you determine if training is necessary. It's the first of eight CDC Quality Training Standards.

SoheeAnd it's usually driven by a perceived gap in reaching a program goal. Their own worked example is about exactly our situation: the program goal is to vaccinate 70 percent of the population, and the rates are lower than that. The evaluation question becomes, why are vaccination rates low?

RyanThe guidance says to focus on understanding the current situation, not solutions. And to describe potential causes of the gap in terms of both people and systems.

SoheeIf the analysis shows a need to change knowledge, skills or behavior, there's a performance gap, and training can be a solution. But other solutions might fit better: changes in policy, technology, organization, or standard operating procedures.

RyanLet's pause and think. Medication errors have risen on a hospital unit, and the chief nursing officer asks a psychologist to build a refresher training on medication safety. The cause hasn't been studied, and a new electronic ordering system was installed this year. What should the psychologist do first?

SoheeConduct a needs assessment, to find the gap and what's causing it. That new ordering system is a flashing light. The errors could come from the system, not from what nurses know.

RyanAnd the guidance lists the ways to collect that information: reviewing existing data, interviews, surveys or focus groups. If you build new data collection tools, it says to pilot test them with the intended audience first. The same habits we've talked about all series.

RyanAnd the wrong answers assume the cause. Develop the training leadership asked for. Pilot the training and collect reaction ratings. Or screen staff with a personality test to find the poor performers, which assumes the problem lives inside individual workers.

SoheeNow, once a program exists, how do we evaluate it? CDC frames program evaluation around a handful of questions. Are activities being completed as planned? Is the program achieving what was intended? Did the outcomes happen because of the program? Is it using the appropriate resources? And it applies the same types of evaluation to programs, policies and whole organizations.

RyanFormative evaluation is typically done to assess whether a program is feasible, appropriate and acceptable before it's fully implemented.

SoheeProcess, or implementation, evaluation assesses how well implementation followed the original plan. It often includes the content, quality, quantity and structure of what was delivered.

RyanOutcome evaluation measures how well the program achieved its intended outcomes. And here's the line to underline: it cannot determine what caused the outcomes, only whether they were achieved.

SoheeImpact evaluation compares outcomes with estimates of what the outcomes would have been without the program. It usually seeks to determine whether the program's activities caused the observed outcomes.

RyanAnd economic evaluation examines the program's effects relative to its costs, using approaches like cost-benefit analysis.

SoheeCDC also separates evaluation from performance measurement. Performance measurement is the ongoing monitoring and reporting of accomplishments, especially progress toward pre-set goals. It can show performance rising or falling. Evaluation helps identify the reason behind those changes.

SoheeSecond pause and think. A school district has delivered a 12-session bullying-prevention curriculum in every middle school for a year. Before examining bullying rates, the evaluator checks how many sessions each class received, whether teachers covered the required content, and how closely delivery matched the plan. What kind of evaluation?

RyanProcess evaluation. Sessions delivered, content covered, fidelity to the plan. Not formative, because the program is already fully running. Not outcome or impact, because no results are being measured yet.

SoheeThird pause and think. Vaping among students fell after a district introduced a prevention curriculum, but vaping also fell in many other districts over the same period. The board wants to know whether its curriculum caused the drop. What does the evaluator need?

RyanAn impact evaluation, comparing outcomes with an estimate of what would have happened without the curriculum. Because vaping fell elsewhere too, showing that it fell here proves nothing about cause.

SoheeAnd the tempting distractors: document that the program hit its target, which is outcome evaluation. Confirm activities were delivered as planned, which is process evaluation. Or survey participants about satisfaction, which tells you how they felt, not what the program caused.

RyanOrganizations also assess people, through performance appraisal. OpenStax notes that appraisals can identify opportunities for training, or whether a particular training program has been successful. So personnel assessment can feed right back into needs assessment.

SoheeNow let's zoom all the way in. A client finishes 12 sessions of cognitive behavioral therapy, and her depression score has dropped several points. Did she really improve? Or is that wobble?

RyanGroup statistics can't answer that. Ferrer-Urbina and colleagues explain that the reliable change index was proposed for assessing individual change, as an alternative to the group-level answers you get from significance tests and effect sizes.

SoheeAnd they describe it as the most widely used index of individual change in pre-post designs in clinical settings. The best-known version comes from Jacobson and Truax, published in 1991.

RyanWhy not look at the raw difference, or a percentage change? O'Connell and colleagues explain that those change scores can reflect measurement error, or practice effects from taking the test before, in addition to real change.

SoheeHere's the link back to our very first episode. The reliable change index is built on the standard error of measurement, which depends on two things: how spread out the scores are, and how reliable the test is.

RyanSo a less reliable test has more error, and a client has to change more before that change counts as reliable. Reliability isn't an abstract property. It decides whether you can see progress.

SoheeGronchi and colleagues explain that computing the index is like computing a z-score. A value larger than 1.96, in either direction, represents real change, because it's unlikely to happen by chance at the usual 5 percent level.

RyanAnd there's a variant for repeated cognitive testing that also accounts for practice effects. That matters, because they note memory test performance is the most influenced by practice.

SoheeWhich is exactly the situation in one of our practice items. A neuropsychologist retests a patient on a memory measure six months after brain surgery and finds a lower score than before. Is the decline bigger than the test's measurement error could produce? That's a job for the reliable change index.

SoheeOne more distinction. Reliable change asks whether the change exceeds measurement error. Clinically significant change asks whether it matters: in one definition, whether the person moved from being more like a clinical population to being more like a non-clinical one, or the reverse.

RyanLet's recap. Needs assessment comes first: find the gap and its cause, in people and in systems, before choosing training.

SoheeFormative evaluation checks feasibility before launch. Process evaluation checks delivery against the plan. Outcome evaluation shows whether targets were met but not why. Impact evaluation compares with what would have happened without the program, to judge cause.

RyanFor one person, the reliable change index tests whether pre-post change exceeds measurement error, using the standard error of measurement. Beyond 1.96 counts as reliable. Clinically significant change, in one definition, means moving from looking more like a clinical population to looking more like a non-clinical one.

SoheeAnd that's the end of our Domain 5 series. Reliability, validity, base rates, bias, instruments, diagnosis, classification, models and outcomes. That is a lot of ground, and you covered it.

RyanSo here's a reliable change index for you, so to speak. Go back to a practice set you found hard a few weeks ago, and take it again. Whatever the result, you'll learn something real about where to focus next. Thank you for studying with us on EPPP Prep.

Loading the read-along timings…