Inside the test: item analysis, item response theory and testing by computer

Transcript

SoheeA psychology instructor is going through her midterm results, question by question. And on one question she finds something strange. The students in the lowest-scoring group picked the keyed answer more often than the students in the highest-scoring group.

RyanThe strongest students got it wrong, and the weakest got it right.

SoheeWhich tells her something is wrong with the question itself. She decides to rewrite it before she ever uses it again.

RyanWelcome to EPPP Prep, the podcast. I'm Ryan.

SoheeI'm Sohee. Today we're going inside the test, one question at a time. How test builders decide whether an item is pulling its weight, what item response theory adds, and how computers now pick questions on the fly.

RyanThat instructor was using item analysis. Let's start with the first number she'd look at, the item difficulty index. In the item analysis article we're drawing on, it's calculated as the percentage of examinees who answer the item correctly.

SoheeAnd here's the twist that trips people up. Because it's the percent who get it right, a high difficulty index means more people got it right. The article even lists easiness as another name for the same statistic.

RyanSo the number runs backwards from its name. That article treats items at or above 78 percent correct as the least difficult, items between 25 and 78 percent as acceptable, and items under 25 percent as difficult.

SoheeThen the second number, the one our instructor really needed. The discrimination index. The article defines it as an item's ability to discriminate between higher and lower achievers, using the top and bottom 27 percent of total scorers.

RyanSo you rank everyone by total score, then ask, on this one item, did the stronger group outperform the weaker group?

SoheeExactly. The value runs from minus one to plus one. Positive values mean the item separates the groups in the right direction. In that article's scheme, point three five or higher counts as excellent. And a negative value, like our midterm question, means the item is non-discriminating.

RyanLet's pause and think. A hospital education office gives nurses a 50-item medication-safety quiz. On one item, the nurses with the highest total scores and the nurses with the lowest total scores answered correctly at almost exactly the same rate. Which statistic captures that?

SoheeThe discrimination index. When high and low scorers succeed at the same rate, the item isn't telling them apart.

RyanAnd the tempting wrong answer is the difficulty index. Remember, difficulty is one overall percent correct for everybody. It never compares high scorers with low scorers.

SoheeAnother wrong answer in that question is internal consistency. That describes how the items of a whole test correlate with one another. It's a property of the test, not of one item.

RyanThere's a third piece of item analysis, about the wrong options. The article says distractors should look similar to the keyed answer and reflect a misconception. A functioning distractor is chosen by more than 5 percent of examinees.

SoheeAnd an option chosen by fewer than 5 percent is a non-functional distractor. Students can spot and eliminate it, so it makes the item less difficult and affects how well it discriminates.

RyanSo a four-option question with two silly options is really a two-option question.

SoheeNow, those statistics come from classical test theory. Seo's review of adaptive testing notes that classical test theory showed several limitations, which is why item response theory has been applied to licensing and certification exams since the 1970s.

RyanGive us item response theory in plain words.

SoheeEach item gets its own parameters. There's a difficulty parameter, called b. Items with b near minus two have very low difficulty, and items near plus two are very difficult. There's a discrimination parameter, called a, which is the slope of the item's curve. And in the three-parameter model there's a guessing parameter, called c.

RyanSo the same three ideas, difficulty, discrimination and guessing, but now built into a model of how ability relates to getting the item right.

SoheeAnd here's the payoff. Item response theory places item difficulty and candidate ability on the same scale. The review says that makes it possible to compare candidates on the same scale even when they took different tests.

RyanWhich is exactly what you need if every candidate is going to get a different set of questions. Enter computerized adaptive testing.

SoheeThe review lists five components: an item bank, a starting item, an item selection rule, a scoring procedure, and a termination criterion.

RyanAnd the heart of it is the selection rule. Given the current estimate of a candidate's ability, the most informative remaining item is chosen next. The estimate is updated after each answer, so the next item is always the one that tells you the most at the candidate's latest estimated level.

SoheeWhen does it stop? One option is to stop when the standard error of the ability estimate reaches a pre-set value. In other words, when the score is precise enough.

RyanAnd the big advantages. Adaptive testing yields a high and equal degree of precision for all candidates, and needs fewer items than a conventional test to reach a given precision. The review says studies have repeatedly found adaptive tests to be, on average, 50 percent shorter than paper-and-pencil tests, with equal precision.

SoheeIt's been around a while, too. According to the review, exams for nurse candidates in the United States have been given by adaptive testing at test centers since 1994.

RyanBut it isn't free. The review names the cost of building the computer platform and the large item bank. Items have to be pretested, calibrated, and linked before they're used.

SoheeAnd there's a security wrinkle. Every test starts somewhere, and if every candidate starts with the same middle-difficulty item, that item gets overexposed. One remedy is to pick the first few items at random from a subset of the bank.

RyanSecond pause and think. A hospital wants to measure patients' fatigue with as few questions as possible, because the patients tire quickly. It needs scores equally precise for mildly and severely fatigued patients. It has a large bank of fatigue items calibrated with item response theory, and tablets. What's the best approach?

SoheeComputerized adaptive testing. Tailor each question to the patient's current estimated level.

RyanAnd the trap in that question: putting a single fixed form on a tablet. Delivering a test by computer doesn't make it adaptive. Everyone still gets the same items. A shorter fixed form saves time but gives up precision, and a longer one adds the very items you were trying to cut.

SoheeNow let's widen the lens, because the knowledge statement on technology asks about validity, cost-effectiveness and consumer acceptability. Telehealth assessment is a live example.

RyanYu and colleagues reviewed autism assessment, and they describe how the shift to telehealth during the pandemic forced modifications to standardized testing. Many gold-standard tools rely on seeing eye contact and facial expressions. Face masks raised questions about the validity of widely used tools.

SoheeAnd they're candid that the impact of those modifications on validity remains to be fully determined. That's the validity question in one sentence. Change the way a standardized test is given, and you can't assume the old evidence still holds.

RyanOn acceptability, the early news was encouraging. For one tele-assessment tool, preliminary data suggested most providers felt comfortable making diagnoses afterward, and all providers valued observing children and families at home and relieving families of the burden of travel. The challenges were minor and infrequent: things like technological issues, dropped calls and inconsistent audio.

SoheeAnother protocol, for infants, showed positive caregiver feedback in its initial study.

RyanLet's recap. The difficulty index is the percent who answer correctly, so a higher number means more people got it right.

SoheeThe discrimination index compares the top and bottom scorers on one item. It runs from minus one to plus one, and a negative value means low scorers beat high scorers on that item.

RyanDistractors that almost nobody picks aren't doing their job.

SoheeItem response theory gives each item a difficulty and a discrimination parameter, plus a guessing parameter in the three-parameter model, and puts items and people on one scale.

RyanAdaptive testing picks the most informative next item for your current ability estimate. It reaches equal, high precision with fewer items, but it needs a large calibrated bank and a costly platform.

SoheeAnd any new technology raises three questions: is it still valid, is it worth the cost, and will people accept it.

RyanThis was a technical episode, so if your head is full, that's a sign you were working, not failing. Take a short break before you do anything else.

SoheeThen try a few item analysis questions, and for each one, ask yourself: is this one number for everyone, or a comparison of high and low scorers? You've got this. Talk soon.

Loading the read-along timings…