Fair testing across groups, cultures and languages

Transcript

RyanA medical school admission test in the Czech Republic. One biology item asks about childhood illnesses. And when the test developers run their statistics, women do better on that item than men.

SoheeSo, is the item biased?

RyanContent experts looked closely. They concluded women did better because, in the Czech Republic, women spend more time with children, and so have more experience with childhood illnesses.

SoheeAnd the faculty kept the item. They decided it was fair, because doctors need to know about childhood illnesses.

RyanWelcome to EPPP Prep, the podcast. I'm Ryan.

SoheeI'm Sohee. That story comes from an article by Martinková and colleagues, and it's the perfect doorway into today's topic: fairness. How we detect possible bias in a test item, why the same test can mislead in a new culture, and how to assess someone in a language you don't share.

RyanThe statistical tool in that story is differential item functioning. The idea is to test whether people who are matched for ability, but come from different groups, perform similarly on a given item.

SoheeMatched is the key word. For example, you can match people on their total score. Then you ask, among people with the same total score, does one group still do worse on this particular item?

RyanAnd here's why item-level analysis matters. The article makes two points that sound backwards at first. A significant difference between groups on total scores can exist even when the items are not biased.

SoheeIn their own data set, men outscored women on total score, but no items showed differential functioning. They concluded the gap reflected real differences in understanding of the concept, not item bias.

RyanAnd the second point. Item bias can exist even when two groups have exactly the same distribution of total scores. Comparing totals alone can miss it.

SoheeThen there's the lesson of the Czech story. The presence of differential item functioning is a signal that an item may be biased, but it does not guarantee the item is unfair. Flagged items get reviewed by content experts, who decide whether the difference reflects the concept being tested, or something unrelated like cultural, curricular, or language knowledge.

RyanLet's pause and think. A state science test is reviewed for fairness. English learners and native English speakers have nearly identical total-score distributions, and the items correlate well with one another. Most students answer item 14 correctly, and it uses an idiom unrelated to science. But among students with the same total score, English learners get it right much less often. What is this?

SoheeDifferential item functioning. Matched on total score, different groups, different performance on one item.

RyanAnd look at how the stem closes the other doors. Total-score distributions are about the same, so it isn't a group difference in total scores. The items correlate well, so it isn't low internal consistency. Most students get item 14 right, so it isn't a hard item for everyone.

SoheeAnd what happens next is a content review. Unlike the childhood illnesses item, an idiom unrelated to science isn't part of what a science test is meant to measure, so that item is a much stronger candidate for revision.

SoheeNow zoom out from one item to a whole test. OpenStax describes a 1979 case, Larry P. versus Riles. Parents of African American students challenged the testing method the State of California was using, because they believed it was culturally unfair: the tests were normed and standardized using White children.

RyanThe textbook quotes a summary of the case. The state had used standardized intelligence tests that were racially and culturally biased, and that had not been validated for placing Black children in isolated, stigmatizing classes for the so-called educable mentally retarded. That's an outdated label; the accepted term now is intellectual disability. Many students were incorrectly classified.

SoheeNorms are a recurring problem. O'Donald and colleagues, writing about children's neuropsychological testing, say assessments need careful adaptation rather than direct translation. They note that many translated versions of the Wechsler Intelligence Scale for Children lack independent validation.

RyanAnd norms don't always travel even between similar countries. Their example: Canadian children assessed using the United States norms tend to score slightly higher on non-verbal reasoning tasks.

SoheeEducation and literacy matter too. Franzen and colleagues report that illiteracy, a limited number of years of education, and low quality of education significantly affect neuropsychological test scores across several cognitive domains.

RyanSo a low score might reflect someone's schooling rather than their brain. Which is why culture and education have to be part of interpretation, not an afterthought.

SoheeCulture shapes the testing session itself, too. Franzen's group notes that a patient may have different expectations about the purpose of the assessment, about what counts as relevant information, and about what may be shared with a stranger.

RyanAnd it isn't only cognitive tests. OpenStax reports that traditional projective tests were used for decades in cross-cultural personality assessment, but test bias limited their usefulness. When the Thematic Apperception Test was used with African American test takers, the result was often shorter stories and low levels of cultural identification.

SoheeThat's why culturally specific tests were developed, like the Contemporized-Themes Concerning Blacks Test, and TEMAS, designed to be culturally relevant to minority groups, especially Hispanic youths. We'll meet both properly in our episode on personality tests.

SoheeOkay, translation. Say you need a depression measure in a language it was never written in. What do guidelines recommend?

RyanCruchinho and colleagues reviewed the guidelines. The fastest and cheapest approach is a one-way translation by a bilingual person. But forward and back translation is the most recommended method. It requires at least two independent translators: one translates into the new language, and the other translates that version back into the original.

SoheeAnd why back-translate? Tsang and colleagues explain that misunderstandings or unclear wording in the first translation may be revealed in the back-translation.

RyanIt doesn't stop there. Cruchinho's group proposes an eight-step guideline: forward translation, synthesis of translations, back translation, harmonization, pre-testing, field testing, psychometric validation, and analysis of psychometric properties.

SoheeAnd Tsang suggests pilot testing the prefinal version on a small sample, about 30 to 50 of the intended respondents, asking them what they thought each item meant.

RyanSo in an exam question, the tempting shortcut is one bilingual staff member translating the test and keeping the original norms. The better answer is the multi-step route ending in validation.

SoheeNow, testing through an interpreter. Franzen's expert panel said it's critical to have an interpreter present when the patient has little understanding of the examiner's language. And they agreed formal interpreters are generally preferable to informal ones, like relatives, because of concerns about interpretation quality and the bias informal interpreters can introduce.

RyanWhat else did they recommend?

SoheeA briefing with the interpreter before the assessment, covering the aims, the case, the instruments, and the procedures. Interpreters should translate as literally as possible, give no hints, and use the official translated instructions instead of interpreting freely. And testing with a telephone interpreter should be avoided, mostly because the interpreter can't see the test materials.

RyanOne detail I love from that briefing list: the patient failing items is a vital part of the examination. A kind interpreter who helps the patient succeed is, without meaning to, ruining the test.

SoheeAnd a caution from O'Donald's team: conversational fluency doesn't always mean someone is proficient in the language needed for cognitive testing.

RyanEven dialect matters. The panel's list includes awareness of regional variation in language. Their example: it isn't ideal to use a Spanish interpreter from Spain to assess Latin American patients.

RyanSecond pause and think. On a rehabilitation unit, a psychologist must assess memory and attention in a Tigrinya-speaking man after a stroke. He understands very little English. His nephew offers to help, and a phone interpreter could join today. A trained interpreter can come tomorrow, and the testing can wait a day. Best choice?

SoheeThe trained interpreter, in person, tomorrow, briefed beforehand. Relatives are informal interpreters, and a phone interpreter can't see the materials.

RyanLet's recap. Differential item functioning compares groups matched on ability, item by item. Totals can differ without item bias, and items can be biased even when totals are identical.

SoheeA flagged item isn't automatically unfair. Content experts decide whether the difference reflects what the test is supposed to measure.

RyanNorms from one population may not fit another, and education and literacy shape scores.

SoheeTranslate forward and back with independent translators, then pre-test, field test and validate. And use a formal, in-person, briefed interpreter who translates literally from official instructions.

RyanThis topic matters beyond the exam. It's about making sure the people we assess are measured fairly. If you've ever felt a test didn't capture what you know, you understand exactly why it matters.

SoheeBring that understanding to your practice questions this week. See you next time on EPPP Prep.

Loading the read-along timings…