Testing ethics and the Standards for Educational and Psychological Testing

Transcript

RyanA regional company's human resources director has found an online personality test. The vendor's website says it's validated. She asks a consulting psychologist to start using it to screen applicants for supervisor jobs.

SoheeAnd the psychologist's first question isn't, is it valid? It's, valid for what, and for whom?

RyanThat question is the whole episode. Welcome to EPPP Prep, the podcast. I'm Ryan.

SoheeAnd I'm Sohee. Today's knowledge statement is professional standards and guidelines, and the example the exam blueprint itself names is the standards for educational and psychological testing. We'll pair that with the Ethics Code's own standards on assessment.

RyanThe Standards for Educational and Psychological Testing come from three organizations working jointly: the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. The edition we draw on today is the 2014 one.

SoheeLet's start with how the Standards define a test. A test is a sample of behavior gathered to draw an inference about some domain or construct within a particular population.

RyanWithin a particular population. Hold onto that. Now validity. The Standards define it as the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests.

SoheeTwo big ideas live in that sentence. First, validity isn't a property of the test sitting on a shelf. It's about the inferences you draw from the scores, for a particular use.

RyanSecond, it's a matter of degree. Not valid or invalid, but more or less supported. As one primer on the Standards puts it, a test can't be stamped valid for all purposes and test-taker populations.

SoheeWhich is exactly the human resources director's problem. Validated tells you nothing until you know for which use, and with which group.

RyanHere's another shift worth knowing. Older validity theory talked about separate types of validity, like content validity and criterion validity. Contemporary theory treats validity as one unitary concept, supported by different kinds of evidence.

SoheeAnd the Standards name five sources of that evidence. Evidence based on test content. On response processes. On internal structure. On relations with other variables. And on the consequences of testing.

RyanLet's make each one concrete. Test content is about whether the items represent the domain the test is meant to measure. A common way to gather it is review by subject-matter experts, or building the test from a table of specifications, a test blueprint.

SoheeResponse processes ask whether test takers are actually doing the thinking the construct calls for. That evidence is most often gathered with think-aloud procedures, where people explain their reasoning as they answer.

RyanInternal structure asks whether the relationships among items fit the construct. If a test claims to measure two separate skills, techniques like factor analysis check whether the items really cluster that way.

SoheeRelations with other variables is the one most people remember. Scores should correlate with measures of the same or similar constructs, which is convergent evidence, and less with measures of different constructs, which is discriminant evidence.

RyanAnd consequences of testing covers the intended and unintended effects of using and interpreting the scores.

SoheeThe Standards also name two main threats to validity. Construct underrepresentation is when a test fails to capture important aspects of the construct.

RyanAnd construct-irrelevant variance is when scores move for reasons other than the construct. The primer's examples: English skills affecting scores of non-native English speakers on a test written in English, or computer skills affecting a test given by computer.

SoheeAnd reliability? The primer describes it simply as test score consistency, a separate dimension of quality alongside validity.

RyanNow watch how the Ethics Code lines up. Standard 9.02b: psychologists use assessment instruments whose validity and reliability have been established for use with members of the population tested.

SoheeAnd when that hasn't been established, they describe the strengths and limitations of the test results and interpretation. Standard 9.02c adds that assessment methods should fit the person's language preference and competence, unless an alternative language is relevant to the assessment.

RyanPause and think. The HR director's test was validated only with college students. The applicants are experienced workers. What does the psychologist owe her?

SoheeValidity evidence for this use, with this population. If it hasn't been established, the psychologist describes the strengths and limitations of the results. Validated, on its own, isn't the answer.

RyanAnd employment testing has a federal layer. The Uniform Guidelines on Employee Selection Procedures say their validation provisions are meant to be consistent with generally accepted professional standards, naming the Standards for Educational and Psychological Tests.

SoheeThose Guidelines still accept criterion-related, content, or construct validity studies, the older vocabulary. We'll come back to the Guidelines' four-fifths rule in the next episode.

RyanBack to the Code. Standard 9.03 is informed consent in assessment. It includes the nature and purpose of the assessment, fees, involvement of third parties, limits of confidentiality, and a chance to ask questions.

SoheeWith three exceptions. Testing mandated by law or regulation. Consent that's implied because testing is a routine educational, institutional, or organizational activity, like voluntarily taking a test when applying for a job. And testing where one purpose is to evaluate decisional capacity.

RyanAnd when an interpreter is used, the psychologist obtains the client's consent to use that interpreter, and keeps test results confidential and the test secure.

SoheeStandard 9.01 covers the basis for opinions. Psychologists give opinions about someone's psychological characteristics only after an examination adequate to support them.

RyanIf that examination isn't practical despite reasonable efforts, they document their efforts, clarify how the limited information affects reliability and validity, and limit their conclusions. And for a record review, they explain the sources they relied on.

SoheeNow the distinction the exam loves: test data versus test materials. Under Standard 9.04, test data means raw and scaled scores, the client's responses to test questions or stimuli, and the psychologist's notes on the client's statements and behavior during the exam.

RyanTest materials, under Standard 9.11, are manuals, instruments, protocols, and test questions or stimuli. Psychologists make reasonable efforts to keep those secure.

SoheeWith a client's release, psychologists provide test data to the client or to the other persons identified in the release. They may refrain to protect the client or others from substantial harm, or to prevent misuse or misrepresentation of the data or the test.

RyanWithout a release, test data go out only as required by law or court order.

SoheePause and think, number two. A client signs a release naming her attorney. The attorney asks for the test data and a copy of the test manual. What goes?

RyanThe test data go to the attorney, because the release names him. Standard 9.04 doesn't limit release to psychologists. The manual is a test material, so it stays secure under 9.11.

SoheeAnd notice the tempting wrong answers on that kind of item. Release only to a psychologist qualified to interpret the data. Send the manual and stimuli along with the data. Or hold everything until a court order arrives.

RyanEach one misreads the standard. The release controls who receives test data. A court order matters only when there's no release. And materials are a separate category with their own protection.

SoheeThree more standards round it out. Standard 9.06: when interpreting results, including automated interpretations, psychologists consider the test's purpose and the person's situational, personal, linguistic, and cultural characteristics, and they note significant limitations.

RyanStandard 9.09: psychologists who use scoring and interpretation services, including automated ones, retain responsibility for how the instruments are applied, interpreted, and used. The software doesn't take the responsibility for you.

SoheeA related one is Standard 9.07. Psychologists don't promote the use of psychological assessment techniques by unqualified persons, except for training purposes with appropriate supervision.

RyanStandard 9.08: no decisions based on test results that are outdated for the current purpose, or on obsolete tests. And Standard 9.10: explanations of results are given, unless the relationship precludes it, as in some preemployment or forensic evaluations, and the person was clearly told that in advance.

SoheeLet's recap. Validity is the degree to which evidence and theory support score interpretations for a proposed use. It's a matter of degree, and a test is never valid for all purposes and populations.

RyanFive sources of evidence: content, response processes, internal structure, relations with other variables, and consequences. Two threats: construct underrepresentation and construct-irrelevant variance. Reliability is score consistency.

SoheeIn the Code: use tests validated for the population, or describe limitations. Get informed consent, except when testing is mandated, routine, or evaluates decisional capacity.

RyanTest data go where the client's release says, even to non-psychologists. Test materials stay secure. And psychologists keep responsibility for interpretations, even automated ones.

SoheePsychometrics can feel like the driest corner of the exam. But you just built it around one human question: what can these scores honestly tell us about this person?

RyanHold onto that question and these items get much friendlier. You're doing good work. Thanks for listening to EPPP Prep.

Loading the read-along timings…