Reliability and validity: consistent is not the same as correct

Transcript

RyanHere's a small experiment you could do in your kitchen tomorrow morning. Pour your usual bowl of cereal onto a kitchen scale and note the number. Pour it back, weigh it again. Same number. Do it a third time. Same number.

SoheeSounds like a great scale.

RyanUnless nobody ever calibrated it. Then it can hand you the same wrong number, every single time.

SoheeAnd that bowl of cereal holds the whole difference between reliability and validity. Welcome to EPPP Prep, the podcast. I'm Sohee.

RyanAnd I'm Ryan. This is our first episode on Assessment and Diagnosis, the domain that makes up 16 percent of the exam. And full credit where it's due: the cereal and the miscalibrated scale come straight from the OpenStax psychology textbook.

SoheeLet's start with the definitions, because exam items lean on exact wording. OpenStax says reliability refers to the ability to consistently produce a given result. For a psychological measure, that means it collects data in consistent, reproducible ways.

RyanAnd validity refers to the extent to which a tool accurately measures what it's supposed to measure. So reliability asks, do I get the same answer again? Validity asks, is it the right answer?

SoheeThe textbook's own warning is worth memorizing: being consistent in measurement does not necessarily mean that you have measured something correctly. That's the kitchen scale.

RyanWhich leads to the one relationship you want to know cold. In the textbook's words, any valid measure is by necessity reliable, but the reverse is not necessarily true.

SoheeThink about why. If a scale truly captures someone's level of worry, its scores can't bounce around at random while their worry stays put. Accuracy requires consistency. But consistency alone doesn't buy you accuracy.

RyanSo an answer choice claiming a test is valid but unreliable should make you suspicious. A reliable but invalid test, on the other hand, is the miscalibrated scale.

SoheeNow, reliability comes in several kinds, and questions love to make you choose among them. OpenStax names three of them. Test-retest reliability is the degree to which the outcomes of a particular measure remain consistent over multiple administrations.

RyanSame test, same people, two different times, and then you compare the two sets of scores.

SoheeInter-rater reliability is the degree to which two or more different observers agree on what has been observed.

RyanSo that one needs human judges. Two clinicians coding the same recorded session. Two observers rating the same playground.

SoheeAnd internal consistency is the degree to which different items on a survey that measure the same thing correlate with one another. It's usually estimated with Cronbach's coefficient alpha.

RyanOne form, one sitting, and the question is whether the items hang together.

SoheeHere's a pattern from our own practice questions. When the stem says the test is scored by a computer, inter-rater reliability drops out, because there's no second rater to disagree. And when the stem describes two separate administrations, internal consistency drops out, because it looks within a single sitting.

RyanLet's pause and think. A psychologist builds a 20-item, computer-scored self-report scale of chronic worry, a trait she expects to stay stable over a few weeks. She gives it to 150 outpatients, gives the identical scale to the same patients three weeks later, and correlates the two sets of scores. What is she estimating?

SoheeTest-retest reliability. Same measure, same people, two occasions, correlated.

RyanAnd notice that little phrase, a trait she expects to stay stable. It isn't decoration.

SoheeRight. Researchers who study change point out that test-retest reliability is reduced by the presence of true individual change between the two testings. If the thing itself shifts, a low correlation might be real change rather than a sloppy test. A stable trait keeps the comparison fair.

RyanAnd reliability usually shows up as a correlation. In another of our practice items, an industrial-organizational psychologist gives a computer-scored conscientiousness inventory to 300 employees, then gives it again two months later, and the two sets of scores correlate at point eight five. That number is a test-retest reliability coefficient.

SoheeThe flip side shows up too. A neuropsychologist on a rehabilitation unit notices that patients' scores on a brief, computer-scored attention test swing widely between two sessions a few days apart, even though nothing about their medical condition has changed. Her concern is weak test-retest reliability.

RyanBecause the person stayed the same, and the score moved anyway. That movement is measurement error.

RyanOkay, validity. The textbook lists several types. Let's start with the pair that gets confused most. Face validity is the degree to which a variable seems valid on the surface.

SoheeDoes it look like it measures the thing? Show it to a few people and they nod.

RyanConstruct validity is the degree to which a variable actually captures or measures what it is intended to measure.

SoheeAnd the gap between those two is where hard questions live. Picture a new 30-item test anxiety scale. Students agree the items look like they're about test anxiety. The items correlate strongly with one another. But scores correlate highly with a reading-speed test, and only weakly with established measures of test anxiety.

RyanSo it looks right, and it hangs together, but it seems to be measuring how fast people read. That's a construct validity problem.

SoheeAnd here's what makes the item tricky. Two of the wrong options are properties the scale already has. Face validity: the stem tells you the items look right. Internal consistency: the stem tells you they correlate well. The answer is the one property that's missing.

RyanThat's a habit worth building for the whole domain. When a stem goes out of its way to give you a property, it's usually ruling that option out.

SoheeTwo more validity terms. Ecological validity, in the textbook's definition, is the degree to which research results generalize to real-world applications.

RyanAnd OpenStax connects it to naturalistic observation. When people behave as they normally would in a given situation, you get a higher degree of ecological validity, or realism, than you might get with other research approaches.

SoheeThen there's predictive validity. The textbook's example is college admissions tests, where predictive validity refers to the test's ability to predict the grade point average of college freshmen.

RyanAnd it doesn't stop at the praise. It notes that some researchers assert these tests are biased, and that some research suggests their predictive validity is grossly exaggerated. So validity evidence, even for famous tests, gets debated.

SoheeSecond pause and think. A clinic adopts a new self-report depression questionnaire. Patients say the questions clearly seem to be about mood, and the items correlate highly with one another. Yet scores track patients' physical pain ratings closely and agree poorly with clinicians' structured assessments of depression. What's the most serious problem?

RyanConstruct validity. If the scores mostly reflect pain, the questionnaire isn't capturing depression, however much the items look like depression items.

SoheeAnd again, look at what the stem handed you. It looks right. It hangs together. Those are face validity and internal consistency, and they're there to tempt you.

RyanOne last link before we wrap, because inter-rater reliability isn't only a statistic. OpenStax notes that observers may unconsciously skew what they record to fit their expectations. To protect against that, researchers set clear criteria for what to record, and compare several observers' records of the same event to test inter-rater reliability.

SoheeSo reliability is also a safeguard. Agreement between independent observers is evidence that the record reflects the behavior, and not one observer's hopes.

RyanLet's pull it together. Reliability is consistency. Validity is accuracy: measuring what you meant to measure.

SoheeA valid measure has to be reliable. A reliable measure can still be invalid, like the scale that's consistently wrong.

RyanTest-retest: same test, same people, two times. Inter-rater: two or more observers agree. Internal consistency: the items on one form correlate with each other.

SoheeFace validity: it looks right. Construct validity: it truly captures the intended trait. Ecological validity: results carry over to real life. Predictive validity: scores forecast a later outcome.

RyanAnd the test-taking habit: notice which properties the stem already gives you, and pick the one that's missing.

SoheeIf psychometrics has ever felt like alphabet soup to you, you're in very good company. These terms are close cousins, and they take a few passes to settle. That's not a sign you can't learn them.

RyanAnd now you've got a cereal bowl to hang them on. Try a few reliability and validity questions while this is fresh, and say each definition out loud before you look at the options. See you next time on EPPP Prep.

Loading the read-along timings…