Assessment methods: interviews, self-report, informants, observation and physiology

Transcript

RyanA school psychologist sits at the front of a second-grade classroom with a clipboard. She's there to observe a boy who keeps leaving his seat. A second trained observer uses the same clear coding criteria, and their records agree closely.

SoheeAnd for all three visits, the boy stays in his seat.

RyanMeanwhile, on days without visitors, his teacher's records show him out of his seat again and again.

SoheeWelcome to EPPP Prep, the podcast. I'm Sohee.

RyanI'm Ryan. Today is all about methods. The knowledge statement lists them: self-report, multi-informant reports, psychophysiological measures, work samples, assessment centers, direct observation, and structured and semi-structured interviews. For each, we want its strengths and its limits. And that second-grader is our first lesson.

SoheeOpenStax puts the problem plainly: when people know they are being watched, they are less likely to behave naturally. Their everyday example is driving down an empty highway, versus driving the same road with a police car behind you.

RyanSo observers should be as unobtrusive and inconspicuous as possible. A clipboard at the front of the room is the opposite.

SoheeNow, the exam twist. Why isn't the answer observer bias? Observer bias is when observers unconsciously skew what they record to fit their expectations. But in this story, two trained observers used clear criteria and agreed closely.

RyanWhich is exactly the protection OpenStax recommends against observer bias: clear criteria, and comparing multiple observers to test inter-rater reliability. And the problem hasn't resolved, because it continues whenever no observer is there. So the answer is the boy behaving differently because he knew he was being watched.

SoheeWhen it's done well, though, observation has a big strength. The textbook says the greatest benefit of naturalistic observation is the validity, or accuracy, of information collected unobtrusively in a natural setting.

RyanAnd the textbook names the downside too: naturalistic observation is often difficult to set up and control. Its example is a researcher waiting all day in a restroom to record hand washing, and nobody comes in.

SoheeSo observation buys you realism, and costs you time and control. Every method in this episode has a trade like that.

RyanLet's move to interviews, which OpenStax covers in the context of hiring. In an unstructured interview, the interviewer may ask different questions of different candidates, and responses generally aren't scored on a standard system.

SoheeIn a structured interview, the interviewer asks the same questions of every candidate, the questions are prepared in advance, and there's a standardized rating system for each response. That lets the interviewer accurately compare candidates.

RyanAnd the payoff: in a meta-analysis by McDaniel and colleagues in 1994, structured interviews were more effective at predicting later job performance.

SoheeWhy does structure help? OpenStax notes that social factors and body language can sway interview outcomes, things like how similar the applicant is to the interviewer, and nonverbal behaviors like hand gestures, head nodding and smiling.

RyanLet's pause and think. A hospital's nurse managers interview applicants informally, each asking whatever questions occur to them and writing a general impression afterward. Their hiring decisions have predicted later job performance poorly. What should a consulting psychologist recommend?

SoheeA structured interview. The same prepared questions for every applicant, and each answer rated on a standard system.

RyanAnd watch for the distractors. An interview rated mainly on likability and first impressions. Or a personality test with no evidence linking its scores to job performance. OpenStax says any use of personality tests should come with a verified assessment of which scores relate to good performance.

SoheeIn clinical work, you'll also meet the semi-structured interview, which sits in between. Yu and colleagues describe the Autism Diagnostic Interview, Revised, as a comprehensive, semi-structured caregiver interview, used as part of a comprehensive evaluation along with other measures.

RyanNext, self-report. OpenStax says self-report inventories are generally simple to give and cost effective. That's their strength.

SoheeTheir weakness is the person filling them out. Test takers may answer in ways that are more socially desirable, exaggerated, or misleading, on purpose or not. A job applicant will likely try to present themselves in a positive light, perhaps as an even better candidate than they are.

RyanAnd from the textbook's discussion of surveys: people don't always give accurate responses. They may lie, misremember, or answer in a way they think makes them look good. People may report drinking less alcohol than they really do.

SoheeWhich is a big reason to gather information from more than one person. Yu's team says a comprehensive autism evaluation should include informant reports from various sources, such as caregiver, teacher and self-report, plus direct observation by a trained provider.

RyanSAMHSA, the federal Substance Abuse and Mental Health Services Administration, describes the same approach for attention deficit hyperactivity disorder. There's no single test. Clinicians may observe the patient, use rating scales, conduct psychological tests, and interview or collect ratings from relatives, teachers and others in the person's life.

SoheeOrganizations have their own multi-informant method: the 360-degree feedback appraisal. OpenStax describes ratings from supervisors, peers, the people the employee supervises, the employee herself, and sometimes outside observers like customers.

RyanAnd here's the honest limitation. Few studies have tested 360-degree methods. One study the textbook cites found self and peer ratings were unreliable as an assessment of performance. Another view treats that disagreement as useful, because discussing the discrepancies helps employees learn.

SoheeSo multiple informants don't automatically agree, and that's informative. The disagreement is data.

RyanAnd notice how the autism evaluation we described combines methods on purpose: a caregiver interview, reports from several informants, and direct observation. Each method covers some of the others' blind spots.

RyanTwo more methods from the workplace. OpenStax says candidate evaluations can involve testing, an interview, and work samples or exercises. And one method built around exercises is the assessment center.

SoheeBuckett and colleagues describe it this way: multiple trained assessors observe candidates across multiple behavioral simulation exercises, in a standardized way, and rate their performance on several dimensions. In their study, the exercises included a role-play with an underperforming subordinate, a group problem-solving exercise, and an in-basket exercise of emails to handle.

RyanAnd the evidence? Meta-analyses have consistently supported the criterion-related validity of assessment center ratings, and there's evidence they improve prediction beyond cognitive ability and personality. But support for the dimensions they claim to measure remains mixed.

SoheeSo the ratings do predict outcomes, even though it's less clear exactly which skills they capture.

RyanLast, psychophysiological measures, and the famous one: the polygraph. OpenStax says it measures physiological arousal, like increased heart rate, breathing rate and sweating, while a person answers a series of questions.

SoheeAnd the verdict is blunt. While polygraphs are still commonly used, their validity and accuracy are highly questionable, because there is no evidence that lying is associated with any particular pattern of physiological arousal.

RyanSo when a question says an experienced examiner found arousal spiked on the key question, the experience doesn't fix the core problem. And arousal being hard to control on purpose doesn't make it a sign of lying either.

SoheeSecond pause and think. In a custody dispute, a mother's attorney submits a polygraph report showing that the father's pulse and breathing rose sharply when he denied hitting the children. What's the most accurate interpretation?

RyanThe arousal can't show he lied. No particular pattern of arousal has been shown to go with lying, so the report can't settle the question.

SoheeSame logic in a hiring version from our practice set. A security company requires applicants to take a polygraph. One applicant's heart rate and sweating jump when he's asked about past drug use, and the veteran examiner labels him deceptive. A psychologist reviewing that practice should point out that arousal isn't a specific sign of lying, however seasoned the examiner.

SoheeLet's recap. Direct observation: people behave less naturally when they know they're watched, so stay unobtrusive. Clear criteria and inter-rater checks guard against observer bias.

RyanInterviews: structured interviews, with the same prepared questions and standardized scoring, predict job performance better than unstructured ones. Semi-structured interviews, like the caregiver interview used in autism evaluations, sit in between.

SoheeSelf-report is simple to give and inexpensive, but open to socially desirable answers and inaccurate recall. Multiple informants and 360-degree ratings add perspectives, and their disagreements are informative.

RyanAssessment center ratings have consistent support for criterion-related validity, though support for the dimensions they claim to measure is mixed. And the polygraph measures arousal, not lying.

SoheeThat was a lot of methods in one sitting. If they're blurring, pick two that confuse you and compare their strengths side by side tonight. Small steps add up.

RyanAnd remember that second-grader. Sometimes the most useful insight in an assessment is noticing how your own presence changed what you saw. See you next time on EPPP Prep.

Loading the read-along timings…