Samples, populations and gathering the data

Transcript

RyanPicture a researcher who wants to know how burned out graduate students in her state are. She posts a survey link in an online group where students vent about exhaustion. Hundreds answer within a day, and nearly everyone says they're running on empty.

SoheeAnd it's so tempting to write the headline. Graduate students in this state are burned out.

RyanThis is EPPP Prep, the podcast. I'm Ryan.

SoheeAnd I'm Sohee. Today we're on the very first decision in any study: who you study, and how you collect the information. The questions on this are very learnable once the vocabulary clicks.

RyanTwo words to start. The OpenStax psychology textbook defines a sample as a subset of individuals selected from a population. And the population is the overall group the researchers are interested in.

SoheeSo in our story, the population is every graduate student in the state. The sample is whoever answered that post.

RyanAnd why sample at all? The OpenStax statistics textbook says gathering information about an entire population often costs too much or is virtually impossible.

SoheeThe goal, in that text's words, is a sample with the same characteristics as the population it represents. A small mirror of the whole.

RyanSo what went wrong with our survey?

SoheeTwo things. First, the people who answered chose themselves. The statistics text warns that responses only from people who choose to respond, like call-in surveys, are often unreliable.

RyanSecond, she posted it where exhausted students gather. That's sampling bias, which the text defines as a sample where some members of the population are not as likely to be chosen as others.

SoheeWhich brings us to the big divide on the exam. Probability sampling versus nonprobability sampling.

RyanAn educational review by Elfil and Negida lays it out. In probability sampling, everyone in the target population has an equal chance of being selected. In nonprobability sampling, the process doesn't guarantee that equal chance.

SoheeAnd the review adds that probability samples are more representative of the target population. Let's walk through the four probability methods, because the exam loves to describe one and ask you to name it.

RyanFirst, the simple random sample. Every possible sample of the same size has an equal chance of being the one selected. Think names drawn from a hat, or a computer generating random numbers.

SoheeTo do that, you need a list of everyone in the population. The review calls that list the sampling frame.

RyanSecond, the stratified sample. You divide the population into groups called strata, then take a proportionate number from each stratum.

SoheeStrata might be age groups, gender, or diagnosis. Why go to the trouble?

RyanBecause the review points out that stratifying lets researchers get adequate samples from minority or underrepresented groups. With simple random sampling alone, a small group can stay underrepresented.

SoheeThird, the cluster sample. This is the one people confuse with stratified.

RyanIn the statistics text's definition of a cluster sample, you divide the population into clusters, randomly select some of the clusters, and everyone in the chosen clusters is in the sample.

SoheeSo here's the clean distinction. Stratified means you sample from every group. Cluster means you pick whole groups and skip the rest.

RyanThe review says cluster sampling is used when building a sampling frame is nearly impossible because the population is so large. Its example is primary school students across a whole country. Nobody has that list of students.

SoheeBut you can list the schools. The review's version randomly picks some schools, then randomly picks students inside them, instead of taking everyone. That adds a second stage of selection, which is why the review calls it multistage sampling.

RyanFourth, the systematic sample. You randomly choose a starting point, then take every so-many-th person on the list. Every tenth name, every fiftieth name.

SoheeAnd the statistics text notes systematic sampling is frequently chosen because it's a simple method.

RyanNow the nonprobability side. The most common is convenience sampling, which the statistics text describes as using results that are readily available.

SoheeThe review calls it the most widely used method in clinical research, because it's quick, inexpensive and convenient. And the statistics text gives the honest verdict: convenience results may be very good in some cases and highly biased in others.

RyanNext is judgmental sampling, where the investigators choose who's in, based on what they think the sample should look like. The review says it's widely criticized because of the likelihood of bias from the investigator's judgment.

SoheeAnd snowball sampling, for people who can't be found in one place. The investigator asks each participant to connect them with others from the same population.

RyanThe review's example is a survey of street children. There's no list, and no single school or clinic to visit. So the researchers reach one child, who helps them reach others.

SoheeHere's a trap the statistics text sets up beautifully. An assistant stands in front of the library one Wednesday and asks the first hundred students he meets. It feels random. The textbook's answer is convenience.

RyanRight, because students who weren't walking past the library that day had no chance of being chosen.

SoheeOkay, pause and think. Three quick ones, all from the OpenStax statistics text. A pollster interviews all the human resource staff at five different tech companies. What kind of sample?

RyanWhole groups, everyone inside them. Cluster.

SoheeA researcher interviews 50 public high school teachers and 50 private high school teachers.

RyanSampling from each group. Stratified.

SoheeAnd a medical researcher interviews every third cancer patient on a hospital's list.

RyanEvery so-many-th person from a list. Systematic. The textbook's answers match all three.

SoheeNow let's untangle two ideas that sound alike. Sampling error and sampling bias.

RyanThe statistics text says the process of sampling itself causes sampling error. A sample will never be exactly representative of the population, so there's always some. And as a rule, the larger the sample, the smaller the sampling error.

SoheeThink of it as the natural wobble of chance. The text says two random samples from the same population won't come out exactly the same.

RyanBias is a different animal. It comes from how people got into the sample. And the text is blunt: biased samples that aren't representative give results that are inaccurate and not valid.

SoheeSo a thousand more answers from that same exhausted-students group won't rescue the survey. The doorway people came through is the problem.

RyanOne more surprise from the statistics text. The sample size you need depends on the precision you want, not on the size of the population. It says a sample of 1,000 can be adequate for a population of 100,000 or of 1,000,000, at the same precision.

SoheeAnd the textbook itself admits that's counterintuitive. So be wary of any answer choice claiming a bigger population automatically needs a bigger sample.

RyanLet's shift to the second half of this knowledge statement, how the data get collected. The OpenStax psychology chapter walks through the main options, and each one has a trade-off.

SoheeSurveys first. The strength is reach. Surveys let researchers gather data from larger samples than many other methods.

RyanThe weakness is self-report. People don't always give accurate responses. They may lie, misremember, or answer in a way they think makes them look good. The textbook's example is people reporting drinking less alcohol than they really do.

SoheeThen naturalistic observation, observing behavior in its natural setting. The observer has to be inconspicuous, because when people know they're being watched, they're less likely to behave naturally.

RyanThe textbook's image is driving down an empty highway, versus driving with a police car right behind you.

SoheeThe payoff is what the text calls ecological validity, meaning realism. The cost is control. You have no control over when, or even if, the behavior you want to see will happen.

RyanAnd observation invites observer bias. Observers may unconsciously skew what they record to fit their expectations. The fixes are clear criteria for classifying each behavior, and comparing several observers to check inter-rater reliability, the consistency of observations by different observers.

SoheeFinally, archival research, which looks at past records or data sets for patterns. The researcher never interacts with a participant, so it costs far less time and money.

RyanThe catch is that you have no control over what was originally collected. The research question has to fit the records that already exist.

SoheeSo how does the exam test all this? Usually a short scenario and the question: which sampling method is this, or what is the main limitation of this approach?

RyanThe tempting wrong answers are calling a cluster sample stratified, calling a convenience sample random because the scene feels casual, and blaming a biased sample on ordinary sampling error.

SoheeLet's recap. A sample is a subset of the population. Probability sampling gives everyone an equal chance: simple random, stratified, cluster and systematic.

RyanStratified samples from every group. Cluster takes whole groups. Systematic takes every so-many-th person from a random start.

SoheeNonprobability methods are convenience, judgmental and snowball. Useful, practical, and more open to bias.

RyanSampling error is chance, and it shrinks as samples grow. Sampling bias comes from who had a chance to be chosen.

SoheeSurveys trade depth for reach and lean on honest self-report. Observation trades control for realism. Archival records trade control over what was measured for low cost.

RyanIf research methods has always felt like the dry corner of this exam, I hope today showed it's mostly a handful of clear distinctions. You can learn these, and they stay learned.

SoheeTry a few sampling questions while this is fresh, and name the method before you look at the choices. Thanks for spending this time with us on EPPP Prep.

Loading the read-along timings…