Choosing the right statistical test

Transcript

SoheePicture a clinic director who walks into your office with three questions. Do clients in the new therapy end up with lower depression scores than clients on the waitlist? Do our three locations differ in average wait time? And is a client's diagnosis related to whether they drop out?

RyanThree questions, three different statistical tests. And choosing the analysis for a described study is a classic exam task.

SoheeThis is EPPP Prep, the podcast. I'm Sohee.

RyanI'm Ryan. Today we build a mental decision tree for choosing a test. No formulas, only the questions you ask yourself.

SoheeOur main guide is an open-access article by Mishra and colleagues on selecting statistical methods. They say the choice depends on three things: the aim of the study, the type and distribution of the data, and whether the observations are paired or unpaired.

RyanLet's take those in turn. The aim: are you comparing groups, looking at a relationship, or predicting an outcome? The article's example: to find predictors of an outcome, use regression. To compare the means of two independent samples, use an unpaired t-test.

SoheeType of data connects to the levels of measurement from our descriptive statistics episode. The article says nominal and ordinal data call for nonparametric methods, while continuous data can use parametric or nonparametric methods.

RyanAnd paired versus unpaired. Paired means the same people measured at different times or by different methods. Unpaired means each group has different people.

SoheeNow the big umbrella terms. The article says methods that compare means are called parametric, and methods that compare other things, like medians, mean ranks or proportions, are called nonparametric.

RyanAnd parametric tests rely on the assumption that the variable is continuous and approximately normally distributed.

SoheeLet's start with the workhorse. The t-test compares the means of two groups.

RyanThe OpenStax statistics text gives a nice intuition. A difference between two samples depends on both the means and the standard deviations. Very different means can happen by chance if there's a lot of variation within the samples.

SoheeSo the test takes the difference between the means and divides it by the standard error, which standardizes it. Think of it as signal compared with noise.

RyanAnd there are two flavors. An independent samples t-test for two separate groups, like therapy versus waitlist. That answers the director's first question.

SoheeAnd a paired samples t-test when the two measurements come from the same people or matched pairs. The text's example: pain ratings before and after hypnotism in the same subjects. The differences become the data.

RyanAnd getting this wrong matters. The Mishra article shows a real example. Two separate groups, analyzed correctly with an independent samples t-test, gave a p-value of .061, not significant. The same data run with a paired t-test gave .011.

SoheeWhich falsely suggested a significant difference. Wrong test, wrong conclusion.

RyanNow, more than two groups. The director's second question has three locations. That's one-way analysis of variance, or ANOVA.

SoheeThe OpenStax text says one-way ANOVA tests whether there's a statistically significant difference among several group means. The null is that all the group means are the same. The alternative is that at least two of them differ.

RyanAnd here's the clever part. It actually uses variances to decide about means. It compares the variation between the groups with the variation within the groups.

SoheeThat comparison is the F ratio, named after Sir Ronald Fisher. The text explains that if the null is true, both estimates should be about the same, so F should be close to one.

RyanAnd if the null is false, the variation between groups tends to be bigger, so F climbs above one.

SoheeThe text lists the assumptions: each population is normal, the samples are random and independent, the populations have equal standard deviations, the factor is categorical, and the response is numerical.

RyanSo why not run three separate t-tests, location one versus two, one versus three, two versus three?

SoheeBecause the text says performing multiple tests introduces the likelihood of making a Type I error. Every extra test is another chance for a false alarm. ANOVA asks the overall question once.

RyanAnd the article lists repeated measures ANOVA among these F tests too, which you'll meet when the same people are measured several times.

SoheeNow the third question. Diagnosis category and dropout, yes or no. Both variables are categories. You can't take a mean of a diagnosis.

RyanThat's chi-square territory. The Mishra article says methods comparing proportions are nonparametric, and Pearson's chi-square test and Fisher's exact test compare proportions between independent groups.

SoheeThe OpenStax text describes two main versions. The goodness-of-fit test asks whether your data fit a particular distribution. The test of independence asks whether two factors are independent, using a contingency table.

RyanThe director's question is a test of independence. Is dropout independent of diagnosis?

SoheeAnd a detail examiners like: the text says the expected value in each cell needs to be at least five to use these tests.

RyanOkay, pause and think. A researcher measures anxiety in the same twenty clients before and after a mindfulness course. Which test?

SoheeSame people, two measurements, comparing means. A paired samples t-test.

RyanAnd a survey asks whether people prefer video or in-person sessions, and records their age group as young, middle or older. Is preference related to age group?

SoheeTwo categorical variables. Chi-square test of independence.

RyanNow relationships and prediction. To measure the straight-line relationship between two continuous variables, the Pearson correlation coefficient. The article lists it, along with linear regression, as parametric.

SoheeRegression goes one step further and predicts. The OpenStax text describes the least-squares line, the line of best fit, chosen so that the sum of the squared errors is as small as possible.

RyanAnd each error, the vertical distance from a real data point to the line, is called a residual. The text notes it isn't an error in the sense of a mistake.

SoheeHere's a number to understand, not memorize. Square the correlation and you get the coefficient of determination, r squared. The text says it's the percent of variation in the predicted variable that can be explained by the predictor, using the regression line.

RyanThe text's example: a correlation of about .66 between a third exam and the final exam gives an r squared of about .44. So about 44 percent of the variation in final exam scores is explained by the third exam, and about 56 percent is not.

SoheeAnd one caution. Use the line to predict only within the range of values in your data. The text says you shouldn't predict for values outside that range.

RyanIf the outcome is a category, like relapsed or not, the article points to logistic regression instead.

SoheeFinally, the nonparametric backups. The article's table pairs them up. Mann-Whitney U stands in for the independent samples t-test, and the Wilcoxon signed-rank test for the paired t-test. Kruskal-Wallis stands in for one-way ANOVA, and Friedman for repeated measures ANOVA. Spearman's rank correlation stands in for Pearson.

RyanA little history from an open-access review by Nahm on nonparametric tests. In 1945, Frank Wilcoxon introduced a method using ranks. In 1947, Henry Mann and his student Donald Ransom Whitney expanded on it to compare two groups of different sizes.

SoheeRanks are the key idea. Instead of using the raw scores, you line everyone up from lowest to highest and work with their positions.

RyanWhy not always use them? The article says parametric methods are stronger at detecting differences between groups, but their strict assumptions, like normally distributed data, mean they can't be used in every situation. And nonparametric methods use medians or mean ranks, which outliers don't throw off.

SoheePause and think, last one. Two small groups, and the outcome data are badly skewed with a couple of extreme values. You want to compare the groups. What's a sensible choice?

RyanA nonparametric test, like Mann-Whitney U. The data break the normality assumption, and ranks resist those extreme values.

SoheeSo how does the exam test this? A study description, then which statistical test is most appropriate.

RyanThe tempting wrong answers: multiple t-tests instead of ANOVA, an independent t-test for before-and-after data on the same people, and a t-test or ANOVA when the outcome is categorical.

SoheeRecap. Two group means, t-test, independent or paired. Three or more group means, ANOVA, which compares between-group and within-group variation in the F ratio.

RyanCategories and counts, chi-square, goodness of fit or independence, with expected counts of at least five per cell.

SoheeRelationship, Pearson correlation. Prediction, regression, with r squared as the share of variation explained. Categorical outcome, logistic regression.

RyanData that break the assumptions: Mann-Whitney, Wilcoxon, Kruskal-Wallis, Friedman, Spearman.

SoheeThis topic can feel like a wall of names. But you've seen today that three questions sort almost everything: what's the aim, what kind of data, and are the observations paired?

RyanPractice asking those three questions on a handful of items, and watch the choices narrow. Thanks for listening to EPPP Prep.

Loading the read-along timings…