Multiple Comparisons Fallacy
When many statistical tests are run, there are more opportunities for false positives. The error is presenting selected significant findings without accounting for the full set of comparisons.
THE IDEA
Multiple Comparisons Fallacy
A “confidence level” describes a statistical procedure, not the probability that every conclusion in a study is true. Statistical tests do not provide 100% certainty. At the threshold commonly associated with a 95% confidence interval, a valid test rejects a true null hypothesis about once in 20 repetitions. Running 20 tests at that nominal 95% level creates more opportunities for a false positive.
The error arises when selected significant results are presented without accounting for how many comparisons were made. Multiple testing does not make every significant result false; it changes how the collection of results should be assessed.
Example #1
A researcher compares many groups and reports only the comparison that crosses a significance threshold. The reader needs to know about the other tests to assess whether the reported result is persuasive.
Example #2
In an illustrative study, researchers make over 500 comparisons between groups, including sleep quality, sleeping position, and breakfast choices. They highlight the strongest sleeping-position difference and omit the rest. Even without a real effect, a large search can produce apparently striking results.
Researchers can address this problem through a planned analysis, suitable adjustments for multiple testing, and transparent reporting. Further testing can help assess whether an exploratory finding holds up.
When to use it
When interpreting results from large-scale medical trials or epidemiological studies that test many hypotheses simultaneously.
In A/B testing for website optimization, where many variants are compared and a few may show significant differences by chance.
When evaluating correlations from a large dataset with many variables, to avoid interpreting spurious correlations as meaningful.
In psychological research where multiple personality traits or behaviors are compared across different groups, requiring correction for multiple comparisons.
FROM THEORY TO EVERYDAY LIFE
In practice
Multiple group comparisons
A researcher tests many differences between groups and highlights the only statistically significant result. Without reporting the full set of comparisons or addressing multiple testing, the selected result can look stronger than it is.
Study with 500 comparisons
Imagine a study making over 500 comparisons between groups, from sleep quality to breakfast choices. If it reports only the strongest sleeping-position difference, readers cannot judge that result without knowing how many other possibilities were tested.
A MOMENT TO REFLECT
Put it to work.
Can you recall a situation where multiple tests were run and the only significant result was highlighted, while the non-significant ones were ignored? How might this be misleading?
How would you design a study to account for the multiple comparisons problem? What statistical corrections would you consider?
Think of a news headline claiming a new risk factor or benefit. How could the multiple comparisons fallacy be at play in the underlying research?
CONTINUE THE CONVERSATION
Source texts
- Logically Fallacious - The Ultimate Collection of Over 300 Logical Fallacies