Post-Designation (Fishing for Data)
Post-designation treats a pattern noticed after inspecting data as though it had been predicted in advance. Exploratory analysis can generate useful hypotheses; confirmation requires accounting for the search and testing the pattern appropriately.
THE IDEA
Description
The problem arises when a correlation discovered during exploration is presented as confirmation of a prior prediction. Searching many possible patterns increases the opportunity to find chance anomalies. Exploration itself is useful when the resulting hypotheses are identified as exploratory and tested accordingly.
Example #1
The original example is absent. The commentary describes a sample that seemed “odd” because 9 out of 10 children had no siblings. Whether that is surprising depends on the population and how the sample was selected. If the pattern was noticed during a broader search, it should be tested in new data rather than treated as a confirmed prediction.
Example #2
The original tuna example is also absent. The intended point is that searching widely gives more opportunities to find an apparently interesting pattern by chance.
Exception
An unexpected pattern can be real. Report how it was discovered, distinguish exploration from confirmation, and assess it with methods suited to the search or with independent data.
When to use it
Distinguishing hypotheses generated by exploratory analysis from findings tested against a prior prediction.
Data mining in large datasets where many correlations are tested
Post-hoc analysis of clinical trial results without multiple comparison correction
Finding patterns in observational studies without adjusting for the number of comparisons made
FROM THEORY TO EVERYDAY LIFE
In practice
Example #1: Sibling Anomaly
A sample appears “odd” because 9 out of 10 children have no siblings. Before interpreting that result, identify the population baseline, sampling method, and other patterns examined. The count alone does not establish either an anomaly or a general population claim.
Example #2: Fishing for Tuna
The source mentions tuna while discussing data fishing but omits the original scenario. A search that considers many possible relationships needs to account for that search when assessing a promising result.
A MOMENT TO REFLECT
Put it to work.
How can post-designation lead to false conclusions in data analysis?
What steps can researchers take to avoid this fallacy in their studies?
Think of a time you observed a pattern after looking at data; how could post-designation have influenced your interpretation?
CONTINUE THE CONVERSATION
Source texts
- Logically Fallacious - The Ultimate Collection of Over 300 Logical Fallacies