Generalization Performance of Fast and Frugal Heuristics
Across 20 datasets, Take the Best and Dawes’ Rule outperformed multiple regression on average when predicting held-out cases. Minimalist finished close behind regression. The results illustrate how simple rules can generalize well despite using less information.
THE IDEA
Generalization
The study compared prediction methods across 20 empirical datasets rather than relying on artificial, multivariate normal data. The datasets ranged from 17 to 395 objects and from 3 cues (the minimum needed to distinguish the heuristics) to 19 cues. Topics included high school dropout rates, oxidant production in chemical reactions, and the number of eggs in individual fish.
Table 31.3. Trade-off between Accuracy and Cues Looked Up, Averaged Across 20 Data Sets
- When test set = training set, multiple linear regression is the most accurate (2 percentage points higher), followed by Take the Best.
- When test set ≠ training set, Take the Best is the most accurate (4 percentage points higher than regression), followed by Dawes' Rule (3 percentage points higher than regression), and Minimalist (2 percentage points behind regression).
[Image omitted: Table 31.3 showing performance averages]
Note that multiple regression has all the information Take the Best uses plus more. Dawes’ Rule lives up to its reputation for robustness in the literature (Dawes, 1979) by taking second place and beating regression by three percentage points. Finally, Minimalist performs surprisingly well, only two percentage points behind regression. These averages show that simple models can generalize well across varied tasks. Section 3 of the source examines why such heuristics can be accurate.
When to use it
Testing a forecast based on a few key variables against a more complex model on held-out data
Comparing a short diagnostic rule with a more complex model using an appropriate validation dataset
Evaluating a simple recommendation baseline before adopting a more complex system
Testing whether an investment screening rule generalizes beyond the data used to design it
FROM THEORY TO EVERYDAY LIFE
In practice
Predicting high school dropout rates
High school dropout rates were among the tasks studied. A Take the Best comparison would examine cues in order of predictive validity and stop at the first cue that distinguishes the cases. The reported table summarizes results across datasets rather than establishing which method won on this particular task.
Chemical reaction prediction
Oxidant production in chemical reactions was another task in the dataset collection. Minimalist checks cues in random order and stops when a cue distinguishes the cases. Its average accuracy across the full collection was close to regression’s.
A MOMENT TO REFLECT
Put it to work.
How might the trade-off between fitting and generalization apply to your own decision-making processes?
Can you identify a situation where relying on a simple heuristic would have been more effective than a complex analysis?
What are the risks of overfitting when using complex models on limited data?
Often confused with
References
- The robust beauty of improper linear models in decision making — Dawes, R. M. (1979)
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)