How to Be Right
A model can produce a precise number even when uncertainty in its assumptions makes that number uninformative. Demanding a single forecast can conceal how widely plausible outcomes vary.
THE IDEA
HOW TO BE RIGHT
Between my sophomore and junior years of college, I had a summer job working for a researcher in public health. The researcher—it will be clear in a minute why I don’t use his name—wanted to hire a math major because he wanted to know how many people were going to have tuberculosis in the year 2050. That was my summer job, to figure that out. The researcher gave me a big folder of papers about tuberculosis: how transmissible it was under various circumstances, the typical course of infection and the length of the maximally contagious period, survival curves and medication compliance rates and breakdowns of all of the above by age, race, sex, and HIV status. Big folder. Lots of papers. And I got to work, doing what math majors do: I made a model of TB prevalence, using the data the researcher had given me to estimate how the TB infection rates in different population groups would change and interact over time, decade by decade, until 2050, when the simulation terminated.
And what I learned was this: I did not have a clue as to how many people were going to have tuberculosis in the year 2050. Each one of those empirical studies had some uncertainty built in; they thought the transmission rate was 20%, but maybe it was 13%, or maybe it was 25%, though they were pretty sure it wasn’t 60% or 0%. Each one of these little local uncertainties percolated through the simulation, and the uncertainties about different parameters of the model fed back into each other, and by 2050, the noise had engulfed the signal. I could make the simulation come out any which way. Maybe there was going to be no such thing as tuberculosis in 2050, or maybe most of the world’s population would be infected. I had no principled way to choose.
This was not what the researcher wanted to hear. It was not what he was paying me for. He was paying me for a number, and he patiently repeated his request for one. I know there’s uncertainty, he said, that’s how medical research is, I get that, just give me your best guess. It didn’t matter how much I protested that any single guess would be worse than no guess at all. He insisted. And he was my boss, and eventually I gave in. I have no doubt he told many people, afterward, that X million people were going to have tuberculosis in the year 2050. And I’ll bet if anyone asked him how he knew this, he would say, I hired a guy who did the math.
When to use it
When evaluating economic forecasts, beware of overly precise predictions.
When assessing long-term public-health forecasts, examine how uncertain inputs affect the range of projected outcomes.
When estimating project timelines, communicate the uncertainty rather than giving a single date.
When interpreting scientific studies, look for confidence intervals rather than just point estimates.
FROM THEORY TO EVERYDAY LIFE
In practice
Tuberculosis Prediction
A researcher demands a single best guess for tuberculosis prevalence in 2050 despite huge uncertainty in the model inputs, leading to a false sense of precision.
A MOMENT TO REFLECT
Put it to work.
Think of a time when you demanded a single number from an expert despite uncertainty. What were the consequences?
How can we better communicate uncertainty in decision-making?
Why might people prefer a point estimate over a range, even when the range is more honest?
CONTINUE THE CONVERSATION
Source texts
- How Not to Be Wrong The Power of Mathematical Thinking (Ellenberg, Jordan)