Beware the Forecasters
The text warns against overreliance on predictive models, especially for nonlinear systems exhibiting Extremistan characteristics, due to calibration problems and the risk of using the wrong model.
THE IDEA
ONCE AGAIN, BEWARE THE FORECASTERS
Let me take the problem one step higher up. As I mentioned earlier, plenty of fashionable models attempt to explain the genesis of Extremistan. In fact, they are grouped into two broad classes, but there are occasionally more approaches. The first class includes the simple rich-get-richer (or big-get-bigger) style model that is used to explain the lumping of people around cities, the market domination of Microsoft and VHS (instead of Apple and Betamax), the dynamics of academic reputations, etc. The second class concerns what are generally called “percolation models,” which address not the behavior of the individual, but rather the terrain in which he operates. When you pour water on a porous surface, the structure of that surface matters more than does the liquid. When a grain of sand hits a pile of other grains of sand, how the terrain is organized is what determines whether there will be an avalanche.
Most models, of course, attempt to be precisely predictive, not just descriptive; I find this infuriating. They are nice tools for illustrating the genesis of Extremistan, but I insist that the “generator” of reality does not appear to obey them closely enough to make them helpful in precise forecasting. At least to judge by anything you find in the current literature on the subject of Extremistan. Once again we face grave calibration problems, so it would be a great idea to avoid the common mistakes made while calibrating a nonlinear process. Recall that nonlinear processes have greater degrees of freedom than linear ones (as we saw in Chapter 11), with the implication that you run a great risk of using the wrong model. Yet once in a while you run into a book or articles advocating the application of models from statistical physics to reality. Beautiful books like Philip Ball’s illustrate and inform, but they should not lead to precise quantitative models. Do not take them at face value.
But let us see what we can take home from these models.
When to use it
Assessing the reliability of economic growth forecasts that ignore the possibility of rare, impactful events.
Understanding why certain companies dominate their industry despite similar initial conditions.
Evaluating risk models in insurance or finance that may underestimate tail risks.
Analyzing the spread of information or diseases using percolation theory rather than individual behavior models.
FROM THEORY TO EVERYDAY LIFE
In practice
Rich-get-richer models
Rich-get-richer models explain concentration of wealth, city growth, and market dominance like Microsoft over Apple and VHS over Betamax. They illustrate how small advantages amplify over time, leading to extreme outcomes.
Percolation in avalanches
Percolation models focus on the structure of the environment rather than individual events. When pouring water on porous surfaces or adding sand to a pile, the terrain determines whether an avalanche occurs. The system's organization can be more important than individual actions.
A MOMENT TO REFLECT
Put it to work.
When evaluating a forecast, how can you determine if the underlying system is linear or nonlinear?
What are the dangers of using a model that is precisely calibrated to past data but fails to capture the generating mechanism of extreme events?
In your own field, identify a situation where a rich-get-richer model could explain extreme outcomes better than a traditional average-based model.
Often confused with
CONTINUE THE CONVERSATION
Source texts
- The Black Swan The Impact of the Highly Improbable With a new section On Robustness and Fragility (Nassim Nicholas Taleb)