Circularity of Statistics (Statistical Regress Argument)
The circularity of statistics arises when we need past data to determine the probability distribution, but we need the distribution to know how much data is sufficient. This regress is particularly problematic for heavy-tailed distributions (Extremistan) compared to Gaussian distributions.
THE IDEA
From Representation to Reality
The above idea links all the parts of this book. While many study psychology, mathematics, or evolutionary theory and look for ways to take it to the bank by applying their ideas to business, I suggest the exact opposite: study the intense, uncharted, humbling uncertainty in the markets as a means to get insights about the nature of randomness that is applicable to psychology, probability, mathematics, decision theory, and even statistical physics. You will see the sneaky manifestations of the narrative fallacy, the ludic fallacy, and the great errors of Platonicity, of going from representation to reality.
When I first met Mandelbrot I asked him why an established scientist like him who should have more valuable things to do with his life would take an interest in such a vulgar topic as finance. I thought that finance and economics were just a place where one learned from various empirical phenomena and filled up one’s bank account with f*** you cash before leaving for bigger and better things. Mandelbrot’s answer was, “Data, a gold mine of data.” Indeed, everyone forgets that he started in economics before moving on to physics and the geometry of nature. Working with such abundant data humbles us; it provides the intuition of the following error: traveling the road between representation and reality in the wrong direction.
The problem of the circularity of statistics (which we can also call the statistical regress argument) is as follows. Say you need past data to discover whether a probability distribution is Gaussian, fractal, or something else. You will need to establish whether you have enough data to back up your claim. How do we know if we have enough data? From the probability distribution—a distribution does tell you whether you have enough data to “build confidence” about what you are inferring. If it is a Gaussian bell curve, then a few points will suffice (the law of large numbers once again). And how do you know if the distribution is Gaussian? Well, from the data. So we need the data to tell us what the probability distribution is, and a probability distribution to tell us how much data we need. This causes a severe regress argument.
This regress does not occur if you assume beforehand that the distribution is Gaussian. It happens that, for some reason, the Gaussian yields its properties rather easily. Extremistan distributions do not do so. So selecting the Gaussian while invoking some general law appears to be convenient. The Gaussian is used as a default distribution for that very reason. As I keep repeating, assuming its application beforehand may work with a small number of fields such as crime statistics, mortality rates, matters from Mediocristan. But not for historical data of unknown attributes and not for matters from Extremistan.
Now, why aren’t statisticians who work with historical data aware of this problem? First, they do not like to hear that their entire business has been canceled by the problem of induction. Second, they are not confronted with the results of their predictions in rigorous ways. As we saw with the Makridakis competition, they are grounded in the narrative fallacy, and they do not want to hear it.
When to use it
When estimating Value-at-Risk, comparing plausible tail models and checking how sensitive the result is to distributional assumptions.
Scientific research: In fields like climatology, where extreme events are rare, the circularity affects the reliability of confidence intervals.
Policy decisions: Using historical crime data to predict future crime rates without knowing the underlying distribution can lead to improper resource allocation.
Business forecasting: Companies using past sales data to project future demand must be aware of the circularity when the distribution of demand is unknown.
FROM THEORY TO EVERYDAY LIFE
In practice
Financial Risk Estimation
An analyst estimates market risk from historical returns. A sample with few extreme losses may fit a Gaussian model while providing little evidence about the far tail. Comparing alternative models and examining how the risk estimate changes can expose the uncertainty that a single fitted distribution hides.
Hurricane Intensity Prediction
A researcher uses historical hurricane intensities to estimate the chance of an unusually severe storm. With few extreme observations, the answer may depend heavily on the assumed distribution. More ordinary observations do not necessarily resolve that uncertainty.
A MOMENT TO REFLECT
Put it to work.
How might you test whether a dataset comes from a Gaussian or fat-tailed distribution without falling into the circularity?
Can you think of a real-world situation where the statistical regress argument caused a flawed prediction?
What are the implications of the circularity of statistics for machine learning models that rely on historical data?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- The Black Swan The Impact of the Highly Improbable With a new section On Robustness and Fragility (Nassim Nicholas Taleb)