Regression to the Mean

Understanding Regression to the Mean

Whether undetected or wrongly explained, the phenomenon of regression is strange to the human mind. So strange, indeed, that it was first identified and understood two hundred years after the theory of gravitation and differential calculus. Furthermore, it took one of the best minds of nineteenth-century Britain to make sense of it, and that with great difficulty.

Regression to the mean was discovered and named late in the nineteenth century by Sir Francis Galton, a half cousin of Charles Darwin and a renowned polymath. You can sense the thrill of discovery in an article he published in 1886 under the title “Regression towards Mediocrity in Hereditary Stature,” which reports measurements of size in successive generations of seeds and in comparisons of the height of children to the height of their parents. He writes about his studies of seeds:

They yielded results that seemed very noteworthy, and I used them as the basis of a lecture before the Royal Institution on February 9th, 1877. It appeared from these experiments that the offspring did not tend to resemble their parent seeds in size, but to be always more mediocre than they—to be smaller than the parents, if the parents were large; to be larger than the parents, if the parents were very small … The experiments showed further that the mean filial regression towards mediocrity was directly proportional to the parental deviation from it.

Galton obviously expected his learned audience at the Royal Institution—the oldest independent research society in the world—to be as surprised by his “noteworthy observation” as he had been. What is truly noteworthy is that he was surprised by a statistical regularity that is as common as the air we breathe. Regression effects can be found wherever we look, but we do not recognize them for what they are. They hide in plain sight. It took Galton several years to work his way from his discovery of filial regression in size to the broader notion that regression inevitably occurs when the correlation between two measures is less than perfect, and he needed the help of the most brilliant statisticians of his time to reach that conclusion.

One of the hurdles Galton had to overcome was the problem of measuring regression between variables that are measured on different scales, such as weight and piano playing. This is done by using the population as a standard of reference. Researchers often transform raw scores into standard scores, setting the mean at 0 and the standard deviation at 1, so that variables can be compared directly. Imagine that weight and piano playing have been measured for 100 children in all grades of an elementary school, and that they have been ranked from high to low on each measure. If Jane ranks third in piano playing and twenty-seventh in weight, it is appropriate to say that her piano-playing rank is higher than her weight rank. Let us make some assumptions that will simplify things:

At any age,

• Piano-playing success depends only on weekly hours of practice.
• Weight depends only on consumption of ice cream.
• Ice cream consumption and weekly hours of practice are unrelated.

Now, using ranks (or the standard scores that statisticians prefer), we can write some equations:

weight = age + ice cream consumption piano playing = age + weekly hours of practice

You can see that there will be regression to the mean when we predict piano playing from weight, or vice versa. If all you know about Tom is that he ranks twelfth in weight (well above average), you can infer (statistically) that he is probably older than average and also that he probably consumes more ice cream than other children. If all you know about Barbara is that she is eighty-fifth in piano (far below the average of the group), you can infer that she is likely to be young and that she is likely to practice less than most other children.

For the positive associations discussed here, the correlation coefficient lies between 0 and 1 and summarizes the strength of a linear relationship. Correlations can also be negative, and a coefficient does not by itself identify shared causes. For example, we all share half our genes with each of our parents, and for traits in which environmental factors have relatively little influence, such as height, the correlation between parent and child is not far from .50. However, environmental factors like nutrition also affect height, so the correlation is not perfect. To appreciate the meaning of the correlation measure, the following are some examples of coefficients:

• The correlation between the size of objects measured with precision in English or in metric units is 1. Any factor that influences one measure also influences the other; 100% of determinants are shared.
• The correlation between self-reported height and weight among adult American males is .41. If you included women and children, the correlation would be much higher, because individuals’ gender and age influence both their height and their weight, boosting the relative weight of shared factors. Still, even with all those shared factors, the correlation is not perfect.
• The correlation between SAT scores and college GPA is approximately .60. However, the correlation between aptitude tests and success in graduate school is much lower, largely because measured aptitude varies little in this selected group. If everyone has similar aptitude, differences in this measure are unlikely to play a large role in measures of success.
• The correlation between income and education level in the United States is approximately .40. The text illustrates a model-based reduction in income dispersion of about 9%, using the square root of (1 – r²). That residual-variation calculation does not establish what an intervention that equalized education would do to incomes.
• The correlation between family income and the last four digits of their phone number is 0.

It took Francis Galton several years to figure out that correlation and regression are not two concepts—they are different perspectives on the same concept. In standardized linear prediction, an imperfect positive correlation produces a less extreme predicted score. This is a statement about average predictions, not a guarantee for each individual observation. Because of this shared mathematical foundation, the term “regression” can easily cause confusion. In modern statistics, “regression analysis” refers to a set of methods for examining how one variable relates to another—a simple regression between two variables is equivalent to a correlation, while multiple regression examines the relation of a set of variables to another variable. This analytical framework was formalized largely by Karl Pearson, who invented the correlation technique and first used it to study relationships between related individuals (such as fathers’ and sons’ heights). On the other hand, “regression to the mean” is the observed phenomenon: the tendency for extreme outcomes to be followed by more average ones. Though distinct, the two are deeply connected—regression analysis can quantify the less-extreme predictions associated with an imperfect correlation under the model’s assumptions.

To illustrate Galton’s insight, take a proposition that most people find quite interesting:

Highly intelligent women tend to marry men who are less intelligent than they are.

You can get a good conversation started at a party by asking for an explanation, and your friends will readily oblige. Even people who have had some exposure to statistics will spontaneously interpret the statement in causal terms. Some may think of highly intelligent women wanting to avoid the competition of equally intelligent men, or being forced to compromise in their choice of spouse because intelligent men do not want to compete with intelligent women. More far-fetched explanations will come up at a good party. Now consider this statement:

The correlation between the intelligence scores of spouses is less than perfect.

This statement is obviously true and not interesting at all. Who would expect the correlation to be perfect? There is nothing to explain. In a linear prediction model with comparable score distributions, the two statements express the same regression effect. An unusually high score predicts a less extreme partner score when the correlation is imperfect; the reasoning also applies with the spouses’ roles reversed. The pattern alone does not establish the motives behind anyone’s choice of partner.

How large a regression effect you see in practice depends on how much of the variation is due to stable, lasting factors and how much is due to transitory influences. A clear illustration comes from weather. Take thirteen weather stations in southern Wisconsin, all within an hour’s drive of each other. The average January temperature in 2011 and 2012 shows a definite regression pattern: the three warmest stations in 2011 (Madison’s Charmany, Madison airport, and Stoughton) ranked warm, seventh warmest, and eighth warmest in 2012, while the three coldest in 2011 (Cottage Grove, Lodi, and Portage) moved up—Portage was tied for fourth coldest, Lodi second coldest, and Cottage Grove was actually warmer than most others. Both extremes drifted toward the middle. Now compare ten California stations, from cool Truckee in the Sierra Nevada to warm San Diego and Los Angeles. Their January temperatures across the same two years show almost no regression: the cold places stay cold, the hot places stay hot. The difference is a matter of signal and noise. In Wisconsin, year-to-year fluctuations are large relative to the tiny inherent climatic differences between towns, so chance dominates the rankings. In California, the underlying geographical differences are huge and swamp the chance variation. The correlation between successive years is much lower in Wisconsin, and regression appears much stronger.

You probably sympathize with Galton’s struggle with the concept of regression. Indeed, the statistician David Freedman used to say that if the topic of regression comes up in a criminal or civil trial, the side that must explain regression to the jury will lose the case. Why is it so hard? Kahneman’s explanation is that our mind is strongly biased toward causal explanations and does not deal well with “mere statistics.” When our attention is called to an event, associative memory will look for its cause—more precisely, activation will automatically spread to any cause that is already stored in memory. When a change fits a regression pattern, we may invent a causal story that the evidence does not support. A statistical expectation does not identify the cause of a particular outcome. The event that attracts our attention in the golfing tournament is the frequent deterioration of the performance of the golfers who were successful on day 1. The best explanation of it is that those golfers were unusually lucky that day, but this explanation lacks the causal force that our minds prefer. Indeed, we pay people quite well to provide interesting explanations of regression effects. A business commentator who correctly announces that “the business did better this year because it had done poorly last year” is likely to have a short tenure on the air.

Our difficulties with the concept of regression originate with both System 1 and System 2. Without special instruction, and in quite a few cases even after some statistical instruction, the relationship between correlation and regression remains obscure. System 2 finds it difficult to understand and learn. This is due in part to the insistent demand for causal interpretations, which is a feature of System 1.

Incorrect causal interpretations of regression effects are not restricted to casual party talk. The statistician Howard Wainer has drawn up a long list of eminent researchers who have made the same mistake—confusing mere correlation with causation. A spectacular example is the economist Horace Secrist, who in 1933 published a massive book titled The Triumph of Mediocrity in Business after a decade of painstaking work collecting data on hundreds of business enterprises. Secrist observed that firms with the highest profits or sales in a given year tended to fall back toward the average in later years, while those with the worst results improved. He interpreted this as a profound law of commerce—that competitive pressures inevitably eroded exceptional performance, grinding everything toward mediocrity. He was not wrong about the pattern, but he was catastrophically wrong about its meaning.

The statistician Harold Hotelling delivered the definitive rebuttal. In a review that was polite but devastating, he pointed out that the regression Secrist had so laboriously documented is “mathematically obvious from general considerations, and does not need the vast accumulation of data adduced to prove it.” Hotelling then made a decisive observation. If the regression were really caused by competitive forces over time, those forces should not work backward. But if you select the stores that were top performers in 1922 and look at their results in 1916, they also regress toward the mean. Competitive pressure cannot explain why a store’s luck would reverse in the past. The regression Secrist found is exactly what you would expect from the mingling of skill and chance, and it says nothing about a unique law of business physics. As Hotelling put it, “To ‘prove’ such a mathematical result by a costly and prolonged numerical study of many kinds of business profit and expense ratios is analogous to proving the multiplication table by arranging elephants in rows and columns, and then doing the same for numerous other kinds of animals. The performance, though perhaps entertaining, and having a certain pedagogical value, is not an important contribution either to zoölogy or mathematics.”

Secrist’s own data illustrate the point perfectly. He also studied temperatures in U.S. cities and noted that the hottest cities in 1922 were still the hottest in 1931, with no visible regression. He seized on this to argue that regression was a special feature of human competition. But he was merely comparing cities with wildly different climates—akin to the California stations. When you restrict your attention to a region where differences are small and chance dominates, like southern Wisconsin, the same regression emerges for temperatures. The authors interpret Secrist’s firms as more like the Wisconsin weather stations: temporary variation plays a substantial role alongside persistent differences. The regression pattern alone does not establish equal contributions from luck and management skill. If businesses were as fundamentally different from one another as Los Angeles is from Truckee, regression would be minimal.

Depressed children treated with an energy drink improve significantly over a three-month period.

Kahneman invents this headline to illustrate a problem with before-and-after claims. A group selected for unusually high depression scores might show lower scores at a later measurement even without an effective intervention. Regression can contribute because the initial measurements include temporary influences; it does not guarantee clinically significant improvement for a group or recovery for a child. To estimate an intervention’s effect, researchers need a suitable “control group” and a design that separates treatment effects from regression and other changes over time. Improvement after an intervention alone is insufficient evidence that it worked.

One of my favorite examples of the errors of intuitive prediction is adapted from Max Bazerman’s excellent text Judgment in Managerial Decision Making:

You are the sales forecaster for a department store chain. All stores are similar in size and merchandise selection, but their sales differ because of location, competition, and random factors. You are given the results for 2011 and asked to forecast sales for 2012. You have been instructed to accept the overall forecast of economists that sales will increase overall by 10%. How would you complete the following table?

Store	2011	2012
1	$11,000,000	____
2	$23,000,000	____
3	$18,000,000	____
4	$29,000,000	____
Total	$81,000,000	$89,100,000

Adding 10% uniformly assumes each store’s relative position will persist. If some of the differences are temporary, a regressive forecast would instead give low-performing branches more than 10% growth and high-performing branches less, while preserving the overall total. The amount of adjustment requires evidence about how persistent store differences are. But if you ask other people, you are likely to encounter puzzlement: Why do you bother them with an obvious question? As Galton painfully discovered, the concept of regression is far from obvious.

An extreme result is a reason to say “not so fast.” Ask how much of it reflects persistent differences and how much reflects temporary variation. Under stable conditions, the latter can make the next observation less extreme on average; no force pulls every exceptional performer back to the mean.