7 Overfitting

When to Think Less

When Charles Darwin was trying to decide whether he should propose to his cousin Emma Wedgwood, he got out a pencil and paper and weighed every possible consequence. In favor of marriage he listed children, companionship, and the "charms of music & female chit-chat." Against marriage he listed the "terrible loss of time," lack of freedom to go where he wished, the burden of visiting relatives, the expense and anxiety provoked by children, the concern that "perhaps my wife won't like London," and having less money to spend on books. Weighing one column against the other produced a narrow margin of victory, and at the bottom Darwin scrawled, "Marry—Marry—Marry Q.E.D." Quod erat demonstrandum, the mathematical sign-off that Darwin himself then restated in English: "It being proved necessary to Marry."

The pro-and-con list was already a time-honored algorithm by Darwin's time, being endorsed by Benjamin Franklin a century before. To get over "the Uncertainty that perplexes us," Franklin wrote, my Way is, divide half a Sheet of Paper by a Line into two Columns, writing over the one Pro, and over the other Con. Then during three or four Days Consideration I put down under the different Heads short Hints of the different Motives that at different Times occur to me for or against the Measure. When I have thus got them all together in one View, I endeavour to estimate their respective Weights; and where I find two, one on each side, that seem equal, I strike them both out: If I find a Reason pro equal to some two Reasons con, I strike out the three. If I judge some two Reasons con equal to some three Reasons pro, I strike out the five; and thus proceeding I find at length where the Ballance lies; and if after a Day or two of farther Consideration nothing new that is of Importance occurs on either side, I come to a Determination accordingly. Franklin even thought about this as something like a computation, saying, "I have found great Advantage from this kind of Equation, in what may be called Moral or Prudential Algebra."

Darwin's journal, July 1838. Reprinted with permission of Cambridge University Library.

When we think about thinking, it's easy to assume that more is better: that you will make a better decision the more pros and cons you list, make a better prediction about the price of a stock the more relevant factors you identify, and write a better report the more time you spend working on it. This is certainly the premise behind Franklin's system. In this sense, Darwin's "algebraic" approach to matrimony, despite its obvious eccentricity, seems remarkably and maybe even laudably rational.

However, if Franklin or Darwin had lived into the era of machine-learning research—the science of teaching computers to make good judgments from experience—they'd have seen Moral Algebra shaken to its foundations. The question of how hard to think, and how many factors to consider, is at the heart of a knotty problem that statisticians and machine-learning researchers call "overfitting." And dealing with that problem reveals that there's a wisdom to deliberately thinking less. Being aware of overfitting changes how we should approach the market, the dining table, the gym … and the altar.

The Case Against Complexity

Anything you can do I can do better; I can do anything better than you. — ANNIE GET YOUR GUN

Every decision is a kind of prediction: about how much you'll like something you haven't tried yet, about where a certain trend is heading, about how the road less traveled (or more so) is likely to pan out. And every prediction, crucially, involves thinking about two distinct things: what you know and what you don't. That is, it's an attempt to formulate a theory that will account for the experiences you've had to date and say something about the future ones you're guessing at. A good theory, of course, will do both. But the fact that every prediction must in effect pull double duty creates a certain unavoidable tension.

Life satisfaction as a function of time since marriage.

As one illustration of this tension, let's look at a data set that might have been relevant to Darwin: people's life satisfaction over their first ten years of marriage, from a recent study conducted in Germany. Each point on that chart is taken from the study itself; our job is to figure out the formula for a line that would fit those points and extend into the future, allowing us to make predictions past the ten-year mark. One possible formula would use just a single factor to predict life satisfaction: the time since marriage. This would create a straight line on the chart. Another possibility is to use two factors, time and time squared; the resulting line would have a parabolic U-shape, letting it capture a potentially more complex relationship between time and happiness. And if we expand the formula to include yet more factors (time cubed and so on), the line will acquire ever more inflection points, getting more and more "bendy" and flexible. By the time we get to a nine-factor formula, we can capture very complex relationships indeed.

Mathematically speaking, our two-factor model incorporates all the information that goes into the one-factor model, and has another term it could use as well. Likewise, the nine-factor model leverages all of the information at the disposal of the two-factor model, plus potentially lots more. By this logic, it seems like the nine-factor model ought to always give us the best predictions. As it turns out, things are not quite so simple.

Predictions of life satisfaction using models with different numbers of factors.

The result of applying these models to the data is shown above. The one-factor model, unsurprisingly, misses a lot of the exact data points, though it captures the basic trend—a comedown after the honeymoon bliss. However, its straight-line prediction forecasts that this decrease will continue forever, ultimately resulting in infinite misery. Something about that trajectory doesn't sound quite right.

The two-factor model comes closer to fitting the survey data, and its curved shape makes a different long-term prediction, suggesting that after the initial decline life satisfaction more or less levels out over time. Finally, the nine-factor model passes through each and every point on the chart; it is essentially a perfect fit for all the data from the study. In that sense it seems like the nine-factor formula is indeed our best model. But if you look at the predictions it makes for the years not included in the study, you might wonder about just how useful it really is: it predicts misery at the altar, a giddily abrupt rise in satisfaction after several months of marriage, a bumpy roller-coaster ride thereafter, and a sheer drop after year ten. By contrast, the leveling off predicted by the two-factor model is the forecast most consistent with what psychologists and economists say about marriage and happiness. (They believe, incidentally, that it simply reflects a return to normalcy—to people's baseline level of satisfaction with their lives—rather than any displeasure with marriage itself.)

The lesson is this: it is indeed true that including more factors in a model will always, by definition, make it a better fit for the data we have already. But a better fit for the available data does not necessarily mean a better prediction. This tension is formalized as the bias-variance tradeoff. Bias refers to the error introduced by a model that is too simple to capture the underlying relationship (underfitting), while variance is the error from a model that is too complex and overly sensitive to random noise in the training data (overfitting). Reducing bias tends to increase variance, and vice versa, limiting the predictive power of any model. For the marriage data, a simple straight line has high bias but low variance, remaining stable if the study were repeated. The nine-factor model has low bias but high variance, gyrating wildly with minor changes in the data. Neither extreme is ideal; the goal is to find a middle ground that balances these errors.

The Idolatry of Data

If we had copious data, drawn from a perfectly representative sample, completely mistake-free, and representing exactly what we're trying to evaluate, then using the most complex model available would indeed be the best approach. But if we try to perfectly fit our model to the data when any of these factors fails to hold, we risk overfitting. In other words, overfitting poses a danger any time we're dealing with noise or mismeasurement—and we almost always are. There can be errors in how the data were collected, or in how they were reported. Sometimes the phenomena being investigated, such as human happiness, are hard to even define, let alone measure. Thanks to their flexibility, the most complex models available to us can fit any patterns that appear in the data, but this means that they will also do so even when those patterns are mere phantoms and mirages in the noise.

Throughout history, religious texts have warned their followers against idolatry: the worshipping of statues, paintings, relics, and other tangible artifacts in lieu of the intangible deities those artifacts represent. The First Commandment, for instance, warns against bowing down to "any graven image, or any likeness of any thing that is in heaven." And in the Book of Kings, a bronze snake made at God's orders becomes an object of worship and incense-burning, instead of God himself. (God is not amused.) Fundamentally, overfitting is a kind of idolatry of data, a consequence of focusing on what we've been able to measure rather than what matters.

This gap between the data we have and the predictions we want is virtually everywhere. When making a big decision, we can only guess at what will please us later by thinking about the factors important to us right now. (As Harvard's Daniel Gilbert puts it, our future selves often "pay good money to remove the tattoos that we paid good money to get.") When making a financial forecast, we can only look at what correlated with the price of a stock in the past, not what might in the future. Even in our small daily acts this pattern holds: writing an email, we use our own read-through of the text to predict that of the recipient. No less than in public surveys, the data in our own lives are thus also always noisy, at best a proxy metric for the things we really care about. As a consequence, considering more and more factors and expending more effort to model them can lead us into the error of optimizing for the wrong thing—offering prayers to the bronze snake of data rather than the larger force behind it.

Overfitting Everywhere

Once you know about overfitting, you see it everywhere.

Overfitting, for instance, explains the irony of our palates. How can it be that the foods that taste best to us are broadly considered to be bad for our health, when the entire function of taste buds, evolutionarily speaking, is to prevent us from eating things that are bad? The answer is that taste is our body's proxy metric for health. Fat, sugar, and salt are important nutrients, and for a couple hundred thousand years, being drawn to foods containing them was a reasonable measure for a sustaining diet. But being able to modify the foods available to us broke that relationship. We can now add fat and sugar to foods beyond amounts that are good for us, and then eat those foods exclusively rather than the mix of plants, grains, and meats that historically made up the human diet. In other words, we can overfit taste. And the more skillfully we can manipulate food (and the more our lifestyles diverge from those of our ancestors), the more imperfect a metric taste becomes. Our human agency thus turns into a curse, making us dangerously able to have exactly what we want even when we don't quite want exactly the right thing.

Beware: when you go to the gym to work off the extra weight from all that sugar, you can also risk overfitting fitness. Certain visible signs of fitness—low body fat and high muscle mass, for example—are easy to measure, and they are related to, say, minimizing the risk of heart disease and other ailments. But they, too, are an imperfect proxy measure. Overfitting the signals—adopting an extreme diet to lower body fat and taking steroids to build muscle, perhaps—can make you the picture of good health, but only the picture.

Overfitting also shows up in sports. For instance, Tom has been a fencer, on and off, since he was a teenager. The original goal of fencing was to teach people how to defend themselves in a duel, hence the name: "defencing." And the weapons used in modern fencing are similar to those that were used to train for such encounters. (This is particularly true of the épée, which was still used in formal duels less than fifty years ago.) But the introduction of electronic scoring equipment—a button on the tip of the blade that registers a hit—has changed the nature of the sport, and techniques that would serve you poorly in a serious duel have become critical skills in competition. Modern fencers use flexible blades that allow them to "flick" the button at their opponent, grazing just hard enough to register and score. As a result, they can look more like they're cracking thin metal whips at each other than cutting or thrusting. It's as exciting a sport as ever, but as athletes overfit their tactics to the quirks of scorekeeping, it becomes less useful in instilling the skills of real-world swordsmanship. Similarly, a fencer who practices too specifically for a single opponent may overfit their tactics, becoming so attuned to that style that they struggle to adapt to others. Broadening training to a diverse set of opponents and situations helps keep strategies robust and prevents this form of overfitting.

Perhaps nowhere, however, is overfitting as powerful and troublesome as in the world of business. "Incentive structures work," as Steve Jobs put it. "So you have to be very careful of what you incent people to do, because various incentive structures create all sorts of consequences that you can't anticipate." Sam Altman, president of the startup incubator Y Combinator, echoes Jobs's words of caution: "It really is true that the company will build whatever the CEO decides to measure." In fact, it's incredibly difficult to come up with incentives or measurements that do not have some kind of perverse effect. In the 1950s, Cornell management professor V. F. Ridgway cataloged a host of such "Dysfunctional Consequences of Performance Measurements." At a job-placement firm, staffers were evaluated on the number of interviews they conducted, which motivated them to run through the meetings as quickly as possible, without spending much time actually helping their clients find jobs. At a federal law enforcement agency, investigators given monthly performance quotas were found to pick easy cases at the end of the month rather than the most urgent ones. And at a factory, focusing on production metrics led supervisors to neglect maintenance and repairs, setting up future catastrophe. Such problems can't simply be dismissed as a failure to achieve management goals. Rather, they are the opposite: the ruthless and clever optimization of the wrong thing.

The twenty-first-century shift into real-time analytics has only made the danger of metrics more intense. Avinash Kaushik, digital marketing evangelist at Google, warns that trying to get website users to see as many ads as possible naturally devolves into trying to cram sites with ads: "When you are paid on a [cost per thousand impressions] basis the incentive is to figure out how to show the most possible ads on every page [and] ensure the visitor sees the most possible pages on the site.… That incentive removes a focus from the important entity, your customer, and places it on the secondary entity, your advertiser." The website might gain a little more money in the short term, but ad-crammed articles, slow-loading multi-page slide shows, and sensationalist clickbait headlines will drive away readers in the long run. Kaushik's conclusion: "Friends don't let friends measure Page Views. Ever."

In some cases, the difference between a model and the real world is literally a matter of life and death. In the military and in law enforcement, for example, repetitive, rote training is considered a key means for instilling line-of-fire skills. The goal is to drill certain motions and tactics to the point that they become totally automatic. But when overfitting creeps in, it can prove disastrous. There are stories of police officers who find themselves, for instance, taking time out during a gunfight to put their spent casings in their pockets—good etiquette on a firing range. As former Army Ranger and West Point psychology professor Dave Grossman writes, "After the smoke had settled in many real gunfights, officers were shocked to discover empty brass in their pockets with no memory of how it got there. On several occasions, dead cops were found with brass in their hands, dying in the middle of an administrative procedure that had been drilled into them." Similarly, the FBI was forced to change its training after agents were found reflexively firing two shots and then holstering their weapon—a standard cadence in training—regardless of whether their shots had hit the target and whether there was still a threat. Mistakes like these are known in law enforcement and the military as "training scars," and they reflect the fact that it's possible to overfit one's own preparation. In one particularly dramatic case, an officer instinctively grabbed the gun out of the hands of an assailant and then instinctively handed it right back—just as he had done time and time again with his trainers in practice.

Detecting Overfitting: Cross-Validation

Because overfitting presents itself initially as a theory that perfectly fits the available data, it may seem insidiously hard to detect. How can we expect to tell the difference between a genuinely good model and one that's overfitting? In an educational setting, how can we distinguish between a class of students excelling at the subject matter and a class merely being "taught to the test"? In the business world, how can we tell a genuine star performer from an employee who has just cannily overfit their work to the company's key performance indicators—or to the boss's perception? Teasing apart those scenarios is indeed challenging, but it is not impossible.

Research in machine learning has yielded several concrete strategies for detecting overfitting, and one of the most important is what's known as Cross-Validation. Simply put, Cross-Validation means assessing not only how well a model fits the data it's given, but how well it generalizes to data it hasn't seen. Paradoxically, this may involve using less data. In the marriage example, we might "hold back," say, two points at random, and fit our models only to the other eight. We'd then take those two test points and use them to gauge how well our various functions generalize beyond the eight "training" points they've been given. The two held-back points function as canaries in the coal mine: if a complex model nails the eight training points but wildly misses the two test points, it's a good bet that overfitting is at work.

Aside from withholding some of the available data points, it is also useful to consider testing the model with data derived from some other form of evaluation entirely. As we have seen, the use of proxy metrics—taste as a proxy for nutrition, number of cases solved as a proxy for investigator diligence—can also lead to overfitting. In these cases, we'll need to cross-validate the primary performance measure we're using against other possible measures. In schools, for example, standardized tests offer a number of benefits, including a distinct economy of scale: they can be graded cheaply and rapidly by the thousands. Alongside such tests, however, schools could randomly assess some small fraction of the students—one per class, say, or one in a hundred—using a different evaluation method, perhaps something like an essay or an oral exam. (Since only a few students would be tested this way, having this secondary method scale well is not a big concern.) The standardized tests would provide immediate feedback—you could have students take a short computerized exam every week and chart the class's progress almost in real time, for instance—while the secondary data points would serve to cross-validate: to make sure that the students were actually acquiring the knowledge that the standardized test is meant to measure, and not simply getting better at test-taking. If a school's standardized scores rose while its "nonstandardized" performance moved in the opposite direction, administrators would have a clear warning sign that "teaching to the test" had set in, and the pupils' skills were beginning to overfit the mechanics of the test itself.

Cross-Validation also offers a suggestion for law enforcement and military personnel looking to instill good reflexes without hammering in habits from the training process itself. Just as essays and oral exams can cross-validate standardized tests, so occasional unfamiliar "cross-training" assessments might be used to measure whether reaction time and shooting accuracy are generalizing to unfamiliar tasks. If they aren't, then that's a strong signal to change the training regimen. While nothing may truly prepare one for actual combat, exercises like this may at least warn in advance where "training scars" are likely to have formed.

How to Combat Overfitting: Penalizing Complexity

If you can't explain it simply, you don't understand it well enough. —ANONYMOUS

We've seen some of the ways that overfitting can rear its head, and we've looked at some of the methods to detect and measure it. But what can we actually do to alleviate it? From a statistics viewpoint, overfitting is a symptom of being too sensitive to the actual data we've seen. The solution, then, is straightforward: we must balance our desire to find a good fit against the complexity of the models we use to do so.

One way to choose among several competing models is the Occam's razor principle, which suggests that, all things being equal, the simplest possible hypothesis is probably the correct one. Of course, things are rarely completely equal, so it's not immediately obvious how to apply something like Occam's razor in a mathematical context. Grappling with this challenge in the 1960s, Russian mathematician Andrey Tikhonov proposed one answer: introduce an additional term to your calculations that penalizes more complex solutions. If we introduce a complexity penalty, then more complex models need to do not merely a better job but a significantly better job of explaining the data to justify their greater complexity. Computer scientists refer to this principle—using constraints that penalize models for their complexity—as Regularization.

So what do these complexity penalties look like? One algorithm, discovered in 1996 by biostatistician Robert Tibshirani, is called the Lasso and uses as its penalty the total weight of the different factors in the model. By putting this downward pressure on the weights of the factors, the Lasso drives as many of them as possible completely to zero. Only the factors that have a big impact on the results remain in the equation—thus potentially transforming, say, an overfitted nine-factor model into a simpler, more robust formula with just a couple of the most critical factors.

Techniques like the Lasso are now ubiquitous in machine learning, but the same kind of principle—a penalty for complexity—also appears in nature. Living organisms get a certain push toward simplicity almost automatically, thanks to the constraints of time, memory, energy, and attention. The burden of metabolism, for instance, acts as a brake on the complexity of organisms, introducing a caloric penalty for overly elaborate machinery. The fact that the human brain burns about a fifth of humans' total daily caloric intake is a testament to the evolutionary advantages that our intellectual abilities provide us with: the brain's contributions must somehow more than pay for that sizable fuel bill. On the other hand, we can also infer that a substantially more complex brain probably didn't provide sufficient dividends, evolutionarily speaking. We're as brainy as we have needed to be, but not extravagantly more so.

The same kind of process is also believed to play a role at the neural level. In computer science, software models based on the brain, known as "artificial neural networks," can learn arbitrarily complex functions—they're even more flexible than our nine-factor model above—but precisely because of this very flexibility they are notoriously vulnerable to overfitting. Actual, biological neural networks sidestep some of this problem because they need to trade off their performance against the costs of maintaining it. Neuroscientists have suggested, for instance, that brains try to minimize the number of neurons that are firing at any given moment—implementing the same kind of downward pressure on complexity as the Lasso.

Language forms yet another natural Lasso: complexity is punished by the labor of speaking at greater length and the taxing of our listener's attention span. Business plans get compressed to an elevator pitch; life advice becomes proverbial wisdom only if it is sufficiently concise and catchy. And anything that needs to be remembered has to pass through the inherent Lasso of memory.

The Upside of Heuristics

The economist Harry Markowitz won the 1990 Nobel Prize in Economics for developing modern portfolio theory: his groundbreaking "mean-variance portfolio optimization" showed how an investor could make an optimal allocation among various funds and assets to maximize returns at a given level of risk. So when it came time to invest his own retirement savings, it seems like Markowitz should have been the one person perfectly equipped for the job. What did he decide to do? I should have computed the historical covariances of the asset classes and drawn an efficient frontier. Instead, I visualized my grief if the stock market went way up and I wasn't in it—or if it went way down and I was completely in it. My intention was to minimize my future regret. So I split my contributions fifty-fifty between bonds and equities.

Why in the world would he do that? The story of the Nobel Prize winner and his investment strategy could be presented as an example of human irrationality: faced with the complexity of real life, he abandoned the rational model and followed a simple heuristic. But it's precisely because of the complexity of real life that a simple heuristic might in fact be the rational solution. When it comes to portfolio management, it turns out that unless you're highly confident in the information you have about the markets, you may actually be better off ignoring that information altogether. Applying Markowitz's optimal portfolio allocation scheme requires having good estimates of the statistical properties of different investments. An error in those estimates can result in very different asset allocations, potentially increasing risk. In contrast, splitting your money evenly across stocks and bonds is not affected at all by what data you've observed. This strategy doesn't even try to fit itself to the historical performance of those investment types—so there's no way it can over fit.

Of course, just using a fifty-fifty split is not necessarily the complexity sweet spot, but there's something to be said for it. If you happen to know the expected mean and expected variance of a set of investments, then use mean-variance portfolio optimization—the optimal algorithm is optimal for a reason. But when the odds of estimating them all correctly are low, and the weight that the model puts on those untrustworthy quantities is high, then an alarm should be going off in the decision-making process: it's time to regularize.

Inspired by examples like Markowitz's retirement savings, psychologists Gerd Gigerenzer and Henry Brighton have argued that the decision-making shortcuts people use in the real world are in many cases exactly the kind of thinking that makes for good decisions. "In contrast to the widely held view that less processing reduces accuracy," they write, "the study of heuristics shows that less information, computation, and time can in fact improve accuracy." A heuristic that favors simpler answers—with fewer factors, or less computation—offers precisely these "less is more" effects.

Imposing penalties on the ultimate complexity of a model is not the only way to alleviate overfitting, however. You can also nudge a model toward simplicity by controlling the speed with which you allow it to adapt to incoming data. This makes the study of overfitting an illuminating guide to our history—both as a society and as a species.

The Weight of History

Every food a living rat has eaten has, necessarily, not killed it. —SAMUEL REVUSKY AND ERWIN BEDARF, "ASSOCIATION OF ILLNESS WITH PRIOR INGESTION OF NOVEL FOODS"

The soy milk market in the United States more than quadrupled from the mid-1990s to 2013. But by the end of 2013, according to news headlines, it already seemed to be a thing of the past, a distant second place to almond milk. As food and beverage researcher Larry Finkel told Bloomberg Businessweek: "Nuts are trendy now. Soy sounds more like old-fashioned health food." The Silk company, famous for popularizing soy milk (as the name implies), reported in late 2013 that its almond milk products had grown by more than 50% in the previous quarter alone. Meanwhile, in other beverage news, the leading coconut water brand, Vita Coco, reported in 2014 that its sales had doubled since 2011—and had increased an astounding three-hundred-fold since 2004. As the New York Times put it, "coconut water seems to have jumped from invisible to unavoidable without a pause in the realm of the vaguely familiar." Meanwhile, the kale market grew by 40% in 2013 alone. The biggest purchaser of kale the year before had been Pizza Hut, which put it in their salad bars—as decoration.

Some of the most fundamental domains of human life, such as the question of what we should put in our bodies, seem curiously to be the ones most dominated by short-lived fads. Part of what enables these fads to take the world by storm is how quickly our culture can change. Information now flows through society faster than ever before, while global supply chains enable consumers to rapidly change their buying habits en masse (and marketing encourages them to do so). If some particular study happens to suggest a health benefit from, say, star anise, it can be all over the blogosphere within the week, on television the week after that, and in seemingly every supermarket in six months, with dedicated star anise cookbooks soon rolling off the presses. This breathtaking speed is both a blessing and a curse.

In contrast, if we look at the way organisms—including humans—evolve, we notice something intriguing: change happens slowly. This means that the properties of modern-day organisms are shaped not only by their present environments, but also by their history. For example, the oddly cross-wired arrangement of our nervous system (the left side of our body controlled by the right side of our brain and vice versa) reflects the evolutionary history of vertebrates. This phenomenon, called "decussation," is theorized to have arisen at a point in evolution when early vertebrates' bodies twisted 180 degrees with respect to their heads; whereas the nerve cords of invertebrates such as lobsters and earthworms run on the "belly" side of the animal, vertebrates have their nerve cords along the spine instead. The human ear offers another example. Viewed from a functional perspective, it is a system for translating sound waves into electrical signals by way of amplification via three bones: the malleus, incus, and stapes. This amplification system is impressive—but the specifics of how it works have a lot to do with historical constraints. Reptiles, it turns out, have just a single bone in their ear, but additional bones in the jaw that mammals lack. Those jawbones were apparently repurposed in the mammalian ear. So the exact form and configuration of our ear anatomy reflects our evolutionary history at least as much as it does the auditory problem being solved.

The concept of overfitting gives us a way of seeing the virtue in such evolutionary baggage. Though crossed-over nerve fibers and repurposed jawbones may seem like suboptimal arrangements, we don't necessarily want evolution to fully optimize an organism to every shift in its environmental niche—or, at least, we should recognize that doing so would make it extremely sensitive to further environmental changes. Having to make use of existing materials, on the other hand, imposes a kind of useful restraint. It makes it harder to induce drastic changes in the structure of organisms, harder to overfit. As a species, being constrained by the past makes us less perfectly adjusted to the present we know but helps keep us robust for the future we don't.

A similar insight might help us resist the quick-moving fads of human society. When it comes to culture, tradition plays the role of the evolutionary constraints. A bit of conservatism, a certain bias in favor of history, can buffer us against the boom-and-bust cycle of fads. That doesn't mean we ought to ignore the latest data either, of course. Jump toward the bandwagon, by all means—but not necessarily on it.

In machine learning, the advantages of moving slowly emerge most concretely in a regularization technique known as Early Stopping. When we looked at the German marriage survey data at the beginning of the chapter, we went straight to examining the best-fitted one-, two-, and nine-factor models. In many situations, however, tuning the parameters to find the best possible fit for given data is a process in and of itself. What happens if we stop that process early and simply don't allow a model the time to become too complex? Again, what might seem at first blush like being halfhearted or unthorough emerges, instead, as an important strategy in its own right. Many prediction algorithms, for instance, start out by searching for the single most important factor rather than jumping to a multi-factor model. Only after finding that first factor do they look for the next most important factor to add to the model, then the next, and so on. Their models can therefore be kept from becoming overly complex simply by stopping the process short, before overfitting has had a chance to creep in. A related approach to calculating predictions considers one data point at a time, with the model tweaked to account for each new point before more points are added; there, too, the complexity of the model increases gradually, so stopping the process short can help keep it from overfitting.

This kind of setup—where more time means more complexity—characterizes a lot of human endeavors. Giving yourself more time to decide about something does not necessarily mean that you'll make a better decision. But it does guarantee that you'll end up considering more factors, more hypotheticals, more pros and cons, and thus risk overfitting.

When to Think Less

Tom had exactly this experience when he became a professor. His first semester, teaching his first class ever, he spent a huge amount of time perfecting his lectures—more than ten hours of preparation for every hour of class. His second semester, teaching a different class, he wasn't able to put in as much time, and worried that it would be a disaster. But a strange thing happened: the students liked the second class. In fact, they liked it more than the first one. Those extra hours, it turned out, had been spent nailing down nitty-gritty details that only confused the students, and wound up getting cut from the lectures the next time Tom taught the class. The underlying issue, Tom eventually realized, was that he'd been using his own taste and judgment as a kind of proxy metric for his students'. This proxy metric worked reasonably well as an approximation, but it wasn't worth overfitting—which explained why spending extra hours painstakingly "perfecting" all the slides had been counterproductive.

The effectiveness of regularization in all kinds of machine-learning tasks suggests that we can make better decisions by deliberately thinking and doing less. If the factors we come up with first are likely to be the most important ones, then beyond a certain point thinking more about a problem is not only going to be a waste of time and effort—it will lead us to worse solutions. Early Stopping provides the foundation for a reasoned argument against reasoning, the thinking person's case against thought. But turning this into practical advice requires answering one more question: when should we stop thinking?

As with all issues involving overfitting, how early to stop depends on the gap between what you can measure and what really matters. If you have all the facts, they're free of all error and uncertainty, and you can directly assess whatever is important to you, then don't stop early. Think long and hard: the complexity and effort are appropriate. But that's almost never the case. If you have high uncertainty and limited data, then do stop early by all means. If you don't have a clear read on how your work will be evaluated, and by whom, then it's not worth the extra time to make it perfect with respect to your own (or anyone else's) idiosyncratic guess at what perfection might be. The greater the uncertainty, the bigger the gap between what you can measure and what matters, the more you should watch out for overfitting—that is, the more you should prefer simplicity, and the earlier you should stop. When you're truly in the dark, the best-laid plans will be the simplest.

When our expectations are uncertain and the data are noisy, the best bet is to paint with a broad brush, to think in broad strokes. Sometimes literally. As entrepreneurs Jason Fried and David Heinemeier Hansson explain, the further ahead they need to brainstorm, the thicker the pen they use—a clever form of simplification by stroke size: When we start designing something, we sketch out ideas with a big, thick Sharpie marker, instead of a ball-point pen. Why? Pen points are too fine. They're too high-resolution. They encourage you to worry about things that you shouldn't worry about yet, like perfecting the shading or whether to use a dotted or dashed line. You end up focusing on things that should still be out of focus. A Sharpie makes it impossible to drill down that deep. You can only draw shapes, lines, and boxes. That's good. The big picture is all you should be worrying about in the beginning. As McGill's Henry Mintzberg puts it, "What would happen if we started from the premise that we can't measure what matters and go from there? Then instead of measurement we'd have to use something very scary: it's called judgment."

The upshot of Early Stopping is that sometimes it's not a matter of choosing between being rational and going with our first instinct. Going with our first instinct can be the rational solution. The more complex, unstable, and uncertain the decision, the more rational an approach that is.

To return to Darwin, his problem of deciding whether to propose could probably have been resolved based on just the first few pros and cons he identified, with the subsequent ones adding to the time and anxiety expended on the decision without necessarily aiding its resolution (and in all likelihood impeding it). What seemed to make up his mind was the thought that "it is intolerable to think of spending one's whole life like a neuter bee, working, working, & nothing after all." Children and companionship—the very first points he mentioned—were precisely those that ultimately swayed him in favor of marriage. His book budget was a distraction.

Before we get too critical of Darwin, however, painting him as an inveterate overthinker, it's worth taking a second look at this page from his diary. Seeing it in facsimile shows something fascinating. Darwin was no Franklin, adding assorted considerations for days. Despite the seriousness with which he approached this life-changing choice, Darwin made up his mind exactly when his notes reached the bottom of the diary sheet. He was regularizing to the page. This is reminiscent of both Early Stopping and the Lasso: anything that doesn't make the page doesn't make the decision. His mind made up to marry, Darwin immediately went on to overthink the timing. "When? Soon or Late," he wrote above another list of pros and cons, considering everything from happiness to expenses to "awkwardness" to his long-standing desire to travel in a hot air balloon and/or to Wales. But by the end of the page he resolved to "Never mind, trust to chance"—and the result, within several months' time, was a proposal to Emma Wedgwood, the start of a fulfilling partnership and a happy family life.