Anything That Can Go Wrong, Will

Anything That Can Go Wrong, Will

Actions can have consequences beyond those intended. Some recur often enough to anticipate: people adapt to incentives, shared resources get overused, and small compromises accumulate.

In 2016, the UK government asked the public to help name a new polar research ship. Individuals could submit names and then vote on them in an online poll. More than seven thousand names were submitted, but one name won easily, with 124,109 votes: RRS Boaty McBoatface . (The ship was eventually named RRS Sir David Attenborough instead.) The exact winning name would have been hard to predict. The possibility of a popular joke entry was easier to foresee. People turn open contests like this into jokes all the time.

In 2012, Mountain Dew held a similar campaign to name a new soda, but they quickly closed it down when "Diabeetus" and "Hitler Did Nothing Wrong" appeared near the top of the rankings. Also that year, Walmart teamed up with Sheets Energy Strips and offered to put on a concert by international recording artist Pitbull at the Walmart location that received the most new Facebook likes. After an internet prankster took hold of the contest, Walmart's most remote store, in Kodiak, Alaska, won. Walmart and Pitbull still held the concert there and they even had the prankster who rigged the contest join Pitbull on the trip!

Unintended consequences are not a laughing matter under more serious circumstances. The book gives prescription opioids as a more serious example. They can relieve pain but also carry risks of addiction and overdose. Some people who misuse them later turn to heroin. According to the National Institutes of Health, in the U.S., nearly half of young people who inject heroin started abusing prescription opioids first. Patients' susceptibility to opioid addiction and abuse has substantially contributed to the deadliest drug crisis in American history. As reported by The New York Times on November 29, 2018, more people died from drug overdoses in 2017 than from HIV/AIDS, car crashes, or gun deaths in the years of their respective peaks. Of course, no doctor prescribing painkillers intends for their patients to die—these deaths are unintended consequences. The models below help identify risks before acting, though they cannot make every consequence predictable.

Harm Thy Neighbor, Unintentionally

There is a class of unintended consequences that arise when a lot of people choose what they think is best for them individually, but the sum total of the decisions creates a worse outcome for everyone. To illustrate how this works, consider Boston Common, the oldest public park in the United States. Before it was a park, way back in the 1630s, this fifty-acre plot of land in downtown Boston, Massachusetts, was a grazing pasture for cows, with local families using it collectively as common land. In England, this type of land is referred to legally as commons . Pasture commons present a problem, though: Each additional cow that a farmer gets benefits their family, but if all the farmers keep getting new cows, then the commons can be depleted. All farmers would experience the negative effects of overgrazing on the health of their herds and land.

In an 1833 essay, "Two Lectures on the Checks to Population," economist William Lloyd described a similar, but hypothetical, overgrazing scenario, now called the tragedy of the commons . However, unbeknownst to him, his hypothetical situation had really occurred in Boston Common two hundred years earlier (and many other times before and since). More affluent families did in fact keep buying more cows, leading to overgrazing, until, in 1646, a limit of seventy cows was imposed on Boston Common.

Any shared resource, or commons , is vulnerable to this tragedy. Overfishing, deforestation, and dumping waste have obvious parallels to overgrazing, though this model extends far beyond environmental issues. Each additional spam message benefits the spammer who sends it while simultaneously degrading the entire email system. Collective overuse of antibiotics in medicine and agriculture is leading to dangerous antibiotic resistance. People make self-serving edits to Wikipedia articles, diminishing the overall reliability of the encyclopedia. In each of these cases, an individual makes what appears to be a rational decision (e.g., prescribing an antibiotic to a patient who might have a bacterial infection). They use the common resource for their own benefit at little or no cost (e.g., each course of treatment has only a small chance of increasing resistance). But as more and more people make the same decision, the common resource is collectively depleted, reducing the ability for everyone to benefit from it in the future (e.g., the antibiotic becomes much less useful).

More broadly, the tragedy of the commons arises from what is called the tyranny of small decisions , where a series of small , individually rational decisions ultimately leads to a system-wide negative consequence, or tyranny . It's death by a thousand cuts.

You've probably gone out to dinner with friends expecting that you will equally split the check. At dinner, each person is faced with a decision to order an expensive meal or a cheaper one. When dining alone, people often order the cheaper meal. However, when they know that the cost of dinner is shared by the whole group, people tend to opt for the expensive meal. If everyone does this then everyone ends up paying more!

Ecologist William E. Odum made the connection between the tyranny of small decisions and environmental degradation in his 1982 BioScience article: "Much of the current confusion and distress surrounding environmental issues can be traced to decisions that were never consciously made, but simply resulted from a series of small decisions." It's the individual decision to place a well here, cut down some trees there, build a factory over there—over time these isolated decisions aggregate to create widespread problems in our environment that are increasingly difficult to reverse.

You can also find the tyranny of small decisions in your own life. Think of those small credit card purchases or expenses that seem individually warranted at the time, but collectively add up to significant credit card bills or cash crunches. Professionally, it may be the occasional distractions and small procrastinations that, in aggregate, make your deadlines hard to reach.

A view of the whole system can reveal costs that are easy to miss in separate decisions. Budgets, shared limits, and coordination can then help control them. When the decisions are all your own, you could do this for yourself. For example, to stop your out-of-control spending, you could self-impose a budget, checking each potential purchase against the budget to see if it's compatible with your spending plan. You could do the same for your time management, by more strictly regulating your calendar.

When several people share a resource, they need a way to coordinate its use. That may involve agreed rules and monitoring by the users themselves, or intervention by an authority, as in Boston’s limit on cows. Company expense policies that help prevent overspending are an organizational example.

Another cause of issues like the tragedy of the commons is the free rider problem , where some people get a free ride by using a resource without paying for it. People or companies who cheat on their taxes are free riders to government services they use, such as infrastructure and the legal system. If you've ever worked on a team project where one person didn't do anything substantive, that person was free-riding on the rest of the group. Another familiar example: Has anyone ever leeched off your wi-fi or Netflix account? Or perhaps you've been the free rider?

Free-riding is commonplace with public goods , such as national militaries, broadcast television, even the air we breathe. As you can see from these examples, it is usually difficult to exclude people from using public goods, because they are broadly available ( public ). Since one person's use does not significantly reduce a public good's availability to others, it might seem as though there is no harm in free-riding. If too few people contribute, a public good may be underprovided or poorly maintained. This funding problem differs from depleting a shared resource through overuse, though both involve conflicts between individual and collective interests.

Vaccinations provide an illustrative example that combines all these models (tragedy of the commons, free rider problem, tyranny of small decisions, public goods), plus one more: herd immunity . Diseases can spread only when they have an eligible host to infect. However, when the vast majority of people are vaccinated against a disease, there are very few eligible new hosts, since most people (in the herd ) are immune from infection due to getting vaccinated. As a result, the overall public is less susceptible to outbreaks of the disease. In this example, the public good is a disease-free environment due to herd immunity, and the free riders are those who take advantage of this public good by not getting vaccinated. The tyranny of small decisions can arise when enough individuals choose not to get vaccinated, resulting in an outbreak of the disease, creating a tragedy of the commons.

In practice, the percentage of people who need to be vaccinated for a given disease to achieve herd immunity varies by how contagious the disease is. For measles, an extremely contagious disease, the threshold is about 95 percent. A rate below 95 percent leaves a community more vulnerable; the threshold is not a sharp boundary that guarantees safety above it.

Before the vaccine’s introduction in 1963, the book cites more than 500,000 reported U.S. measles cases and more than 400 deaths annually. Vaccination dramatically reduced both, though it did not permanently eliminate every measles death.

In recent years, some parents have refused to vaccinate their kids for measles and other diseases due to the belief that vaccines are linked to autism, based on since-discredited and known-to-be-fraudulent research. These people who choose not to vaccinate are free-riding on the herd immunity from the people who do choose to vaccinate. Low disease incidence can obscure the protection vaccination provides. Where vaccination rates fall, susceptibility can accumulate. For example, in 2017, more than seventy-five people in Minnesota, most of whom were unvaccinated, contracted measles. We can expect outbreaks like this to continue as long as there exist communities with vaccination rates below the herd immunity threshold.

Unfortunately, some people cannot be medically immunized, such as infants, people with severe allergies, and those with suppressed immune systems. At no fault of their own, they face the potentially deadly consequences of the anti-vaccination movement, a literal tragedy of the commons.

Herd immunity as a concept is useful beyond the medical context. It also offers an analogy for social, cultural, business, and industry norms, though these do not have the same biological thresholds. If enough infractions are left unchecked, their incidence can start increasing quickly, creating a new negative norm that can be difficult to unwind. The book illustrates this with an Italian phrase about tax evasion: "Only fools pay . " The point of the example is that widespread evasion can normalize noncompliance and become hard to reverse. There is no literal herd-immunity threshold for paying taxes. It can be difficult to put the genie back in the bottle.

Imagine a once pristine place that is now littered with garbage and graffiti. Once it has become dirtied, that state can quickly become the new normal, and the longer it remains dirty, the more likely it will remain in the dirty state. Hollowed-out urban centers like Detroit or disaster-ridden areas like parts of New Orleans have seen this scenario play out in the recent past. People who don't want to live with the effects of the degradation but also don't want to do the hard work to clean it up may simply move out of the area or visit less, further degrading the space due to lack of a tax base to fund proper maintenance. It then takes a much larger effort to revitalize the area than it would have taken to keep it nice in the first place. Not only do the funds need to be found for the revitalization effort, but the expectation that it should be a nice place has to be reset, and then people need to be drawn back to it.

Some of these unintended consequences are externalities: costs or benefits imposed on others that are not fully reflected in a transaction’s price. Externalities need not be unexpected or unintentional. The infant who cannot be vaccinated, for example, receives a positive externality from those who choose to vaccinate (less chance of getting the disease) and a negative externality from those who do not (more chance of getting the disease). Similarly, air pollution by a factory creates a negative externality for the people living nearby—low air quality. If that same company, though, trained all its workers in first aid, residents would receive a positive externality if some of those workers used that training to save lives outside of work.

Externalities occur wherever there are spillover effects , which happen when an effect of an activity spills over outside the core interactions of the activity. The effects of smoking spill over to surrounding people through secondhand smoke and, more broadly, through increased public healthcare expenditures.

Sometimes spillover effects can be more subtle. When you buy a car, you add congestion to the roads you drive on, a cost borne by everyone who drives on the same roads. Or when you keep your neighbors up with loud music, you deprive them of sleep, causing them to be less productive.

Over the next few days, look out for externalities. When you see or hear about someone or some organization taking an action, think about people not directly related to the action who might experience benefit or harm from it. When you see someone litter, be aware of the negative externality borne by everyone else who uses that space. If enough people litter, a dirty space may come to seem normal, encouraging further littering.

Addressing negative externalities is often referred to as internalizing them. Internalizing is an attempt to require the entity that causes the negative externality to pay for it. Ideally the "price" attached to the unwanted activity is high enough that it totally covers the cost of dealing with that activity's consequences. A high price can also stop the harm from occurring in the first place. A five-hundred-dollar littering fine may deter littering, depending partly on whether people expect it to be enforced.

There are many ways to internalize negative externalities, including taxes, fines, regulation, and lawsuits. Smoking externalities are internalized via cigarette taxes and higher health insurance premiums for smokers. Traffic congestion externalities are internalized through tolls. On a personal level, your neighbor might file a noise complaint against you if you consistently play music too loud.

Another way to internalize externalities is through a marketplace. Ronald Coase won the Nobel Prize in economics in 1991 in part for what has become known as the Coase theorem , essentially a description of how a natural marketplace can internalize a negative externality. Coase showed how bargaining can resolve an externality efficiently when rights are clearly defined and enforceable and transaction costs are negligible. Those conditions matter: identifying everyone affected, negotiating an agreement, and enforcing it can be costly.

If you recall the Boston Common example, the externality from overgrazing was internalized by setting a limit on the number of cows per farmer (regulation). There were no individually tradable grazing rights in this example. The Coase theorem holds that instead of limiting the cows, another solution would have been to simply divide the grazing rights to the commons property among the farmers. The farmers could then trade the grazing rights among themselves, creating an efficient marketplace for the use of the commons.

Governments have similarly tried to address the negative externalities from the burning of fossil fuels (e.g., climate change) through cap-and-trade systems, which are modern-day applications of the Coase theorem. The way these systems work is that the government requires emitters to hold permits for the amount of pollutants they emit. The government also sets a fixed number of total permits, which serves as the emission cap in the market, similar to the imposed limit on the number of cows that could graze on Boston Common. Then companies can trade permits on an open exchange. Tradable permits can lower the cost of meeting an emissions cap by allowing reductions where they are cheapest. Their success still depends on setting the cap, monitoring emissions, and enforcing the rules; the existence of a market does not by itself ensure the ideal conditions for Coasean bargaining.

If you're in charge of any system or policy, you want to think through the possible negative externalities ahead of time and devise ways to avoid them. What spillover effects could occur, and who would be affected by them? Is there a common resource that free riders could abuse or that could degrade into a tragedy of the commons? Is there another way to set up the policy or system that would reduce possible negative effects?

Risky Business

Another set of unintended consequences can arise when people assess risk differently based on their individual positions and perspectives. These types of complications happen a lot with insurance, where risk assessments create financial consequences. For example, rental insurance may reduce a driver’s financial exposure to damage and thereby weaken an incentive to take care. This phenomenon, known as moral hazard , is where you take on more risk, or hazard , once you have information that encourages you to believe you are more protected. It has been a concern of the insurance industry since the seventeenth century! A related idea is risk compensation: feeling better protected may sometimes encourage more risk-taking. That possibility does not establish that wearing a bike helmet increases overall risk, or that a crash affects only the rider. Moral hazards can also occur when a person or company serves as an agent for another person or company, making decisions on behalf of this entity, known as the principal . The problem arises when the agent takes on more risk than the principal would if the principal were acting alone, since the agent is more protected when things go wrong. For instance, an adviser rewarded for gains while bearing little of the losses may have an incentive to take more risk than a client wants. The problem depends on the compensation and oversight arrangements, rather than following automatically from delegation.

Agency can lead to other issues as well, collectively known as the principal-agent problem , where the self-interest of the agent may lead to suboptimal results for the principal across a wide variety of circumstances. Politicians don't always act in the best interest of their constituents; real estate agents don't always act in the best interest of their sellers; financial brokers don't always act in the best interest of their clients; corporate management doesn't always act in the best interest of its shareholders—you get the idea. The agent's self-interest can trump the principal's interests.

Some fascinating studies of this concept have measured the behavior of agents when they are serving themselves compared with how they serve others. Real estate agents tend to sell their own houses at higher prices compared with their clients' houses, in large part because they are willing to leave them on the market longer. In Freakonomics, Steven Levitt and Stephen Dubner explain the incentive: an agent may receive only a small fraction of the additional sale price, while bearing the time and effort of keeping a property on the market.

Moral hazard and principal-agent problems can occur because of asymmetric information , where one side of a transaction has different information than the other side; that is, the available information is not symmetrically distributed. Real estate agents have more information about the real estate market than sellers, so it is hard to question their recommendations. Similarly, a financial adviser generally has more information about the financial markets than their clients. It is also not always completely transparent to principals how agents are compensated, which might cause principals to make different decisions than they would if they had the full picture. If you knew that your financial adviser was getting paid to recommend a financial product to you, you might be less likely to invest in it. Disclosure laws and the increase in open information via the internet can reduce the effects of asymmetric information.

Sometimes, though, the consumer has the upper hand when it comes to asymmetric information. This is often the case with insurance products, where the person or company applying for insurance usually knows more about their own risk profile than the insurance company does. When parties select transactions that they think will benefit them, based at least partially on their own private information, that's called adverse selection . People who know they are going to need dental work are more likely to seek out dental insurance. This unfortunately drives up the price for everyone.

Two ways to mitigate adverse selection in the insurance market are to mandate participation, as many localities do for car insurance, and to distinguish subpopulations based on their risk profiles, as life insurers do for smokers.

Like crossing a herd immunity threshold, rampant and persistent asymmetric information in a market can lead to its collapse. Consider a used car market where the sellers know the quality of their cars, but the buyers cannot distinguish between lemons (bad cars) and peaches (good cars). In such a market, buyers will want to pay only an average-quality price for the cars on the market, since they can't tell the difference between peaches and lemons. Sellers who know their cars are peaches, however, will not want to sell them in this market because they know their cars are worth more than the average price. As they pull their peaches out of the market, the average quality drops and, in turn, the price of the used cars left in the market keeps dropping. The sellers of lemons free-ride on the market until it collapses into just a market of lemons.

Adverse selection was an early concern with the state health insurance exchanges as part of the Affordable Care Act (ACA) in the United States. Extending the metaphor, the lemons are sick people applying to the exchanges, and the peaches are healthy people applying. There was an individual mandate requiring health insurance, but the penalties for not complying were low, so the concern was that many healthy people would just opt to pay the fine rather than participate. Sick people, who need the insurance, would therefore make up more of the applicants, causing premiums to rise so that health insurers could cover their costs to care for them. This would, in turn, eject from the market more healthy participants not willing to pay these higher premiums, further raising prices. This was a concern about the exchanges’ early design, rather than a description of their current rules or outcome. The possible cycle is often called a "Death Spiral" of adverse selection.

Sometimes there are ways to break the cycle. In the case of the used car market, services like Carfax try to restore symmetric information. Vehicle-history information can help buyers assess quality, though it does not reveal every defect or necessarily eliminate low-quality cars. In contrast, one of the goals of the ACA was to make sure that people with preexisting conditions were not pushed out of the market. A broad risk pool, including healthy people, helps spread costs. Subsidies and other transfers can also affect premiums and participation. This helps keep premiums from rising too high, making care more affordable for everyone. That's why the ACA mandate was so important to the viability of the overall system.

The mental models from the last section (tragedy of the commons, externalities, etc.) and those from this section (moral hazard, information asymmetry, etc.) are signs of market failure , where open markets without intervention can create suboptimal results, or fail . Corrective responses can include regulation, collective agreements, or changes in how markets are organized. Unfortunately, these interventions themselves can also fail, a result called government failure or political failure .

Antibiotics present a good case study in market and political failure. As we described earlier, overuse of antibiotics can reduce their efficacy as a common resource because bacteria have a chance of evolving to develop resistance each time an antibiotic is used. Overprescription of antibiotics therefore contributes to the negative externality of widespread antibiotic resistance.

Before antibiotics, bacterial infections were a leading cause of death. Scarlet fever, a complication of untreated strep infections (usually strep throat) regularly killed children in the early 1900s, and at its peak tuberculosis caused 25 percent of all deaths in Europe. Anything that could lead to an infection, such as surgery or even a small cut, could be deadly. If bacteria develop resistance to all antibiotics, large-scale bacterial outbreaks could be a reality again. That creates a reason to conserve effective antibiotics and develop new ones, rather than treating them as an unlimited resource. We need a supply of new antibiotics to defend against these bacterial "superbugs" that cannot be killed by any current antibiotics. Conserving new antibiotics can protect their future usefulness, but it can also limit sales. That's because each time they are used, the risk of bacteria developing resistance to them goes up.

Let's suppose the government addresses the market failure of overuse of these antibiotics by permitting their use only in dire circumstances. There is the clear need for these new antibiotics to be created, but at the same time they would not be able to be regularly used or sold. Assuming that their development and production continue to be left primarily to the private market, the government failure from this regulatory environment emerges: How can pharmaceutical companies get a return on their investment given current patent laws? Patent protection is time-limited, so delaying or restricting sales can reduce the returns available to developers. Also, throughout this period, some amount of the drugs still need to be continually produced in case they are needed, with batches of them continually expiring before they can be sold. (Unfortunately, it is not possible to cost-effectively sideline large-scale manufacturing capacity while waiting for peak demand.)

These uncertainties lead to a second market failure of severe underinvestment in the development of new antibiotics, leaving us collectively vulnerable to future outbreaks. The book describes major pharmaceutical companies withdrawing from antibiotic research as an example of this incentive problem.

According to a 2014 report commissioned by the U.S. Department of Health and Human Services, there is a huge disconnect between the value of a new antibiotic for society and the value to the private market. In some cases, as for bacterial ear infections, the expected value to the private market is actually negative , while the value to society is estimated to be approximately $500 billion! The public benefits of antibiotic development can exceed the revenue from selling the drugs. That creates a case for incentives such as shared development costs, while the tradeoffs of extended patent protection also need consideration. With development timelines of around ten years, waiting for an outbreak before investing is a poor strategy.

In any situation where risk and reward are separated across different entities, like this one, you want to look out for risk-related unintended consequences.

Be Careful What You Wish For

"The best laid schemes o' mice an' men [often go awry]," penned poet Robert Burns in 1785. In other words, things don't always go as planned. Consider Gosplan, the agency charged with central economic planning for the Soviet Union for most of the twentieth century. Their plans often involved setting economy-wide target amounts for commodities (wheat, tires, etc.), which broke down into production targets for specific facilities. In 1990, economist Robert Heilbroner described some of the complications with this system in "After Communism," published in The New Yorker. Production quotas can reward the quantity counted while neglecting qualities buyers need.

Goodhart's law summarizes the issue: When a measure becomes a target, it ceases to be a good measure. This more common phrasing is from Cambridge anthropologist Marilyn Strathern in her 1997 paper "'Improving Ratings': Audit in the British University System." However, the "law" is named after English economist Charles Goodhart, whose original formulation in a conference paper presented at the Reserve Bank of Australia in 1975 stated: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

Social psychologist Donald T. Campbell formulated a similar "law" (known as Campbell's law ) in his 1979 study, "Assessing the Impact of Planned Social Change." He explains the concept a bit more precisely: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."

Both describe the same basic phenomenon: When you try to incentivize behavior by setting a measurable target, people focus primarily on achieving that measure, often in ways you didn't intend. Most importantly, their focus on the measure may not correlate to the behavior you hoped to promote.

High-stakes testing culture—be it for school examinations, job interviews, or professional licensing—creates perverse incentives to "teach to the test," or worse, cheat. In the city of Atlanta in 2011, 178 educators were implicated in a widespread scandal involving correcting student answers on standardized tests, ultimately resulting in eleven convictions and sentences of up to twenty years on racketeering charges. Similarly, hospitals and colleges have been increasingly criticized for trying to achieve rankings at the expense of providing quality care and education, the very things the rankings are supposed to be measuring.

Bill Bryson’s A Short History of Nearly Everything describes perverse incentives in Gustav Heinrich Ralph von Koenigswald’s fossil collecting. Paying for individual fragments can reward breaking a specimen apart rather than preserving it.

It's like a wish-granting genie who finds loopholes in your wishes, meeting the letter of the wish but not its spirit, and rendering you worse off than when you started. In fact, there is a mental model for this more specific situation, called the cobra effect , describing when an attempted solution actually makes the problem worse. Its name comes from a commonly retold story about cobras under British rule in India. The story illustrates the mechanism, though its historical details are uncertain. When the British were governing India, they were concerned about the number of these deadly snakes, and so they started offering a monetary reward for every snake brought to them. Initially the policy worked well, and the cobra population decreased. But soon, local entrepreneurs started breeding cobras just to collect the bounties. After the government found out and ended the policy, all the cobras that were being used for breeding were released, increasing the cobra population even further.

A similar thing happened under French rule of Vietnam. In Hanoi, the local government created a bounty program for rats, paying the bounty based on a rat's tail. Enterprising ratcatchers, however, would catch and release the rats after just cutting off their tails; that way the rats could go back and reproduce. Whenever you create an incentive structure, you must heed Goodhart's law and watch out for perverse incentives, lest you be overrun by cobras and rats!

The Streisand effect applies to an even more specific situation: when you unintentionally draw more attention to something when you try to hide it. It's named for entertainer Barbra Streisand, who sued a photographer and website in 2003 for displaying an aerial photo of her mansion, which she wanted to remain private. Before the suit, the image had been downloaded a total of six times from the site; after people saw news stories about the lawsuit, the site was visited hundreds of thousands of times, and now the photo is free to license and is on display on Wikipedia and many other places. As was said of Watergate, It's not the crime, it's the cover-up.

A related model to watch out for is the hydra effect , named after the Lernaean Hydra, a beast from Greek mythology that grows two heads for each one that is cut off. When you arrest one drug dealer, they are quickly replaced by another who steps in to meet the demand. When you shut down an internet site where people share illegal movies or music, more pop up in its place. Regime change in a country can result in an even worse regime. An apt adage is Don't kick a hornet's nest, meaning don't disturb something that is going to create a lot more trouble than it is worth.

With all these traps—Goodhart's law, along with the cobra, hydra, and Streisand effects—if you are going to think about changing a system or situation, you must account for and quickly react to the clever ways people may respond. There will often be individuals who try to game the system or otherwise subvert what you're trying to do for their personal gain or amusement.

If you do engage, another trap to watch out for is the observer effect , where there is an effect on something depending on how you observe it, or even who observes it. An everyday example is using a tire pressure gauge. In order to measure the pressure, you must also let out some of the air, reducing the pressure of the tire in the process. Or, when the big boss comes to town, everyone acts on their best behavior and dresses in nicer clothes.

The observer effect is certainly something to be aware of when making actual measurements, but you should also consider how people might indirectly change their behavior as they become less anonymous. Think of how hard it is to be candid when you know the camera is rolling. Or how differently you might respond to giving a colleague performance feedback in an anonymous survey versus one with your name attached to it.

In "Chilling Effects: Online Surveillance and Wikipedia Use," Oxford researcher Jonathon Penney studied Wikipedia traffic patterns before and after the 2013 revelations by Edward Snowden about the U.S. National Security Agency's internet spying tactics, finding a 20 percent decline in terrorism-related article views involving terms like al-Qaeda , Taliban , and car bomb. One interpretation is that awareness of surveillance discouraged some reading, though the traffic pattern alone cannot identify each reader’s motive. The name for this concept is chilling effect .

In the legal context where the term chilling effect originated, it refers to when people feel discouraged, or chilled , from freely exercising their rights, for fear of lawsuits or prosecution. More generally, chilling effects are a type of observer effect where the threat of retaliation creates a change in behavior. Sometimes chilling effects are intentional, such as when someone is made an example of to send a message to others about how offenders will be treated. For instance, a company will sue another aggressively over its patents to scare off other companies that might be thinking of competing with them.

Many times, though, chilling effects are unintentional. Mandated harassment reporting can give victims pause when contemplating reaching out for help, since they might not yet be ready for that level of scrutiny. Fear of harassment also curbs usage of social media. In a June 6, 2017, Pew Research study, 13 percent of respondents said they stopped using an online service and 27 percent said they chose not to post something online after witnessing online harassment toward others.

In your personal relationships, you might find yourself walking on eggshells around a person you know has an anger management problem. Similarly, some romantic partners may not be totally honest about their relationship grievances if they perceive their partner as having one foot out the door.

Like the Wikipedia study discussed above, another unintentional chilling effect was found by an MIT study, "Government Surveillance and Internet Search Behavior," which showed that post-Snowden, people have also stopped searching for as many health-related terms on Google, even though the terms weren't directly related to illegal activity of any kind. As people understand more about corporate and government tracking, their searching of sensitive topics in general has been chilled . The authors noted: "Suppressing health information searches potentially harms the health of search engine users and, by reducing traffic on easy-to-monetize queries, also harms search engines' bottom line."

This negative unintended consequence could be considered collateral damage . In a military context, this term means injuries, damage , inflicted on unintended, collateral , targets. You can apply this model to any negative side effects that result from an action. The U.S. government maintains a No Fly List of people who are prohibited from commercial air travel within, into, or out of the U.S. There have been many cases of people with the same names as those on the list who experienced the collateral damage of being denied boarding and missing flights, including a U.S. Marine who was prevented from boarding a flight home from his military tour in Iraq.

When people are deported or jailed, even for good reason, collateral damage can be inflicted on their family members. For instance, the loss of income could take a financial toll, or children could experience the trauma of growing up without one or both parents, possibly ending up in foster care.

Sometimes collateral damage can impact the entity that inflicted the damage in the first place, which is called blowback . Blowback sometimes can occur well after the initial action. U.S. support for Afghan insurgents in the 1980s illustrates the difficulty of controlling the later use of support and weapons. The subsequent relationships among insurgent groups, al-Qaeda, and the U.S. were more complicated than the same groups simply switching sides.

Like Goodhart's law and related models, observer and chilling effects concern unintended consequences that can happen after you take a deliberate action—be it a policy, experiment, or campaign. Again, it is best to think ahead about what behaviors you are actually incentivizing by your action, how there might be perverse incentives at play, and what collateral damage or even blowback these perverse incentives might cause.

Take medical care as a modern example. Fee-for-service medicine, prevalent in the United States, pays healthcare providers based on how much treatment is provided. Quite simply, the more treatment that is provided, the more money that is made, effectively incentivizing quantity of treatment. If you have a surgery, any additional care required (follow-up surgeries, tests, physical therapy, medications, etc.) will be billed separately by the provider conducting the treatment, including any care resulting from surgical complications that might arise. Each piece of the treatment is generally individually profitable to the providers.

A bundled payment, one form of value-based care, instead pays a set amount for a defined episode, including specified follow-up care. This payment scheme therefore incentivizes quality over quantity, as the healthcare provider conducting the surgery is also on the hook for some of the additional care, sometimes even if it is administered by other providers. Such a scheme can encourage coordination and reduce unnecessary treatment. It also needs safeguards against undertreatment and avoidance of high-risk patients. This straightforward change in how medicine is billed (one lump-sum payment to one provider versus many payments to multiple providers) significantly changes the incentives for healthcare providers. The book discusses Medicare’s experiments with value-based reimbursement as attempts to reduce costs and improve outcomes. Whether a model does so depends on its design and implementation.

In other words, seemingly small changes in incentive structures can really matter. You should align the outcome you desire as closely as possible with the incentives you provide. You should expect people generally to act in their own perceived self-interest, and so you want to be sure this perceived self-interest directly supports your goals.

It's Getting Hot in Here

In the first section of this chapter, we warned about the tyranny of small decisions, where a series of isolated and seemingly good decisions can nevertheless add up to a bad outcome. There is a broader class of unintended consequences to similarly watch out for, which also involve making seemingly good short-term decisions that can still add up to a bad outcome in the long term. The mental model often used to describe this class of unintended consequences is called the boiling frog : Suppose a frog jumps into a pot of cold water. Slowly the heat is turned up and up and up, eventually boiling the frog to death. It turns out real frogs generally jump out of the hot water in this situation, but the metaphorical boiling frog persists as a useful mental model describing how a gradual change can be hard to react to, or even perceive.

The boiling frog has been used as a cautionary tale in a variety of contexts, from climate change to abusive relationships to the erosion of personal privacy. It is sometimes paired with another animal metaphor, also scientifically untrue—that of the ostrich with its head in the sand , ignoring the signs of danger. In each case the unintended consequence of not acting earlier is eventually an extremely unpleasant state that is hard to get out of—global warming, domestic violence, mass surveillance.

These unintended consequences are likely to arise when people don't plan for the long term. From finance, short-termism describes these types of situations, when you focus on short-term results, such as quarterly earnings, over long-term results, such as five-year profits. If you focus on just short-term financial results, you won't invest enough in the future. Eventually you will be left behind by competitors who are making those long-term investments, or you could be swiftly disrupted by new upstarts (which we cover in Chapter 9).

There are many examples of the deleterious effects of short-termism in everyday life. If you put off learning new skills because of the tasks in front of you, you will never expand your horizons. If you decorate your house one piece at a time in isolation, you won't end up with a cohesive décor. If there are additions to the tax code without any thought to long-term simplification, it eventually becomes a bloated mess.

The software industry has a name for the consequences of short-termism: technical debt . The idea comes from writing code: if you prioritize short-term code fixes, or "hacks," over long-term, well-designed code and processes, then you accumulate debt that will eventually have to be paid down by future code rewrites and refactors.

Accumulating technical debt isn't necessarily harmful—it can help projects move along faster in the short term—but it should be done as a conscientious observer, not as an unaware boiling frog.

If you have been involved in any small home repairs, you're probably familiar with this model. When something small is broken, many people opt for a short-term fix today, DIY-style (or even duct-tape-style), because it is cheaper and faster. However, these "fixes," which may not be up to building standards, may cost you in the long run. In particular, the item may need to be repaired again at greater cost, such as when you want to sell your home.

Startup culture has extended this concept to other forms of "debt": Management debt is the failure to put long-term management team members or processes in place. Design debt means not having a cohesive product design language or brand style guide. Diversity debt refers to neglecting to make necessary hires to ensure a diverse team. This model can likewise be extended to any area to describe the unintended consequences of short-term thinking: relationship debt, diet debt, cleaning debt.

In these scenarios, you need to keep up with your "payments" or else the debt can become overwhelming: the out-of-control messy house, the expanding waistline, or the deteriorating relationship. These outstanding debts impact your long-term flexibility. The general model for this impact comes from economics and is called path dependence , meaning that the set of decisions, or paths , available to you now is dependent on your past decisions. Sometimes an initial decision or event may seem innocuous at first, but it turns out to strongly influence or limit your possible outcomes in the long run.

As a small company, you may choose to use a piece of software for project management without giving it much thought. As you grow, though, you'll have a large group of people using this software, which may eventually turn out to be suboptimal; however, all your data is now stored there, and it would be a huge disruption to switch products.

On a personal level, many people are likely to stay near the town where they went to school once they graduate. This creates a massive long-term impact on their available career and family choices.

The same thing can happen on a larger scale. Similar types of businesses often congregate together—jewelry stores, furniture depots, car dealerships. An early concentration can attract later businesses and customers, reinforcing the location over time.

The song "I Know an Old Lady," written by Rose Bonne and Alan Mills in 1952, captures the dangers of short-termism and path dependence if left unchecked.

To escape the fate of the old lady or the boiling frog, you need to think about the long-term consequences of short-term decisions. For any decision, ask yourself: What kind of debt am I incurring by doing this? What future paths am I taking away by my actions today?

Another model from economics offers some reprieve from the limitations of path dependence: preserving optionality . The idea is to make choices that preserve future options . Maybe as a business you put some excess profits into a rainy-day fund, or as an employee you dedicate some time to learning new skills that might give you options for future employment. Or, when faced with a decision, maybe you can delay deciding at all (see thinking gray in Chapter 1) and, instead, continue to wait for more information, keeping your options open until you are more certain of a better path to embark upon.

Many college freshmen have some idea of what they want to study, but most are not ready to immediately select their major. When selecting a college, it would be a good idea for a student to choose a school that is strong in several fields of interest, not just the one they think they might pick, which preserves their options until they are really ready to decide.

As with most things, though, preserving options must be done in moderation. Even if you choose a college with many possible majors, at some point you do need to pick one in order to be able to graduate on time.

When selecting a graduate school, Lauren chose a program in operations research as a way of preserving optionality, rather than a more narrowly tailored program in biostatistics. However, not having a strong idea of what area she wanted to research for her dissertation ultimately resulted in an extra year of school.

The downside of keeping many options open is that it often requires more resources, increasing costs. Think of going to school while you also have a full-time job, maintaining multiple homes, or exploring several lines of business in one parent company. You need to find the right balance between preserving optionality and path dependence.

One model that can help you figure out how to strike this balance in certain situations is the precautionary principle : when an action could possibly create harm of an unknown magnitude, you should proceed with extreme caution before enacting the policy. It's like the medical principle of "First, do no harm." For example, if there is reason to believe a substance might cause cancer, the precautionary principle advises that it is better to control it tightly now while the scientific community figures out the degree of harm, rather than risk people getting cancer unnecessarily because the substance has not been controlled. A 2012 consolidated version of the Treaty on the Functioning of the European Union includes the precautionary principle among the bases of environmental policy.

On an individual level, the precautionary principle instructs you to take pause when an action could possibly cause you significant personal harm. That seems obvious, but people engage in risky behavior all the time (e.g., drunk or reckless driving). Beyond physical harm, the same concept applies to other kinds of harm: for example, financial harm (gambling or accepting a bad loan) and emotional harm (infidelity or going too far in an argument).

These mental models are the most useful when thinking about existential risks. After all, in the tale of the boiling frog, the frog dies. Therefore, you want first to assess what substantial harms could arise in the long term, then work backward to assess how your short-term decisions (or lack thereof) might be contributing to long-term negative scenarios (a process that we cover in more depth in Chapter 6). Review gradual changes before they become hard to reverse. Address the most serious risks while there is still time to change course.

Too Much of a Good Thing

On the side of an ancient Greek temple, home to the Oracle of Delphi, was inscribed the precept Nothing in excess . Our modern equivalent is too much of a good thing . It's natural to want more of something good, but too much of it can be bad. One slice of cookie dough cheesecake from The Cheesecake Factory is amazing; downing a whole cheesecake will probably cause you some problems, though.

The same goes for information. Complaints from people overwhelmed by too much information are not new. Roman writer Marcus Seneca said, "The abundance of books is a distraction"—in the first century A.D .! Today, researching almost anything online can make your head spin, from the mundane, such as wading through all the Amazon products and reviews for coffeemakers, to the life-changing, such as comparing colleges or choosing a new city to move to. There is just so much data and advice on almost any subject, it can easily be overwhelming.

Of course, you need some information to make good decisions, but too much information leads to information overload , which complicates a decision-making process. The excess information can overload the processing capacity of the system, be it a single person, group, or even computer, causing decision making to take too long.

There is a name for this unintended consequence: analysis paralysis , where your decision making suffers from paralysis because you are over- analyzing the large amount of information available. This is why you can spend too much time trying to make that coffeemaker decision or choosing where to go out to dinner when faced with an endless list of choices from Yelp. More seriously, people often stay in a job they don't like because they are unsure of what to do next given all the possibilities.

The model perfect is the enemy of good drives home this point—if you wait for the perfect decision, or perfect anything, really, you may be waiting a long time indeed. And by not making a choice, you are actually making a choice: you are choosing the status quo, which could be considerably worse than one of the other choices you could already have made.

There is a natural conflict between the desire to make decisions quickly and the feeling that you need to accumulate more information to be sure you are making the right choice. You can deal with this conflict by categorizing decisions as either reversible decisions or irreversible decisions . Irreversible decisions are hard if not impossible to unwind. And they tend to be really important. Think of selling your business or having a kid. This model holds that these decisions require a different decision-making process than their reversible counterparts, which should be treated much more fluidly. In a shareholder letter, Amazon’s then-CEO Jeff Bezos argued for distinguishing decisions that are hard to reverse from those that can be changed quickly.

Another way to help combat analysis paralysis is to limit choice, because the more choices you have, the harder it is to choose between them. In the early 1950s, psychologists William Hick and Ray Hyman separately conducted a number of experiments to try to quantify the mathematical relationship between the number of choices given and how long it takes to decide. They found that a greater number of choices increased the decision time logarithmically, in a formulation now known as Hick's law .

Hick's law is regularly cited as an important factor in user-experience designs, such as in the design of restaurant menus, website navigation, and forms (offline or online). For instance, on a menu, having a vegetarian section allows vegetarians to efficiently narrow the sections of the menu they should read through. Being able to determine quickly whether there are enough vegetarian options on a menu might be a big factor in whether a family with a vegetarian would choose to eat at your restaurant.

In your own life, you can use Hick's law to remember that decision time is going to increase with the number of choices, and so if you want people to make quick decisions, reduce the number of choices. One way to do this is to give yourself or others a multi-step decision with fewer choices at each step, such as asking what type of restaurant to go to (Italian, Mexican, etc.), and then offering another set of choices within the chosen category.

In addition to increased decision-making time, there is evidence that a wealth of options can create anxiety in certain contexts. This anxiety is known as the paradox of choice , named after a 2004 book of the same name by American psychologist Barry Schwartz. Schwartz explains that an overabundance of choice, the fear of making a suboptimal decision, and the potential for lingering regret following missed opportunities can leave people unhappy.

In the context of seeking romantic relationships, people are often reminded that there are "plenty of fish in the sea." With so many fish, this can leave you to question how you will know when you have found "the one." Similarly, you might be left questioning whether a past partner was "the one that got away." This anxiety also arises with smaller-scale decisions, such as when you have young kids and you find yourself finally with an opportunity to go out for the night: Do you go out with friends or with just your partner? Do you go to a nice restaurant or the movies? If the movies, which one? The more choices, the more chance you have for regret later.

While we, the authors, are reasonably happy people, we have experienced the anxiety surrounding the paradox of choice with our own life choices. We were lucky to have sold a startup company at a young age, leaving us with essentially unlimited career options. At the time of the sale, Lauren had just accepted a position at GlaxoSmithKline and was content with continuing down that path. However, over time she wondered whether this was the right path and found herself constantly reading job postings. She also spent a lot of time thinking about going back to school in a different field, fulfilling a different childhood dream, such as becoming an architect or designing prosthetics. Gabriel was left with an entirely open-ended future and took some time off. But soon he started asking, What next? Should I start another for-profit company? Should Lauren and I start a nonprofit together? Write a book? The choices were and are endless. Don't get us wrong—we aren't complaining. We are just acknowledging that we personally sympathize with this model.

Hick's law and the paradox of choice explain downsides of having many choices. There is also a model that explains the downside of making many decisions in a limited period: decision fatigue . Repeated decisions can be tiring, especially when they require sustained attention. A break may help, though it does not automatically reset judgment or guarantee better decisions.

The 2011 study "Extraneous Factors in Judicial Decisions" describes the impact of decision fatigue on parole boards deciding whether to grant freedom to prisoners: "We find that the percentage of favorable rulings drops gradually from [about] 65% to nearly zero within each decision session and returns abruptly to [about] 65% after a break. Our findings suggest that judicial rulings can be swayed by extraneous variables that should have no bearing on legal decisions." The study was observational: the pattern does not by itself establish that fatigue or a break caused the difference, since case order and other factors may matter.

Some productive people, including Steve Jobs and Barack Obama, have tried to combat decision fatigue by reducing the number of everyday decisions, such as what to eat or wear, so that they can reserve their decision-making faculties for more important decisions. Barack Obama chose to wear only blue or gray suits and said of this choice, "I'm trying to pare down decisions. I don't want to make decisions about what I'm eating or wearing. Because I have too many other decisions to make." Gabriel also tends to do this to some extent, usually wearing one of seven identical pairs of dark gray jeans and often eating the same lunch for weeks on end. He swears that it really does make things easier and saves time!

If you want more variety in your life, one suggestion is to front-load the decisions on your outfits and meals for the week to Sunday. Making these decisions on a usually lower-stress day can free up your decision-making capacity for the workweek. Meal planning and even some meal prep on the weekend can help keep you from making unhealthy choices when you are overwhelmed later in the week.

In this chapter we've covered an array of unintended consequences, from market failure to perverse incentives, from too much focus on the short term to too much of a good thing. Most generally, consider heeding Murphy's law : Anything that can go wrong, will go wrong . It's named after aerospace engineer Edward Murphy, from his remarks after his measurement devices failed to perform as expected. It was intended as a defensive suggestion, to remind you to be prepared and to have a plan for when things go wrong.

It is unfortunately impossible to account for all possible unintended consequences. However, the mental models in this chapter can help you identify and avoid negative unintended consequences in a large array of situations. When planning a change, ask how people may adapt, which costs fall on others, and what could accumulate over time. Then monitor the results so that an unexpected response can lead to a correction.