The Illusion of Understanding
The illusion of understanding is a cognitive bias where individuals overestimate their ability to explain past events and predict future outcomes, often due to the narrative fallacy and hindsight bias.
THE IDEA
The Illusion of Understanding
This cognitive bias refers to the tendency to create coherent narratives after the fact, making the past seem more predictable than it actually was. People often believe they understand the causes of events better than they do, leading to overconfidence in their judgments and decisions. Key related concepts include the narrative fallacy (Taleb, The Black Swan), hindsight bias, the halo effect in business (Rosenzweig, The Halo Effect), and regression to the mean. The footnotes below provide references to key studies and examples:
[1] narrative fallacy: Nassim Nicholas Taleb, The Black Swan: The Impact of the Highly Improbable (New York: Random House, 2007). [2] one attribute that is particularly significant: See chapter 7. [3] throwing the ball: Michael Lewis, Moneyball: The Art of Winning an Unfair Game (New York: Norton, 2003). [4] sell their company: Seth Weintraub, "Excite Passed Up Buying Google for $750,000 in 1999," Fortune, September 29, 2011. [5] ever felt differently: Richard E. Nisbett and Timothy D. Wilson, "Telling More Than We Can Know: Verbal Reports on Mental Processes," Psychological Review 84 (1977): 231–59. [6] United States and the Soviet Union: Baruch Fischhoff and Ruth Beyth, "I Knew It Would Happen: Remembered Probabilities of Once Future Things," Organizational Behavior and Human Performance 13 (1975): 1–16. [7] quality of a decision: Jonathan Baron and John C. Hershey, "Outcome Bias in Decision Evaluation," Journal of Personality and Social Psychology 54 (1988): 569–79. [8] should have hired the monitor: Kim A. Kamin and Jeffrey Rachlinski, "Ex Post ≠ Ex Ante: Determining Liability in Hindsight," Law and Human Behavior 19 (1995): 89–104. Jeffrey J. Rachlinski, "A Positive Psychological Theory of Judging in Hindsight," University of Chicago Law Review 65 (1998): 571–625. [9] tidbit of intelligence: Jeffrey Goldberg, "Letter from Washington: Woodward vs. Tenet," New Yorker, May 21, 2007, 35–38. Also Tim Weiner, Legacy of Ashes: The History of the CIA (New York: Doubleday, 2007); "Espionage: Inventing the Dots," Economist, November 3, 2007, 100. [10] reluctance to take risks: Philip E. Tetlock, "Accountability: The Neglected Social Context of Judgment and Choice," Research in Organizational Behavior 7 (1985): 297–332. [11] before their current appointment: Marianne Bertrand and Antoinette Schoar, "Managing with Style: The Effect of Managers on Firm Policies," Quarterly Journal of Economics 118 (2003): 1169–1208. Nick Bloom and John Van Reenen, "Measuring and Explaining Management Practices Across Firms and Countries," Quarterly Journal of Economics 122 (2007): 1351–1408. [12] "How often will you find …": I am indebted to Professor James H. Steiger of Vanderbilt University, who developed an algorithm that answers this question, under plausible assumptions. Steiger’s analysis shows that correlations of .20 and .40 are associated, respectively, with inversion rates of 43% and 37%. [13] his penetrating book: The Halo Effect was praised as one of the best business books of the year by both the Financial Times and The Wall Street Journal: Phil Rosenzweig, The Halo Effect: … and the Eight Other Business Delusions That Deceive Managers (New York: Simon & Schuster, 2007). See also Paul Olk and Phil Rosenzweig, "The Halo Effect and the Challenge of Management Inquiry: A Dialog Between Phil Rosenzweig and Paul Olk," Journal of Management Inquiry 19 (2010): 48–54. [14] "a visionary company": James C. Collins and Jerry I. Porras, Built to Last: Successful Habits of Visionary Companies (New York: Harper, 2002). [15] flip of a coin: In fact, even if you were the CEO yourself, your forecasts would not be impressively reliable; the extensive research on insider trading shows that executives do beat the market when they trade their own stock, but the margin of their outperformance is barely enough to cover the costs of trading. See H. Nejat Seyhun, "The Information Content of Aggregate Insider Trading," Journal of Business 61 (1988): 1–24; Josef Lakonishok and Inmoo Lee, "Are Insider Trades Informative?" Review of Financial Studies 14 (2001): 79–111; Zahid Iqbal and Shekar Shetty, "An Investigation of Causality Between Insider Transactions and Stock Returns," Quarterly Review of Economics and Finance 42 (2002): 41–57. [16] In Search of Excellence: Rosenzweig, The Halo Effect. [17] "Most Admired Companies": Deniz Anginer, Kenneth L. Fisher, and Meir Statman, "Stocks of Admired Companies and Despised Ones," working paper, 2007. [18] regression to the mean: Jason Zweig observes that the lack of appreciation for regression has detrimental implications for the recruitment of CEOs. ... (personal communication, December 29, 2009).
When to use it
Evaluating business strategies: Be cautious when attributing success or failure solely to decisions made, as external factors and luck play a large role.
Historical analysis: Recognize that historians' narratives often simplify complex events and ignore counterfactuals.
Personal reflection: Use this bias to combat overconfidence in your own explanations of past events and predictions.
Investment decisions: Avoid being overly influenced by recent performance stories, as they may be driven by the halo effect and regression.
FROM THEORY TO EVERYDAY LIFE
In practice
Excite and Google
In 1999, Excite had the opportunity to buy Google for $750,000 but passed it up, later regretting the decision. The illusion of understanding can lead to overconfidence in business decisions, with outcomes appearing obvious only in hindsight.
Moneyball
The success of the Oakland Athletics in baseball, as described in Michael Lewis's Moneyball, demonstrates how conventional wisdom often overlooks valuable statistical insights, and how narratives can be constructed after the fact to explain success or failure.
A MOMENT TO REFLECT
Put it to work.
Think of a past decision that turned out well. How might you be overestimating your understanding of why it succeeded?
Consider a time when you were surprised by an outcome. How did you later construct a narrative to explain it, and could that narrative be misleading?
In what areas of your life do you feel most confident in your predictions? How could the illusion of understanding be affecting that confidence?
References
- The Black Swan: The Impact of the Highly Improbable — Nassim Nicholas Taleb (2007)
- Moneyball: The Art of Winning an Unfair Game — Michael Lewis (2003)
- Telling More Than We Can Know: Verbal Reports on Mental Processes — Richard E. Nisbett and Timothy D. Wilson (1977)
- I Knew It Would Happen: Remembered Probabilities of Once Future Things — Baruch Fischhoff and Ruth Beyth (1975)
- Outcome Bias in Decision Evaluation — Jonathan Baron and John C. Hershey (1988)
- Ex Post ≠ Ex Ante: Determining Liability in Hindsight — Kim A. Kamin and Jeffrey Rachlinski (1995)
- A Positive Psychological Theory of Judging in Hindsight — Jeffrey J. Rachlinski (1998)
- The Halo Effect: … and the Eight Other Business Delusions That Deceive Managers — Phil Rosenzweig (2007)
- Built to Last: Successful Habits of Visionary Companies — James C. Collins and Jerry I. Porras (2002)
- Managing with Style: The Effect of Managers on Firm Policies — Marianne Bertrand and Antoinette Schoar (2003)
- Measuring and Explaining Management Practices Across Firms and Countries — Nick Bloom and John Van Reenen (2007)
CONTINUE THE CONVERSATION
Source texts
- Thinking, Fast and Slow (Daniel Kahneman)