Conditions for Expertise
Expert judgment depends partly on opportunities to learn from prompt, clear feedback. The passage contrasts forecasting tasks, reports overconfidence in estimates of physical constants and explains how forgiving decisions can conceal a need to improve.
THE IDEA
Conditions for Expertise
Like the very wealthy (Mills, 1959), the very well trained are hard to study. Investigators face problems of access, as well as the challenge of learning enough about experts’ specialties to understand how well they are doing. Occasionally, deliberate experiments provide useful evidence. A well-known judgment example is the training of weather forecasters (Murphy & Winkler, 1977, 1984). That experience confirms the experimental results: performance improves with good conditions for learning (i.e., prompt, unambiguous feedback, with incentives for performance). It can become excellent for short-term precipitation forecasts (clear, discrete, daily events), but remains poorer for tornadoes (rare, not always observed) and wind velocities (variable, continuous events).
In addition to deliberate experiments, there are natural ones, exposing experts to conditions that should affect heuristics use. Naturally, these raise even greater interpretative challenges. For example, Henrion and Fischhoff (1986; Chapter 37, this volume) found great overconfidence in judgments akin to confidence intervals, assigned by physicists to estimates of fundamental values (e.g., speed of light, Planck’s constant, rest mass of the electron). Shlyakhter et al. (1994) extended this result to other expert estimates (e.g., for population, energy consumption), also finding consistent overconfidence. Many of these estimates arose from group processes, such as the Particle Data Group or the Energy Modeling Forum.
Improving Lay Judgment. Nobody is perfect. Sometimes, no one needs to be. Some situations are forgiving of imperfect judgments. That can be a mixed blessing. Forgiving situations lack the sharp feedback often needed to improve judgment (or even to convince people that improvement is needed). As a result, individuals living in a forgiving world may be particularly vulnerable when they face novel and demanding situations, where they need to “get it right the first time.” Thus, diminishing the local importance of judgmental biases increases the global plausibility of their existence.
One prominent analysis of the sensitivity of decisions to inaccuracy in judgment led to von Winterfeldt and Edwards’s (1986) flat maximum principle. They described a broad class of choices with continuous options (e.g., invest $*X$, respond in Y msec, travel at Z mph), whose expected utility varies little with estimates of any input parameter, as long as it is around the correct value. However, Lichtenstein et al. (1982) describe a class of choices with discrete options (e.g., operate/don’t, sell/hold), where sensitivity can be much greater. Henrion (1982; also Morgan & Henrion, 1990) characterized decisions by their sensitivity to the appropriateness of confidence judgments (or calibration). Dawes and Corrigan (1974) and others (e.g., Camerer, 1981; Goldberg, 1968) identified conditions under which choices are insensitive to the weights assigned to different considerations. If specific decisions can be characterized in these general terms, then one has a prior probability for the need to improve judgments.
When to use it
A meteorologist uses daily precipitation forecast feedback to improve short-term predictions, but must be cautious with rare events like tornadoes.
A physicist estimating a fundamental constant checks how well earlier uncertainty intervals held up against subsequent measurements.
An investor evaluating a continuous option (e.g., amount to invest) can apply the flat maximum principle to see if precise estimates are necessary.
A manager considering whether to launch tests whether uncertainty in key estimates could change that choice.
FROM THEORY TO EVERYDAY LIFE
In practice
Weather Forecasters
Meteorologists receive prompt, unambiguous feedback on their daily precipitation forecasts. This leads to excellent calibration for short-term, discrete events, but poorer accuracy for rare tornadoes or continuous wind velocities.
Physicists' Overconfidence
Henrion and Fischhoff (1986) found overconfidence in uncertainty estimates for physical constants such as the speed of light. The passage discusses this alongside the broader question of how experts receive feedback and learn; it does not isolate a single cause of their overconfidence.
A MOMENT TO REFLECT
Put it to work.
Think of a domain where you make judgments. How often do you receive prompt, unambiguous feedback? How might that affect your accuracy?
Consider a situation where imperfect judgments are 'forgiving.' How could that mask your need for improvement until a novel challenge arises?
How can the flat maximum principle help you decide when it is worth refining your estimates versus when approximate values are sufficient?
References
- The Sociological Imagination — C. Wright Mills (1959)
- Reliability of subjective probability forecasts of precipitation and temperature — Murphy, A.H. & Winkler, R.L. (1977)
- Probability forecasting in meteorology — Murphy, A.H. & Winkler, R.L. (1984)
- The value of assessing uncertainties — Henrion, M. & Fischhoff, B. (1986)
- Probabilistic estimates of rates of change — Shlyakhter, A. et al. (1994)
- Decision Analysis and Behavioral Research — von Winterfeldt, D. & Edwards, W. (1986)
- Calibration of probabilities: The state of the art — Lichtenstein, S. et al. (1982)
- The value of knowing how little you know — Henrion, M. (1982)
- Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis — Morgan, M.G. & Henrion, M. (1990)
- Linear models in decision making — Dawes, R.M. & Corrigan, B. (1974)
- General conditions for the success of bootstrapping models — Camerer, C. (1981)
- Simple models or simple processes? Some research on clinical judgments — Goldberg, L.R. (1968)
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)