Anecdotal Evidence
Expert judgment, industry benchmarks, and individual stories offer different kinds of evidence. Combining well-informed forecasts can help, but agreement alone does not establish accuracy. A single anecdote can suggest a question without showing how common or causal a pattern is.
THE IDEA
Anecdotal Evidence
Expert judgment draws on experience that may be difficult to express as formal rules. A forecast can be evaluated against outcomes over time, but relying on one expert leaves no way to compare it with other judgments on the same question. One common way to combine forecasts is to average them.5
A linear opinion pool uses a weighted or unweighted average of individual estimates. Equal weights provide a simple starting point when advisors have comparable information and track records. Different weights need justification, ideally from relevant performance rather than seniority or confidence alone. Adding experts does not automatically improve accuracy, particularly if they share the same sources or errors. The account cites applications in medicine, economics, weather forecasting, and psychology.6
Behavioral aggregation lets experts exchange information and revise their judgments. This may happen in a meeting or through repeated questionnaires, as in a Delphi process. The goal may be to clarify disagreements as well as to seek agreement. Experts need not converge, and a consensus is not proof that the resulting forecast is correct.
Examine the spread of opinions alongside their average. Wide disagreement may expose different assumptions or information. Close agreement may reflect shared expertise, but it may also reflect a common blind spot. The variance of the estimates describes disagreement; it does not establish their accuracy.
Industry benchmarks provide reference points for comparing performance or costs. Some come from measured data and others from surveys or professional conventions. Before applying one, check how it was constructed and whether the organizations, outcomes, and operating conditions are comparable.
Individual cases, testimonials, and hearsay require a different assessment. A story can be true while being unrepresentative, and an apparent success does not establish what caused it. Such evidence can help identify questions or possible mechanisms. It should carry little weight as a general conclusion without corroboration.
When to use it
Combining forecasts from multiple economists to predict GDP growth.
Using averaged patient recovery rates from several clinics to set treatment benchmarks.
Avoiding reliance on a single customer testimonial when deciding on a product feature.
Use repeated Delphi questionnaires to identify assumptions and disagreements among project managers assessing a risk.
FROM THEORY TO EVERYDAY LIFE
In practice
Diet success story
A buyer credits a product with success because it worked for a friend. The story does not show how often it works, what else changed, or how results compare with an alternative.
Expert forecast averaging
A company averages sales forecasts from five senior managers. It then compares the combined forecast with individual forecasts and actual sales to see whether pooling improves accuracy.
A MOMENT TO REFLECT
Put it to work.
When evaluating a claim, how can you distinguish between reliable aggregated evidence and a single anecdote?
What are the advantages of averaging multiple expert opinions over relying on the most vocal expert?
In what situations might industry benchmarks be misleading if not adjusted for context?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- Evidence-Based Decision-Making How to Leverage Available Data and Avoid Cognitive Biases (Andrew D. Banasiewicz)