Good Summary Sources
A heuristic for evaluating summary sources by considering their position to know, incentives to be careful with the truth, and whether they are genuinely engaged in providing a summary rather than using it as a setup for a less trustworthy claim.
THE IDEA
What Makes a Good Summary Source?
When reading the room, it helps to start by reading high-quality summary sources. What do those look like?
First, look for a source in a position to know. People in such a position are likely to know more than the average person due to their expertise (e.g., academic) or their job (e.g., reporter). Keep in mind that specificity matters—a physicist summarizing health info is not necessarily helpful; neither is a paranormal practitioner making claims about history (more on that later).
Second, check that these sources have incentives to be careful with the truth, or a good track record of doing so. We trust the meteorologist to try to get the weather right because she pays a price when wrong about the weather. We trust her less on health policy, in part, because the professional consequences for messing up are lower.
Third, and this is a point that is particularly applicable to summaries, consider whether sources are fully engaged in providing a summary of research. Beware of quick summaries that are just being used to set up another, less trustworthy claim.
When to use it
Evaluating a news article that summarizes a scientific study: check if the reporter has relevant expertise or access to experts.
Choosing which expert to cite for a research paper: prioritize those with domain-specific knowledge and a track record of accuracy.
When reading a blog post that summarizes political polling data, look for evidence of the author’s position to know and incentives to be careful (e.g., fact-checking reputation).
Assessing a textbook summary: ensure the author is an established scholar in the field and not just using the summary to push a personal agenda.
FROM THEORY TO EVERYDAY LIFE
In practice
Physician vs. Physicist
A physicist summarizing health information is not necessarily a good summary source because their expertise is in physics, not medicine. Specificity matters.
Meteorologist's Incentives
We trust a meteorologist to try to get the weather right because she pays a price (professional reputation, accuracy) when wrong. That incentive to be careful with the truth does not necessarily extend to health policy, where the consequences for error are lower.
A MOMENT TO REFLECT
Put it to work.
Think of a recent news article you read. Did the author have a strong position to know about the topic? Why or why not?
How can you assess whether a source is genuinely engaged in providing a summary versus using it as a setup for a less trustworthy claim?
Identify a summary source you trust. What specific incentives or track records make you trust it? What are its potential blind spots?
CONTINUE THE CONVERSATION
Source texts
- Verified How to Think Straight, Get Duped Less, and Make Better Decisions about What to Believe Online (Mike Caulfield, Sam Wineburg)