Why Compellingness and Surprise Beat the Checklist

Why Compellingness and Surprise Beat the Checklist

Checklist approaches to web credibility can fail when they rely on cheap signals: the dot-org extension, the well-formatted page, the official-sounding “About” page. Part of their failure was that they didn’t consider known markers of quality—expensive signals of quality such as professional recognition or a hard-won reputation for accuracy. Another problem with these checklists is how they can dull your sense of surprise.

Let’s start with a fundamental truth about checklists. When situations are standardized and routine, they work wonderfully. More than wonderfully. When you get on a plane and are about to take off, you don’t want the pilot and copilot to SIFT their way to figuring out whether everything is prepared for the flight. You want them to go through the checklist, the items for that specific plane, and do the sequence of actions they do every time to make sure you end up climbing to thirty thousand feet and not, alternatively, floating in the bay.

It would be nice to imagine a world in which determining whether something was trustworthy was like preparing a plane for takeoff. Flip the right switches, in the right order, and then look at the result: “This source is 78.4 percent reputable! This claim is sufficiently correct!” Sounds ridiculous, right? But that’s what checklist approaches try to do. Early information literacy checklists, like the one mentioned in chapter 2, even encouraged people to assign points for different things they found and add them up.4 “Seven out of ten—checks out!”

Context-seeking doesn’t work like that for a very simple reason—what you care about when judging a claim or source is going to vary from claim to claim, from source to source, from situation to situation. You can’t use a checklist because you’re never flying the same plane twice. That’s because, in any given instance, what’s important will change.

Take this headline and image about “mutant daisies” near a Japanese nuclear plant that had a partial meltdown. You see this and find it compelling. It seems important. It triggers emotion. Perhaps it makes you think the soil these flowers grew in is radioactive. Or perhaps the photo just produces general anger at the corporations behind such power plants.

Figure 10.1 Story about the “Fukushima daisies” photo (https://www.indiatoday.in/world/story/daisies-fukushima-nuclear-plant-japan-photo-viral-twitter-284282-2015-07-23)

What sort of checklist could help you check this? Let’s imagine you investigate this, and you discover the following facts about these bizarrely deformed flowers found near—let us remind you—a nuclear power plant.

1. The flowers were indeed found near the Fukushima power plant.
2. The photo is unaltered.
3. The person who took the photo is an ordinary resident of the area who posted the picture on Twitter.
4. Flower experts say the process resulting in this unusual appearance is called “fasciation,” a naturally occurring process unrelated to radiation.
5. While the flowers look wild, they were growing in a garden near a city street.

If you initially thought the photo might be solid evidence of unsafe conditions near the plant, we’re going to guess that point number four jumped out at you. You’re looking at a “naturally occurring process unrelated to radiation.” And we’re not expecting jaw-dropping daytime-soap-opera levels of shock on reading that; we’re guessing most of you felt a bit of a “wait, what?” sensation on reading that line. But notice that the photo has a lot of things going for it. It’s a real photo. It’s not photoshopped. It’s from the area it says it is. The person who took it is a local. From a checklist perspective, it seems to be swishing baskets from the three-point line. And yet our experience of this is, quite rightly, that it’s most likely junk.

Why is that?

If you found the photo compelling at first, we imagine you already believed it was real and that it was taken near a nuclear power plant. These factors would have influenced your initial reaction. You may not have guessed that the flowers were growing next to a city street—new information for sure—but that’s unlikely to be what jumped out at you.

What you noticed was something (a) you didn’t know, and (b) related to the compelling aspect of this photo as evidence. Not to use overly technical language here, but the flowers were interesting because they seemed freaky. They were weird. They demanded an explanation. And since they seemed unexplainable by normal means, the idea that their weirdness was caused by a nuclear accident seemed, well, plausible. However, once you know this process happens naturally—that its freakiness is actually normal—your reaction changed.

If you are a lover of checklists, you might think the solution is to add the flowers’ compelling feature, “freakiness,” to checklists. But that is a dead end. Here, the compelling bit was the freakiness. In the “suitcases of ballots” misinformation, it was the “secretiveness.” For the American College of Pediatricians, the compelling aspect was, weirdly, that you thought it was a bigger and blander organization than it turned out to be. It works the opposite way as well—you may find something not particularly useful because you assume it was written by someone without knowledge, a “dude with a Twitter account,” but a short investigation reveals that person to be a well-respected expert. An article you dismiss, assuming it is a non-peer-reviewed preprint, turns out to be published in a major journal.

Checklists can get at some of that. Some qualities of good sources can be defined and listed. But one of the downsides of checklists is they obscure the wonderfully quick processing a sense of surprise can bring. The problem with a checklist is that what you care about varies broadly depending on what you thought something was. No checklist captures this. And when you’re going through a checklist methodically, it’s easy to lose sense of what drew you to the source or claim in the first place. You should absolutely learn about what makes a good source good or a trustworthy claim trustworthy—in ways that help you develop better initial reactions. But in the moment, anything that doesn’t take full advantage of the context of you is a liability. Paying attention to your context will always be more powerful than any checklist.