The Selective Skeptic Test
A thought experiment that asks you to imagine whether you would find evidence equally credible if it supported the other side, helping to reveal double standards in skepticism due to motivated reasoning.
THE IDEA
The Selective Skeptic Test
During my research for this book, I came across a paper claiming to show that soldier mindset makes people successful in life. "Oh, come on," I scoffed to myself, and checked its methodology section for flaws. Sure enough, it turned out to be a poorly designed study.
Then, somewhat grudgingly, I did a thought experiment: What if this study had claimed that soldier mindset makes people unsuccessful in life?
In that case, I realized, my reaction would have been: "Exactly as I suspected. I’ll have to find a place for this study in my book!" That contrast between my reactions in the real and hypothetical worlds was a wake-up call for me, a warning that I needed to be a little less credulous of evidence that happened to support my side. It prompted me to go back through the studies I had been planning to cite in my favor, and scrutinize their methodology for flaws, just as I had done with the pro-soldier mindset study. (Sadly, this ended up disqualifying most of them.)
I call this type of thought experiment the selective skeptic test: Imagine this evidence supported the other side. How credible would you find it then?
Suppose someone criticizes a decision your company made, and your knee-jerk reaction is, "They don’t know what they’re talking about, because they don’t have all the relevant details." Selective skeptic test: Imagine the person had praised your company’s decision instead. Would you still think that only insiders are informed enough to have valid opinions?
Suppose you’re a feminist and you read an article complaining about how feminists hate men. As evidence, the author offers a handful of tweets from people you’ve never heard of, which say something like, "All men need to die in a fire!!! #girlpower #feminism." You think to yourself: "Give me a break. Of course you can find examples of people being idiots or extremists, in any group, if you look hard enough for them. Cherry-picking like that doesn’t prove anything about feminism."
Selective skeptic test: Imagine the article had been full of cherry-picked quotes from a group you dislike, such as conservatives. How would you react? Would you reject the evidence on the same logic, that a few cherry-picked examples of people in a group being jerks don’t prove anything about that group?
When to use it
Evaluating the credibility of studies that align with your political or ideological leanings.
Assessing feedback from colleagues or clients to avoid defensive dismissal.
In debates, checking if your objections to an opponent's evidence would be equally strong if the roles were reversed.
Reviewing news articles and social media posts for biased skepticism based on who is making the claim.
FROM THEORY TO EVERYDAY LIFE
In practice
Soldier Mindset Study
Julia Galef finds a study claiming that soldier mindset makes people successful. Initially skeptical, she checks for flaws. After imagining that the study claimed the opposite, she realizes she would have accepted it uncritically. She then reexamines the studies supporting her own argument.
Company Decision Criticism
When someone criticizes a company decision, the natural reaction is to dismiss them as lacking insider knowledge. The selective skeptic test asks: if they had praised the decision, would you still question their credentials?
Feminism and Cherry-Picked Tweets
An article uses a few extreme tweets to argue that feminists hate men. The reader dismisses this as cherry-picking and unrepresentative. The test asks: if the examples were from a group you dislike (e.g., conservatives), would you apply the same logic?
A MOMENT TO REFLECT
Put it to work.
Think of a time you dismissed evidence against your own view. Would you have dismissed it if it supported your view?
When someone criticizes your work, ask yourself: 'If this were praise, would I accept the same reasoning?'
How can you train yourself to apply the same scrutiny to evidence that flatters your beliefs as you do to evidence that threatens them?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- The Scout Mindset (Julia Galef)