Asking Probing Questions to Counter Confirmation Bias

2. Asking questions that could change your mind

We are all pretty good at digging up disconfirming information to respond to a sales pitch. When a time-share salesman raves about a “once in a lifetime” deal, our shields go up and we become implacably logical, picking apart his exaggerated claims. (“Er, if your resort is so popular that I risk losing a unit if I don’t buy it right now, then why do you seem so desperate? And why did you have to bribe me to be here?”)

The problem comes, of course, when we sort of want to be sold. At dinner, the waiter approaches with a dessert tray featuring a chocolate lava cake large enough to have its own ZIP code, and, as your mouth starts to salivate, you ask hopefully, “Is it good?” Not exactly a tough stand.

Sometimes we think we’re gathering information when we’re actually fishing for support. Take the tradition of calling people’s references when you want to hire them. It’s an exercise in self-justification: We believe someone is worth hiring, and as a final “check” on ourselves, we decide to gather more information about them from past colleagues. So far, so good. Then we allow the candidate to tell us whom we should call, and we dutifully interview those people, who say glowing things about the candidate, and then, absurdly, we feel more confident in our decision to hire the person. (Imagine if we bought a time-share because the salesman had three awesome references.)

In some organizations, hiring managers have become smarter about reference calls. Some ask the references for additional people to contact who weren’t on the original list. Those secondary interviews will tend to yield more neutral information. Other people have reconsidered the kinds of questions they ask in reference calls. Rather than ask for an evaluation of the candidate (“Would you say Steve’s performance was closer to ‘stunning’ or ‘breathtaking’? Be honest.”), many firms now seek specific factual information. For example, Ray Rothrock, a venture capitalist with Venrock, says that one of the best diagnostic questions he’s discovered in assessing entrepreneurs is “How many secretaries has this entrepreneur had in the past few years?” An answer such as five gives you a reason to ask what happened; the count alone does not explain the departures.

This same strategy of fishing for specific information was endorsed in an article called “On Being a Happy, Healthy, and Ethical Member of an Unhappy, Unhealthy, and Unethical Profession,” published in 1999 by U.S. District Court judge Patrick J. Schiltz. In the piece, Schiltz urges law students to ask tough, disconfirming questions before taking a job with a big corporate law firm:

Every big firm claims that it is different. Every big firm denies that it is a sweatshop. Every big firm insists that, although its attorneys work hard, they lead balanced lives. This is almost always false. It has to be. There is no free lunch.…

Ask tough questions of the lawyers you meet. When you are at a recruiting dinner with a couple of lawyers from the firm, don’t just ask them, “So, do you folks have any kind of life outside of work?” They will chuckle, say “sure,” and ask if you want more wine. Instead, ask them how many times last week they had dinner with their families. And then ask them what time dinner was served. And then ask them whether they worked after dinner.

Ask them what their favorite television show is or what is the last good movie they saw. If they respond, respectively, Welcome Back, Kotter and Saturday Night Fever, you will know something’s wrong.… When a lawyer tells you that he gets a lot of interesting assignments, ask for examples. You may be surprised at what passes for “interesting” at the firm. And when a lawyer tells you that associates are happy at the firm, ask for specifics. How many associates were hired five years ago? How many of those associates remain at the firm? Who were the last three associates to leave the firm? What are they doing now? How can you contact them?

A study of questions that seek potentially unfavorable information illustrates the difference. It was titled “There Is Such a Thing as a Stupid Question,” authored by three Wharton researchers, Julie A. Minson, Nicole E. Ruedy, and Maurice E. Schweitzer.

In the study, participants acted as the seller in a role-played negotiation over an iPod. As sellers, they knew everything about the iPod: It was relatively new, had a spiffy cover, and was filled with an impressive collection of songs. On the other hand, it had frozen up twice in the past, forcing a reset that resulted in all the music being deleted.

The researchers wondered what it would take for the sellers to disclose the freezing problem. The buyers in the negotiation, who were cronies of the researchers, tried three different strategies. When the buyers asked about the iPod, “What can you tell me about it?,” only 8% of the sellers disclosed the problem. The question “It doesn’t have any problems, does it?” boosted the disclosure to 61%.

The question that produced the most disclosure in this study was this one: “What problems does it have?” That prompted 89% of the sellers to come clean.

The researchers propose that probing questions signal confidence and experience, making the seller less confident of concealing a problem. There’s a similar signaling effect with Judge Schiltz’s questions. A law student is likely to get straight answers to the questions “How many associates were hired five years ago?” and “How many of those associates remain at the firm?”

THIS PRACTICE OF ASKING probing questions is useful when you are trying to pry information from people who have an incentive to spin you: salesmen, recruiters, employees with agendas, and so on. On the other hand, it can backfire in situations where there’s a clear power dynamic, as between a doctor and a patient. Here’s why: Tough questions work in the iPod situation because they signal confidence and experience in the asker, but when the asker is already the clear “expert,” as the doctor is in the doctor/patient situation, then aggressive questions can reinforce that imbalance. That can cause patients to clam up or to follow the doctor’s lead too eagerly, even if it isn’t the most productive direction.

So, for doctors to gather trustworthy information, they’ve got to be diligent about asking open-ended questions—much more like the generic iPod question, “What can you tell me about it?” That kind of question produced little disclosure in the iPod study, but it gives patients room to describe their experience in their own terms.

The late Dr. Allen Barbour, the chief of the Stanford Diagnostic Clinic at the Stanford University School of Medicine, was a master of open-ended interviewing. In his book Caring for Patients, he recalls seeing a patient named Joseph H., whose ailment had puzzled other doctors for months.

Joseph, 67, had first come to his regular doctor (not Dr. Barbour) reporting feeling “lightheaded, dizzy.” His doctor knew that many diseases might be producing these symptoms, so to diagnose Joseph accurately, the doctor and his colleagues ordered a laundry list of tests: “EKG, EEG, aortic arch and cerebral arteriography, consultations in neurology and ENT with electronystagmography, audiometry, and other special tests.” All the tests came back negative.

So the doctor tried a portfolio of pharmaceuticals: Hydergine, vasodilators, the “anti-vertigo” antihistamines, and anticoagulants. Nothing worked. Joseph’s ailment was a mystery.

Frustrated, the doctor referred Joseph to Dr. Barbour. In their first meeting, Dr. Barbour asked Joseph to describe in depth what he meant by “feeling dizzy” and how often he felt that way. Joseph responded:

“Doctor, I feel dizzy nearly all the time since my wife died. I don’t know what to do with myself. I’m confused. I watch TV, but I’m not interested. I go outside, but there’s no place to go.”

He looked sad indeed as he told of the emptiness of his life. He had moved to California with his wife after retirement. He had no children, no close friends, no special interests.

Dr. Barbour interpreted “Dizzy” as Joseph’s way of expressing confusion and grief. In this account, asking what the word meant revealed an experience the earlier questioning had missed.

According to Barbour’s account, the earlier doctors had not asked Joseph what he meant by “feeling dizzy.” The story illustrates a missed question, rather than a general rule for deciding whether symptoms have physical or emotional causes.

Dr. Barbour argues that doctors are trained to be expert disease detectors, taught to diagnose patients based on fragments of information: a fever, an odd pain, a spell of disorientation. But this disease hunt can backfire, tempting them to lock on to a possible diagnosis prematurely. Dr. Barbour shared the transcript of an interview that illustrates this:

RESIDENT: What’s troubling you? PATIENT: I have this pain in my stomach (indicating with his hands the entire abdomen). RESIDENT: Where is it? PATIENT: Pretty much all over. RESIDENT: Is it here (pointing to the patient’s epigastrium)? PATIENT: Yes, I feel pain there. RESIDENT: When do you get it? PATIENT: A lot of the time. RESIDENT: Before meals? PATIENT: Yes, before meals, but I get it any old time.

Barbour pointed out how quickly the doctor takes over the interview—and how he prematurely localizes the patient’s complaint by asking, “Is it here?” (If the pain is “pretty much all over,” then, yes, it’s probably in that spot but also many others.) Meanwhile, the patient has been trained not to volunteer information. After a very brief exchange, the doctor develops a tentative diagnosis of peptic ulcer disease and orders the related tests. The tests prove him wrong.

Barbour’s assessment of this interview was damning: “Though [the resident] regards himself as objective and scientific, he manipulates the data to fit his concept of disease, but is not aware that he does so. He does not discover a pattern; he generates one.”

Unfortunately, the speed with which the doctor took over the interview in the case above may not be unusual. One study of patient interviews revealed that it took only 18 seconds, on average, for a doctor to interrupt a patient.

Barbour recommends a process that is better equipped to dodge the confirmation bias. When the doctor starts asking questions, she should start broad and open-ended: “What was the pain like? How did you feel?” Then she can move slowly and cautiously toward more directed questions: “Was it sharp or dull?” “Were you sad?” This gives the patient more room to answer before the doctor narrows the inquiry.

How do you know whether to ask probing questions or open-ended ones? A good rule of thumb is to ask yourself, “What’s the most likely way I could fail to get the right information in this situation?” Consider, for example: If you’re buying a used car, you’re most likely to fail by not discovering a flaw of the vehicle, or if you’re a vice president seeking feedback from factory workers, you’re most likely to fail by not uncovering what they really think. You can tailor your questions accordingly—more aggressive in the used-car negotiations and more open-ended with the factory worker.