Outside View (Base Rate Neglect)

1.

The photos on the Web site of Myrtle Beach’s Polynesian Resort depict a beach paradise, a landscape of golden sand and palm trees and colorful umbrellas. People recline on lounge chairs; a catamaran sails in the distance. It looks like just the kind of place you might want to take your family on a summer vacation.

By the time you’re reading this, it might indeed be a lovely place. But in 2011, when we saw the Web site, the Polynesian Resort had a nasty secret—it had been named by TripAdvisor, a travel advice Web site, as one of 2011’s Top 10 Dirtiest Hotels in the United States.

Many of the hotel’s past guests have shared their opinions of the experience on TripAdvisor; their commentaries are often scathing and hilarious:

Terri B (7/24/12): We checked in on July 21st and left in 10 minutes. I cannot put into words the horror.

Fetters26 (6/7/2011): My dog was at a kennel and his accommodations were much cleaner and more plush than the Polynesian!

Jackie503 (6/30/2011): The floors have not been mopped or vacuumed since Moses parted the Red Sea. The beds were old and the sheets were itchy …

4q2 (1/27/11): Many reviews compared this property to a dump. That just isn’t fair to the dumps of the world.

Thanks to sites like TripAdvisor, it’s easier for us to avoid making a hotel choice we’ll regret. We can ignore the glossy pictures and simply look at the reviews. In the case of the Polynesian Resort, 67% of the reviews rated the experience as “Terrible,” compared with a mere 4% who said “Excellent.” (One of those “Excellent” reviews cites the place as perfect for a debaucherous spring break, if you don’t mind the “filthy rooms.”)

Many people take this kind of review shopping for granted. We look up a book on Amazon, a restaurant on Yelp, or a digital camera on CNET. This “obvious” habit acknowledges two things: (1) Our impressions of a product are limited, and promotional material is designed to persuade us; and (2) Other people’s experiences can reveal problems our impressions miss. Reviews still need scrutiny: the sample, the reviewers’ priorities, and changes over time can all matter.

Often in life, though, we do the opposite: We trust our impressions over the averages. For example, many people will accept a new job without consulting a sample of people who currently or formerly held the same title. Shouldn’t their “reviews” be as valuable as a stranger’s assessment of a hotel room or restaurant?

Strange to think that when we make critical decisions, we do less objective research than when we’re picking a sushi joint.

Psychologists distinguish between the “inside view” and “outside view” of a situation. The inside view draws from information that is in our spotlight as we consider a decision—our own impressions and assessments of the situation we’re in. The outside view, by contrast, ignores the particulars and instead analyzes the larger class it’s part of. So in deciding whether to book a reservation at the Polynesian Resort, the inside view relies on our own assessment: Does this look like the kind of place where I would enjoy staying? The outside view trusts the TripAdvisor reviews: How much did people, in general, enjoy staying there?

The outside view can correct an impression built from a narrow set of details. Imagine a restaurateur, Jack, deciding whether to borrow money to start a Thai restaurant in downtown Austin. He focuses on the factors in his favor: I’m a wonderful Thai cook. The location on 4th Street would be perfect. The foot traffic in that area is huge. There’s no other Thai restaurant close by. From the inside view, the opportunity looks good.

The outside view asks how people in comparable circumstances have fared. It looks for “base rates”: outcomes across a relevant class of cases. Suppose Jack finds that 60% of restaurants in a comparison group failed in their first three years. That would be a reason to examine the risks and the comparison group closely, rather than relying entirely on his confidence in the idea.

Yet notice how different this feels from the TripAdvisor situation. It is intuitive for us to accept that we’re likely to have a bad experience at the Polynesian Resort, but it’s not intuitive for Jack to accept that he’s likely to fail. Why?

An outside view starts by setting aside the features that make our situation feel special. Entrepreneurs usually have reasons to expect success. Jack might say, “I know Thai food, and I know Austin, and I know this will work. You can’t lump me in with a guy who sells corn dogs at the mall.” He may have a useful point about the comparison group. But confidence alone is not a reason to discard the base rate. He should ask which restaurants are genuinely comparable and what evidence supports adjusting their results for his circumstances.*

Mind you, for Jack to take the outside view—and accept those bleak odds of restaurant success—does not demand that he give up on the idea. It may be that a successful restaurant would be so lucrative that the risk is worth taking. Or he may consider the restaurant a good investment in his career, even if it fails. The outside view doesn’t require defeatism, but it does require respect for the likely outcomes. Put it this way: If he bets his kids’ college money on the venture, he’s nuts.

Your friends and colleagues will suffer from this same stubbornness: the tendency to trust their own impressions too much. They’ll be trapped in the inside view, but you’ll have an easier time seeing the outside view. Be forewarned, though: Sometimes people can have access to the perfect set of data—and still manage to ignore it.

This was something Daniel Kahneman, the Nobel Prize–winning psychologist, experienced himself early in his career. He and his colleagues were exploring the idea of writing a high-school textbook on the subject of judgment and decision making. They would be the first to develop curricula on those subjects, so they roped in the dean of the School of Education, who was a curriculum expert, to work with them. The team began to write some sample chapters, and they met every Friday to review their progress. One Friday, they were discussing research about how groups think about the future, and it occurred to Kahneman that they should take their own advice. He said, “Let’s see how we think about the future.”

He asked his colleagues to write down the date when they thought the textbook would be completed. The range of estimates was quite narrow—everyone’s projections, including Kahneman’s and the dean’s, ranged from 1.5 to 2.5 years into the future. Then Kahneman suddenly recalled the idea of base rates from his statistics training, so he asked the dean whether he could recall other groups similar to theirs that had written a new curriculum from scratch. The dean said, yes, he could remember quite a few of them. Kahneman asked him to quantify the base rate: How long did it take them to finish?

After some back-and-forth, two disturbing facts had surfaced. One: According to the dean, 40% of the groups never finished writing the curriculum. Two: Of the groups that did finish, all of them took seven to ten years. Then Kahneman asked the dean, “How does our group compare to the others?” (Note that he’s trying to see whether there’s any reason to adjust their prediction up or down from the base rate, based on the group’s skill.) The dean replied, “Below average, but not by much.”

The curriculum took eight years to write.

LIKE JACK THE RESTAURANTEUR, Kahneman and his colleagues were optimistic when they took the inside view. The puzzle here is the dean’s behavior. He knew the base rates for developing a new curriculum, but his spotlight stayed trained on the group’s unique circumstances. From the inside view, it looked like they could wrap it up in two years. “There was no contact between something he knew and something he said.… He had all the information necessary to conclude that the prediction he was writing down was ridiculous,” said Kahneman.

This suggests a useful way to consult people with relevant experience. Ask about comparable cases as well as your own plans. Someone who has handled similar projects may know about delays, costs, or obstacles that you have not considered.

Relevant experience need not come with an impressive title. If your son wants to be a carpenter, talk to working carpenters. If you are considering moving a business to South Carolina, seek out people who have made a similar move. Ask how close their experience is to your situation and consider more than one account.

Be careful what you ask. Someone may know a great deal about past cases while still making an overconfident prediction about yours. Ask what usually happens, how they know, and why they expect your case to differ.

For example, questions for an IP lawyer might include “What are the important variables in a case like this?,” “What kind of evidence can tip the verdict one way or the other?,” “In percentage terms, how many cases get settled before trial?,” and “Of those that go to trial, what are the odds that the plaintiff prevails?” These questions invite an explanation of past cases and legal practice. Ask which answers rest on records and which are estimates from experience.

A question such as “Do you think I can win this case?” asks for a prediction. It can draw attention toward the details that make your case look promising. Follow up by asking how comparable cases turned out and what could make the estimate wrong.

A good lawyer can distinguish a strong case from a long shot. The point is to understand the basis of that assessment. Past outcomes provide a starting point; specific evidence may justify an adjustment. Neither a confident prediction nor a recalled base rate should be accepted without examining where it came from.