Wanna Bet on Science?

Wanna bet (on science)?

Could thinking in bets help scientists assess whether experimental results will replicate? One might guess that scientists would be more accurate if they had to bet on the likelihood that results would replicate as compared to traditional peer review, which can be vulnerable to viewpoint bias. Especially in an anonymous betting market, confirming the strength of your pre-existing ideology or betting solely on the basis that replication of a study confirms your own work or beliefs counts for nothing. The way a scientist would be “right” in such a betting market is by using their skill in a superior way to make the most objective bets on whether results would or would not replicate. Researchers who knew in advance their work would be subject to a market test would also face an additional form of accountability that would likely modulate their reporting of results.

At least one study has found that, yes, a betting market where scientists wager on the likelihood of experimental results replicating was more accurate than expert opinion alone. In psychology, there has been a controversy over the last decade about a potentially large number of published studies with results subsequent researchers could not replicate. The Reproducibility Project: Psychology has been working on replicating studies from top psychology journals. Anna Dreber, a behavioral economist at the Stockholm School of Economics, with several colleagues set up a betting market based on these replication attempts. They recruited a bunch of experts in the relevant fields and asked their opinions on the likelihood the Reproducibility Project would replicate the results of forty-four studies. They then gave those experts money to bet on each study’s replication in a prediction market.

Experts’ individual forecasts of whether an experimental result would replicate were right 58% of the time. A betting market in which the traders were the exact same experts and those experts had money on the line predicted correctly 71% of the time.

A lot of people were surprised to learn that the expert opinion expressed as a bet was more accurate than expert opinion expressed as an individual forecast, since peer review is considered a rock-solid foundation of the scientific method. Of course, this result shouldn’t be surprising to readers of this book. We know that scientists are dedicated to truthseeking and take peer review seriously. Arguably, there is already an implied betting element in the scientific process, in that researchers and peer reviewers have a reputational stake in the quality of their review. But we know that scientists, like judges—and like us—are human and subject to these patterns of confirmatory thought. Making the risk explicit rather than implicit refocuses us all to be more objective.

Duke cites Google, Microsoft, General Electric, Eli Lilly, Pfizer, and Siemens as companies that have used prediction markets to test decisions and encourage contrary opinions. People are more willing to offer their opinion when the goal is to win a bet rather than get along with people in a room.

Accuracy, accountability, and diverse perspectives are Duke’s proposed conditions for a useful decision group. The excerpt then moves to chapter 5, Dissent to Win, whose discussion is not included here.