Perfect Calibration
Well-calibrated confidence matches observed accuracy over many judgments. Galef uses Spock’s fictional predictions and a trivia exercise to show how to compare the two and identify patterns of overconfidence or underconfidence.
THE IDEA
PERFECT CALIBRATION
Perfect calibration is an abstract ideal, not something that’s possible to achieve in reality. Still, it’s a useful benchmark against which to compare yourself. To get a hang of the concept, let’s look at Spock and see how his calibration measures up against perfection.
I went through all of Spock’s appearances in Star Trek: The Original Series, Star Trek: The Animated Series, and the Star Trek movies, searching for the words probability, percent, odds, chance, possible, impossible, likely, unlikely, probable, and improbable. In total, I found twenty-three instances in which Spock made a prediction with a corresponding confidence level, and in which the prediction was proven either true or false. You can read the full details of Spock’s predictions and how I categorized them in Appendix A of The Scout Mindset, but here’s the headline summary:
When Spock thinks something is impossible, it happens 83 percent of the time.
When Spock thinks something is very unlikely, it happens 50 percent of the time.
When Spock thinks something is unlikely, it happens 50 percent of the time.
When Spock thinks something is likely, it happens 80 percent of the time.
When Spock thinks something is more than 99.5 percent likely, it happens 17 percent of the time.9
As you can see, he’s not doing great. The only confidence level at which he seems to be well calibrated is when he judges something to be “likely”; those predictions actually do come true at a rate that matches his confidence level. Other than that, Spock’s predictions are anti-correlated with reality—the less likely he thinks something is, the more likely it is to happen, and the more likely he thinks something is, the less likely it is to happen.
SPOCK’S CALIBRATION (N = 23)
Want to see if you can do better than Spock? You can test your own calibration, and practice feeling the difference between different levels of certainty, by answering a few rounds of trivia questions. Below are forty questions for you to practice on. You don’t need to answer all of them, but the more you answer, the more informative your results will be.
For each question, circle an answer and then indicate how sure you are by circling a confidence level. Since these questions have only two possible answers, your confidence level could range from 50 percent if you truly have no clue (i.e., you might as well be guessing the outcome of a coin flip) to 100 percent if you think there is no chance you could be wrong. For the sake of simplicity, I’ve listed five confidence levels between those extremes: 55%, 65%, 75%, 85%, and 95%. Just circle the one that best represents how sure you are.
As you go through the list, you should notice your level of certainty fluctuating. Some questions might feel easy, and you’ll be near certain of the answer. Others may prompt you to throw up your hands and say, “I have no idea!” That’s perfectly fine. Remember, the goal isn’t to know as much as possible. It’s to know how much you know.
CALIBRATION PRACTICE: CIRCLE YOUR ANSWERS.
| Round 1: Are these animal facts true or false? | How sure are you? |
|---|---|
| 1. The elephant is the world’s largest mammal. (T / F) | 55% 65% 75% 85% 95% |
| 2. Sea otters sometimes hold hands while they sleep. (T / F) | 55% 65% 75% 85% 95% |
| 3. Centipedes have more legs than any other animal. (T / F) | 55% 65% 75% 85% 95% |
| 4. Mammals and dinosaurs coexisted. (T / F) | 55% 65% 75% 85% 95% |
| 5. Bears can’t climb trees. (T / F) | 55% 65% 75% 85% 95% |
| 6. Camels store water in their humps. (T / F) | 55% 65% 75% 85% 95% |
| 7. Flamingos are pink because they eat shrimp. (T / F) | 55% 65% 75% 85% 95% |
| 8. The giant panda eats mostly bamboo. (T / F) | 55% 65% 75% 85% 95% |
| 9. The platypus is the only mammal that lays eggs. (T / F) | 55% 65% 75% 85% 95% |
| 10. A mule is a cross between a male donkey and a female horse. (T / F) | 55% 65% 75% 85% 95% |
| Round 2: Which historical figure was born first? | How sure are you? |
| 11. Julius Caesar or Confucius? | 55% 65% 75% 85% 95% |
| 12. Fidel Castro or Mahatma Gandhi? | 55% 65% 75% 85% 95% |
| 13. Nelson Mandela or Anne Frank? | 55% 65% 75% 85% 95% |
| 14. Cleopatra or Muhammad? | 55% 65% 75% 85% 95% |
| 15. William Shakespeare or Joan of Arc? | 55% 65% 75% 85% 95% |
| 16. George Washington or Sun Tzu? | 55% 65% 75% 85% 95% |
| 17. Genghis Khan or Leonardo da Vinci? | 55% 65% 75% 85% 95% |
| 18. Queen Victoria or Karl Marx? | 55% 65% 75% 85% 95% |
| 19. Saddam Hussein or Marilyn Monroe? | 55% 65% 75% 85% 95% |
| 20. Albert Einstein or Mao Zedong? | 55% 65% 75% 85% 95% |
| Round 3: Which country had more people in 2019? | How sure are you? |
| 21. Germany or France? | 55% 65% 75% 85% 95% |
| 22. Japan or South Korea? | 55% 65% 75% 85% 95% |
| 23. Brazil or Argentina? | 55% 65% 75% 85% 95% |
| 24. Egypt or Botswana? | 55% 65% 75% 85% 95% |
| 25. Mexico or Guatemala? | 55% 65% 75% 85% 95% |
| 26. Panama or Belize? | 55% 65% 75% 85% 95% |
| 27. Jamaica or Haiti? | 55% 65% 75% 85% 95% |
| 28. Greece or Norway? | 55% 65% 75% 85% 95% |
| 29. China or India? | 55% 65% 75% 85% 95% |
| 30. Iraq or Iran? | 55% 65% 75% 85% 95% |
| Round 4: Are these science facts true or false? | How sure are you? |
| 31. Mars has one moon, just like Earth. (T / F) | 55% 65% 75% 85% 95% |
| 32. Scurvy is caused by a deficit of vitamin C. (T / F) | 55% 65% 75% 85% 95% |
| 33. Brass is made from iron and copper. (T / F) | 55% 65% 75% 85% 95% |
| 34. One tablespoon of oil has more calories than one tablespoon of butter. (T / F) | 55% 65% 75% 85% 95% |
| 35. Helium is the lightest element. (T / F) | 55% 65% 75% 85% 95% |
| 36. The common cold is caused by bacteria. (T / F) | 55% 65% 75% 85% 95% |
| 37. The deepest place on Earth is in the Pacific Ocean. (T / F) | 55% 65% 75% 85% 95% |
| 38. Seasons are caused by the earth orbiting the sun in an elliptical path. (T / F) | 55% 65% 75% 85% 95% |
| 39. Jupiter is the largest planet in our solar system. (T / F) | 55% 65% 75% 85% 95% |
| 40. The atoms in a solid are more densely packed than the atoms in a gas. (T / F) | 55% 65% 75% 85% 95% |
Once you’ve finished, or answered as many questions as you want to, it’s time to score yourself. The answer key is in Appendix B of The Scout Mindset, linked in the source as this page. Use that key to find out which answers you got right and which you got wrong.
Next, go through only the questions about which you were “55% sure” and calculate the percentage of those questions you actually got right. For example, if there were 10 questions for which you said you were 55 percent sure of your answer, and you got 6 of those questions right, then your total percentage right at this confidence level would be 6 / 10 = 60 percent. Repeat the calculation for the other confidence groups. With only a few questions in a group, the result will be noisy; look for patterns over repeated rounds.
When to use it
Forecasting outcomes in business or sports, where you need to assign realistic probabilities to events.
Self-assessment in exams or certifications: after scoring, compare your predicted confidence with actual performance to identify areas of overconfidence.
Evaluating risk in financial decisions, such as estimating the probability of a market downturn or a project's success.
Improving team decision-making by encouraging members to state their confidence levels and then calibrating based on outcomes.
FROM THEORY TO EVERYDAY LIFE
In practice
Spock's predictions
Spock's probability judgments from Star Trek were measured: when he said something was impossible, it happened 83% of the time; very unlikely, 50%; unlikely, 50%; likely, 80%; and more than 99.5% likely, 17%. This shows poor calibration, where his confidence often mismatched reality.
Calibration practice trivia
The exercise has 40 two-choice trivia questions about animals, historical figures, country populations and science. Confidence starts at 50% for a pure guess, and the listed practice choices extend up to 95%. Comparing confidence with accuracy across questions helps reveal calibration.
A MOMENT TO REFLECT
Put it to work.
After completing the calibration practice, analyze your results: for each confidence level (55%, 65%, etc.), did your actual accuracy match your confidence? Where were you overconfident or underconfident?
Think of a recent decision or prediction you made with high confidence. Was that confidence justified? How could you have better calibrated your certainty?
How can the ideal of perfect calibration help you improve your everyday judgments, such as in hiring, investing, or even personal relationships?
CONTINUE THE CONVERSATION
Source texts
- The Scout Mindset (Julia Galef)