Reliable-Unreliable Asymmetry

intermediate2 min readstatistics · psychology

In the reported tasks, people were more confident predicting a less reliable measure from a more reliable one than in the reverse direction. The authors connect this asymmetry with treating less reliable information as easier to revise.

THE IDEA

Reliable and Unreliable Knowledge

Tversky and Kahneman (1982) noted an asymmetry in the confidence with which people made inferences and predictions from one attribute to another. Inferences from a reliable measure to an unreliable one are made with greater confidence than inferences in the opposite direction, although correlation is actually symmetric. For example, people were more confident in predicting the score on a short IQ test from a long form of the test than vice versa. They also believed, erroneously, that the statement, “The individual who won the decathlon won the first event” is more probable than, “The individual who won the first event won the decathlon.”

Similar asymmetries are observed when subjects are presented with a statement that includes an apparent discrepancy and are asked to choose how to eliminate the discrepancy. Most subjects do so by altering the less-reliable item (e.g., the performance on the short test or in the first event of the decathlon) to fit the more reliable one. Thus, attributes about which little is known appear to be relatively mutable. Correlation symmetry alone does not establish equal predictive uncertainty in every setting; the outcome scales, error model, and prediction task also matter.

When to use it

When interpreting test scores, account for the reliability of each measure before making predictions or drawing conclusions.
In hiring, if you have a highly reliable assessment and a less reliable interview, avoid overconfidence in predicting interview performance from the assessment.
In sports, if a player wins an early event, resist the urge to assume they will win the overall competition if the early event is less reliable than the overall measure.
In research, distinguish the symmetry of a correlation coefficient from the uncertainty of a particular directional prediction.

FROM THEORY TO EVERYDAY LIFE

In practice

IQ Test Predictions

People are more confident predicting performance on a short IQ test from a long-form test than predicting the long-form score from the short test, despite symmetric correlation.

Decathlon Winner and First Event

People erroneously believe that the statement 'The decathlon winner won the first event' is more probable than 'The first event winner won the decathlon.'

✳

A MOMENT TO REFLECT

Put it to work.

Think of a situation where you are more confident predicting a less reliable variable from a more reliable one. How might this asymmetry lead to errors in judgment?
Consider a correlation between two measures with different reliability. Why do people tend to adjust the less reliable measure to fit the more reliable one, and is this always rational?
How can awareness of this asymmetry improve your decision-making when interpreting data or making predictions?