Overconfidence in Uncertainty Estimates

UNCERTAINTY IN OTHER CONSTANTS AND RECOMMENDED VALUES

Table 37.1 summarizes similar analyses for other physical constants, including the gravitational constant G, the magnetic moment of the proton, two sets of measures of the inverse fine structure constant, α−1, and the ratio of the absolute ohm to the ohm maintained by the National Bureau of Standards. Table 37.1 reveals a similar pattern of overconfidence in all cases but the last. This shows a Birge ratio of significantly less than 1.0 and an interquartile ratio of 100%, in contrast to the other cases.

The “particle lives” and “particle masses” entries in Table 37.1 represent 92 measurements of Kaon and hyperon lifetimes and 214 measurements of meson resonance masses taken from the Particle Data Group’s biennial Review of Particle Properties (Roos et al., 1975; Rosenfeld, 1975). Their thorough scrutiny of all published experimental measurements results in the rejection of about 40% on grounds of suspect assumptions, poor-quality work, unreported errors, or gross inconsistency with other results. Nonetheless, the Birge ratio of 1.24 and surprise index of 6% of the 306 combined results indicate a significant degree of overconfidence in the remaining uncertainty assessments.

Periodically, reviewers propose “recommended values” for physical constants after careful consideration of all the most precise measurements. The diligence of these analyses and the incumbent obligation to confront the inconsistencies among studies should make them particularly sensitive to uncertainty. Figure 37.2 showed the residual overconfidence in such reviews for the velocity of light. Figure 37.3 shows comparable results for five other fundamental constants derived from a series of six reviews between 1952 and 1973 (Cohen et al., 1955, 1957; Taylor et al. 1969; Dumond & Cohen, 1953; Cohen & Dumond 1965; Cohen & Taylor, 1973). In most cases, the best-guess value at each revision was well outside the range of possibilities defined by the uncertainty estimates for the preceding evaluation period. For example, the 1963–1969 shift was 3 to 5 1963 standard deviations for all five of these constants. Table 37.1 (next to last line) gives calibration indices for these same five constants for eight reviews from 1929 to 1969, with residuals calculated relative to 1973 estimates (Cohen et al., 1955, 1957; Taylor et al., 1969; Birge, 1929, 1942; Dumond & Cohen, 1953; Cohen & Dumond, 1965; Cohen & Taylor, 1973). The surprise index of 57% indicates that the estimates used in the review would have come as a surprise to earlier reviewers in more than half the cases. The 1969 review acknowledges these problems, and provides graphs similar to Fig. 37.3 to caution readers to be skeptical (Taylor et al., 1969).