Calibration in Weather Forecasting

Domain 2: Calibration in Weather Forecasting

The calibration of American meteorologists predicting precipitation (e.g., Murphy & Winkler, 1977) has been described as “superb” (Lichtenstein et al., 1982), “champion” (Edwards & von Winterfeldt, 1986) and as an “existence proof” of good calibration (Wallsten & Budescu, 1983). There are at least five non-exclusive explanations for this excellent calibration: the use of sophisticated computer models and centrally provided forecasts that allow direct “pattern matching” of present cues to past outcomes (e.g., Fischhoff, 1988), the clear and unambiguous feedback received by the forecasters coupled with their knowledge that they cannot affect the outcome (Murphy & Winkler, 1984; Keren, 1991), training in probabilistic thinking (Murphy & Brown, 1984), the explicit presentation of base rates that allow their use as an “anchor” at the time of prediction, and, consistent with the direct support account, the combination of accurate cues (i.e., high discrimination) with moderate base rates.

Some historical information casts doubt on the necessary roles of computer models and training in probabilistic thinking. Murphy and Winkler (1984) present data collected by Hallenback (1920) on his probabilistic predictions of rain in the Pecos Valley in New Mexico. These predictions (base rate 42%), based on composite weather maps, show very good calibration, with only a small tendency towards underprediction in the middle and high ranges. However, they cannot be described as a triumph of intuitive judgment; they were really a triumph of early weather technology in the actuarial mode. Hallenback (1920) described a pattern-matching or “table look-up” strategy in which certain prototypical weather patterns were matched with the relative frequency of past outcomes: “The basis of this method is a series of composite weather maps showing the frequency of precipitation, in percentages, with different types of pressure distribution” (p. 645).

Winkler and Poses (1993) noted that the excellent calibration of modern American weather forecasters predicting rain may be related to the moderate base rate (about 25% in the Murphy & Winkler, 1977, analysis), and contrasted this with the poorer calibration of weather forecasters predicting extreme storms (base rate 13%). However, they cautioned that the forecasters also have more experience with forecasting precipitation than forecasting storms. A closer look at the moderate overprediction of storms indicates an intriguing pattern: “a strong tendency to overforecast existed for the smaller areas and . . . a tendency to underforecast existed for the larger areas” (Murphy, 1981, p. 72). That is, when storms are rare (as in smaller forecast areas) forecasters tend to overuse high probabilities; when storms are more common (as in larger forecast areas) forecasters tend to underuse high probabilities, consistent with base rate neglect and the direct support account.

Murphy and Daan (1984) reported another striking finding in their study of Dutch weather forecasters. The forecasters were given detailed probabilistic feedback based on one year’s worth of predictions. Data collected during the second year of the study showed that this feedback served to reduce the overprediction observed in the first year so that the overall calibration curve (averaged across many prediction events) for the second year followed the identity line fairly closely. However, as Murphy and Daan (1984) noted, “the amount of overforecasting increases, in general, as the sample climatological probabilities of the events decrease in both years” (p. 416). [Figure 39.6] displays this pattern for the prediction of various weather events along with the relevant direct support lines (representing complete neglect of base rates and evidence quality). Weather events have been categorized in terms of base rate and discriminability as shown in [Table 39.2].

Note: Neglect model predictions based on RST simulation assuming no focal bias (β = 0) and fixed judgmental extremity (σ = 1). Discriminability (D) and base rate (BR) is matched to average values from the Daan and Murphy (1982) study, with α = 1.19, 0.86, and 0.96 in the very low (BR = 6%), low (BR = 29%), and high (BR = 75%) base rate conditions, respectively.

[Table 39.2] provides target event description, base rate (BR), and discriminability (α) for Daan & Murphy (1982) weather forecasting data.

Consistent with the general predictions of direct support theory, when the meteorological event was common, calibration was very good, whereas when the event was rare, there was marked overprediction. Did this occur because the meteorologists were reporting the strength of their evidence for a given hypothesis without due regard for the base rate, or because the rarer events were also more severe and meteorologists were determined not to miss them? It is impossible to rule out the presence of a “warning bias” in this situation, but data from other types of forecasts cast doubt on this explanation. Much more overprediction is found for predictions of rainfall greater than 1.4 mm (base rate 9%) than predictions of rainfall greater than .2 mm (base rate 20%), although the former is by no means an extreme amount of rain. Similarly, there is more overprediction when judges evaluate whether visibility will be less than 4 km (base rate 17%) than when they evaluate whether visibility will be less than 10 km (base rate 44%). In this case, the 4-km prediction does not include extreme or dangerous events as these form another category, visibility less than 1 km (base rate 3%, with extreme overprediction). Although value-induced or warning biases may add to the tendency for overprediction, the evidence suggests that the base rate had a substantial and independent effect on miscalibration. Inspection of [Fig. 39.6] reveals that the empirical points are somewhat closer to the line of perfect calibration than the RST prediction lines; this indicates that the meteorologists were giving more weight to the base rate than the “neglect” model predicted. This discrepancy from the case-based support model may reflect the technical information provided to the forecasters or their training in statistical models. The cited figure and table are not included here.