Goldberg Rule

RESULTS OF COMPARATIVE STUDIES

Goldberg studied the distinction between neurosis and psychosis based on the Minnesota Multiphasic Personality Inventory (MMPI), a personality test commonly used for such purposes (Goldberg, 1965, 1968). This differential diagnosis is of practical importance. For example, the diagnosis of psychosis may lead to needed but riskier treatments or to denial of future insurance applications. Goldberg derived various decision rules through statistical analysis of scores on 11 MMPI scales and psychiatric patients’ discharge diagnoses. The single most effective rule for distinguishing the two conditions was quite simple: add scores from three scales and then subtract scores from two other scales. If the sum falls below 45, the patient is diagnosed neurotic; if it equals or exceeds 45, the patient is diagnosed psychotic. This has come to be known as the Goldberg Rule.

Goldberg next obtained a total of 861 new MMPIs from seven different settings, including inpatient and outpatient services from either medical school, private, or Veterans Administration hospital systems in California, Minnesota, and Ohio. The accuracy of the decision rules when applied to these new cases was compared with that of 29 judges who analyzed the same material and attempted the same distinction. Some of the judges had little or no prior experience with the MMPI and others were Ph.D. psychologists with extensive MMPI experience.

Across the seven settings, the judges achieved mean validity coefficients ranging from r = 0.15 to 0.43, with a total figure of 0.28 for all cases, or 62% correct decisions. The single best judge achieved an overall coefficient of 0.39, or 67% correct decisions. In each of the seven settings, various decision rules exceeded the judges’ mean accuracy level. The Goldberg Rule performed similarly to the judges in three of the settings and demonstrated a modest to substantial advantage in four of the settings (where the rule’s validity coefficient exceeded that of the judges by 0.16 to 0.31). For the total sample, the Goldberg Rule achieved a validity coefficient of 0.45, or 70% correct decisions, thereby exceeding both the mean accuracy of the 29 judges and that of the single best judge.

Rorer and Goldberg then examined whether additional practice might alter results. Judges were given MMPI training packets consisting of 300 new MMPI profiles with the criterion diagnosis on the back, thus providing immediate and concrete feedback on judgmental accuracy. However, even after repeated sessions with these training protocols culminating in 4,000 practice judgments, none of the judges equaled the Goldberg Rule’s 70% accuracy rate with these test cases. Rorer and Goldberg finally tried giving a subset of judges, including all of the experts, the outcome of the Goldberg Rule for each MMPI. The judges were free to use the rule when they wished and knew its overall effectiveness. Judges generally made modest gains in performance but none could match the rule’s accuracy; every judge would have done better by always following the rule.

In another study using the same 861 MMPI protocols, Goldberg constructed mathematical (linear) models of each of the 29 judges that reproduced their decisions as closely as possible (Goldberg, 1970). Modeling judges’ decisions requires no access to outcome information. Rather, one analyzes relations between the information available to the judge and the judge’s decisions. In principle, if a judge weights variables with perfect consistency or reliability (that is, the same data always lead to the same decision), the model will always reproduce that judge’s decisions. In practice, human decision-makers are not perfectly reliable and thus judge and model will sometimes disagree. Goldberg found that in cases of disagreement, the models were more often correct than the very judges on whom they were based. The perfect reliability of the models likely explains their superior performance in this and related studies (Dawes & Corrigan, 1974).

Leli and Filskov studied the diagnosis of progressive brain dysfunction based on intellectual testing (Leli & Filskov, 1984). A decision rule derived from one set of cases and then applied to a new sample correctly identified 83% of the new cases. Groups of inexperienced and experienced clinicians working from the same data correctly identified 63% and 58% of the new cases, respectively. In another condition, clinicians were also given the results of the actuarial analysis. Both the inexperienced and experienced clinicians showed improvement (68% and 75% correct identifications, respectively), but neither group matched the decision rule’s 83% accuracy. The clinicians’ improvement appeared to depend on the extent to which they used the rule.

Einhorn (1972) studied the prediction of survival time following the initial diagnosis of Hodgkin’s disease as established by biopsy. At the time of the study, survival time was negatively correlated with disease severity; treatment has since changed the disease’s prognosis. All of the 193 patients in the study subsequently died, thus tragically providing objective outcome information.

Three pathologists, one an internationally recognized authority, rated the patients’ initial biopsy slides along nine histological dimensions they identified as relevant in determining disease severity and also provided a global rating of severity. Actuarial formulas were developed by examining relations between the pathologists’ ratings and actual survival time on the first 100 cases, with the remaining 93 cases used for cross-validation and comparison. The pathologists’ own judgments showed virtually no relation to survival time; cross-validated actuarial formulas achieved modest but significant relations. The study revealed more than an actuarial advantage. It also showed that the pathologists’ ratings produced potentially useful information but that only the actuarial method, which was based on these ratings, tapped their predictive value.