Analytic Confidence

Analytic Confidence

While problems come in all varieties, shapes, and sizes, each can be categorized in terms of the roles that fact and judgment play in analysis of the problem. This categorization is called the “Taxonomy of Problem Types.” Figure 2-1, which depicts it, is not included in this excerpt.

Figure 2-1 represents Jones’s proposed inverse relationship between available facts and the judgment required to solve a problem: fewer facts leave more to judgment. From this relationship we can define four basic types of problems:

Simplistic

• There is only one answer.
• Examples: Who is buried in Grant’s tomb? Who is the governor of New York?

Deterministic

• Again there is only one answer, but it is determined by a formula.
• Examples: What is the area of a square whose side is 20 feet? If the circumference of a tire is 50 inches, how many revolutions must the tire make to travel one mile?

Random

• Different answers are possible, and all can be identified.
• Examples: Which of the candidates will win the election? Which of the builders who submitted a bid will get the contract?

Indeterminate

• Different answers are possible, but, because all or some are conjectural, not all possible answers can be identified.
• Examples: How will permitting gays to serve in the military affect the next presidential election? What are the prospects for U.S.-Russian relations?

In the case of the simplistic problem, no judgment is needed as there is only one answer—one fact. If we don’t know that fact, we can’t solve the problem.

In the deterministic problem, to get the answer, we must have all of the data as well as the correct formula. If either is lacking, the problem is insoluble. Again, judgment is not a factor unless we are required to speculate about either the data or the formula, in which case the problem becomes indeterminate.

In the random problem, because different answers are possible, judgment enters into the solution, but its role is limited by the fact that all possible answers are known.

In the indeterminate problem, judgment is unlimited, because not only are all possible answers not known but they are also conjectural.

Figure 2-2, also absent from this excerpt, restates this relationship: as the available facts decrease, the role of judgment increases.

There is also an obvious and direct correlation between the role of judgment and the probability of error. The more judgment we use in analyzing a problem, the greater is the likelihood of our making an error in our findings.

It follows then that, as the probability of error increases, our confidence in our findings must decrease. BUT IT DOESN’T! The human mind does not willingly or usually lose confidence in its conclusions, no matter what the circumstances. It is another of those curious human mental traits that mislead our thinking.

As I said earlier, cognitive experiments have shown that, even when the explanation we come up with doesn’t represent the evidence very well, we use that explanation anyway and feel comfortable doing so. What’s more, we defend that explanation in the face of strong contrary evidence, which we rebut with irrelevancies and sophistries. Regardless of the type of problem we are analyzing—simplistic, deterministic, random, or indeterminate —and regardless of how conjectural and subjective our analysis and its conclusions may be, we tend to defend our findings with the same high confidence.

And that is our third insight, to be wary of conclusions that are based largely on judgment, not facts. And if you are the author of those conclusions, take pains to couch them in language that accurately conveys the degree of your confidence in them. To do otherwise is, frankly, dishonest both to ourselves and to those who base decisions on our analysis.