Hypothesis

Hypotheses

A hypothesis is a claim proposed for testing. In his writings in the 1930s, Karl Popper emphasized that observations cannot conclusively verify an unrestricted empirical generalization, though a reliable counterexample can challenge it. This does not mean that every kind of hypothesis is unprovable. Circumnavigation can supply evidence against a flat-Earth claim, but the expedition begun by Magellan did not introduce the idea of a spherical Earth; that understanding long predated the voyage.

We disprove a hypothesis with evidence. “Information” becomes “evidence” only when we connect it with a hypothesis. Thus, we scan “information” looking for “evidence.”

When we find “evidence,” we should try to establish its validity by answering four questions:

1. Who or what was the source?
2. What was the source’s access? How did the source obtain the information? Was that method plausible? For example, if the source of the information claims he or she read the information in a certain document, is it reasonable that the source had access to that document?
3. What is the source’s reliability? Is the source reputable? Has other information from the source proven to be accurate?
4. Is the information plausible? From the standpoint of everything we know about the problem and from just plain old common sense, does the information seem to make sense? Is such information common or rare?

Illustrative of the problems and issues associated with source validation was the controversy reported in The Washington Post surrounding a Soviet document alleging that the government of North Vietnam had lied about the number of U.S. prisoners of war it held at the end of the Vietnam conflict. In April 1993, U.S. Department of Defense officials and intelligence analysts accepted the document as authentic—that is, it was determined to be a genuine official Soviet intelligence document. But according to press articles, they challenged the validity of the information it contained because it conflicted with strong evidence from reliable sources. The Harvard-based researcher who obtained the document from the Russian government’s archives in Moscow reportedly based the credibility of his research on the validity of Soviet intelligence and categorically denied that the Soviet intelligence on which he planned to base his book was flawed. He was quoted as saying, “If this story is flawed … then the whole Soviet Politburo was misinformed about the Vietnam conflict [in] which they invested billions of rubles.” It was apparently inconceivable to the researcher that an official intelligence document, and a Soviet document as well, could be inaccurate. In short, he seemed to have difficulty distinguishing between the authenticity of a document and the accuracy of the information it contains.

Hypotheses play a vital role in analysis. By yielding to the mind’s proclivity to focus on one solution, a hypothesis helps to narrow the scope of our thinking. It thus provides a framework—a mind-set—within which to analyze and interpret information. While this focusing is normally beneficial to analysis, it can have dramatically negative effects if it leads to satisficing—to focusing on one hypothesis to the exclusion of alternatives.

So what can we do to offset our natural tendency to satisfice? You know the answer. All together, now: structure! And how might we structure analysis to ensure that all hypotheses are considered sufficiently to test their degree of validity? By using a technique called, strangely enough, “hypothesis testing.”

The technique ranks competing hypotheses by how strongly key evidence conflicts with each. A hypothesis that fits many observations may still have a serious contradiction. Evidence consistent with several alternatives does little to distinguish them, so the matrix emphasizes those differences rather than a simple count of supporting items.

The method encourages analysts to seek evidence that could undermine a favored explanation. That does not make confirming evidence worthless: it can be informative when rival hypotheses predict it differently. Source quality and the strength of each contradiction also matter.

A Hypothesis-Testing Matrix, presented as Figure 11-1 in the source, lists key evidence down the left and competing hypotheses across the top. Each cell records how the evidence bears on that hypothesis.