Data, Research, and Decision-Making

Data, Research, and Decision-Making

Whether to automate a decision depends on the task. Some choices have clear alternatives, a defined measure of success, and many relevant past examples. Others involve unfamiliar circumstances, conflicting goals, or consequences that are hard to anticipate. Treating every organizational decision as the same kind of problem obscures these differences.

Recurring tasks, such as routing an insurance claim or selecting a marketing action, may be candidates for rules or learned models. A strategic decision may require people to decide what outcome matters and whether past experience applies. These categories can overlap: a routine task can encounter an unusual case, and a strategic decision can contain smaller tasks suited to automation.

The discussion of cognitive biases in Chapter 1 gives one reason to structure decisions. People cannot attend to all available information at once. The often-cited estimate of about 7 ± 2 informational chunks concerns particular memory tasks; it is not a fixed capacity limit for every kind of decision. Biases and limited attention can nevertheless lead people to overlook relevant evidence.

A dual-process model distinguishes System 1, associated with relatively automatic and experience-based responses, from System 2, associated with more deliberate evaluation.2 These are useful descriptions of processing, not proof that the mind contains two separate decision engines. System 1 generally requires less effort than System 2, and System 1 can supply a quick response before alternatives are examined. Neither speed nor deliberation alone establishes that a choice is sound. Automation can reduce some demands on attention, but its results also need evaluation.

IBM’s Deep Blue matches against world chess champion Garry Kasparov offer a bounded example. Kasparov won in 1996; Deep Blue won the 1997 rematch. Chess supplies explicit rules and a clear result against which performance can be judged. Other applications include speech recognition, spelling correction, and search ranking. Machine learning develops patterns from training data, and deep learning uses multilayer models; neither label by itself establishes that a system will make better decisions. IBM’s Watson and Deep Blue, discussed in Chapter 1, addressed different tasks rather than representing interchangeable versions of one decision method.

For systems that learn from data, performance depends on accuracy, completeness, and the relationship between the training examples and the situation in which the system is used.3 Reliable past data can support automation without guaranteeing future reliability. Missing information, a changed setting, or a poorly chosen objective can undermine the result. Human judgment also needs scrutiny; uncertainty is not a reason to assume intuition will be superior.

Data can remain informative even when they do not cover the whole decision. An insurer’s automobile-accident records may support forecasts of auto claims if the relevant patterns remain stable. The same records cannot represent every line of insurance. Their accuracy as records and their relevance to a wider forecast are separate questions. Expanding the scope calls for additional evidence and revised assumptions, not automatic rejection or reuse of the existing model.

When choices are poorly defined, decision-makers may fall back on intuition. Studies of judgment show why that is not a sufficient safeguard.4 People may accept greater risks to avoid a loss or focus on percentage differences while overlooking absolute amounts. Naming these tendencies does not automatically remove them. State the alternatives, compare their consequences on a consistent basis, and identify what evidence would change the choice. The same discipline is useful when evaluating a person’s judgment or an automated recommendation.