Evidentiary Thinking

Evidentiary Thinking

Consider a case of two brand managers independently thinking about introducing brand extensions to their brands. Further assume that the two brands are competitors, offering functionally comparable bundles of attributes, which amounts to two differently labeled versions of the same type of product. Both managers recently became immersed in EBP and are anxious to put those principles to work, thus are eagerly gathering evidence to help shape their brand extension-related decisions. For reasons that are beyond their control, one manager’s key piece of evidence is characterized as ‘purchase intention’ containing results of survey-gathered prospective customers’ interest in the proposed new product (brand extension), whereas the other manager’s key piece of evidence is described as ‘purchases’, or actual sales of the new product experimentally introduced in a couple of markets. If those are the most informative pieces of evidence available to both managers, should either of the two decision-makers put more stock in his or her evidence?

It has been documented by numerous investigations that the aggregate demand estimates derived from surveys capturing buyers’ intent to purchase a product are quite imperfect; thus, decision-guiding evidence derived from purchase intention estimates should not be considered as strong as evidence stemming from analyses of actual purchase patterns.10 Self-reported intentions are subject to the same cognitive biases that afflict any other cognitive reasoning process, and so even when respondents are honest and forthcoming, they may be providing unreliable information. Actual purchases, even if collected with the help of a limited scale trial, are pure behavioral outcomes, free of self-deceit and other truth-warping influences, and as such offer more dependable decision-making basis.

In a more general sense, this simple illustration underscores a more general conclusion, which is that not all evidence is equally compelling or equally accurate. The strength of evidence is an important consideration when considering the available and decision-pertinent insights, and it is clearly recognized in the idea of hierarchy of evidence discussed in the previous chapter. However, the medical research-inspired hierarchy which places results of systematic reviews and/or meta-analyses of empirical research studies at the top of the pyramid of authority is not an appropriate template for prioritization of management evidence. The primary reason for that is because unlike medical research, which directly benefits the teaching and the practice of medicine, management research only contributes some ideas some of the time – certainly not enough to warrant most authoritative positioning. In addition, the medicine-derived EBP framework does not expressly address organizational – especially transactional – data, which by virtue of its coverage, accuracy, level of detail, and recency offers the most complete, on-time and on-point decision insights. After all, it is not a coincidence that the new ‘commercial arms race’ now takes the form of massive investments in data science, business analytics, and the related technological infrastructure.

However, just as overreliance on subjective reasoning amplifies cognitive bias leading to poor judgment, overreliance on data can amplify the ever-present data error 11 leading to seemingly sound yet unwarranted conclusions. One of the perhaps better known examples of overreliance on data analytic results is offered by the case of collateralized debt obligations (CDOs), and American Insurance Group’s (AIG) accrual of huge and largely unfunded credit default swaps (effectively a form of insurance against CDO’s price collapse), the combination of which helped to trigger the financial crisis of 2007–2008, known as the Great Recession. The same is true of overreliance on scientific research studies, as exemplified by well-known case of drugs developed to fight arrhythmia, an irregular rhythm of the heart. Arrhythmia drugs-related research showed that those drugs indeed stopped the irregular heart rhythm, which was heralded as a major breakthrough. However, subsequent investigations into life-saving, rather than just arrhythmia-stopping value of those drugs revealed that treatments did not actually save lives – in fact, it was shown that patients who received such treatments were actually about 30% less likely to survive. The conclusion is rather obvious: no single type of evidence is immune to bias, and thus no single type of evidence should be deemed informationally sufficient.

At the same time, there is a limit on how many distinct pieces of information a decision-maker can simultaneously consider, as captured in the earlier (Chapter 1) discussed notion of human channel capacity. Based on an older concept of channel capacity, originally introduced in the context of information theory to describe the information handling capacity of a communication channel, the concept of human channel capacity places a numeric limit of 7 ± 2 of discrete pieces of information an average human can hold in his/her working memory. Thus, merely piling on more and more information won’t produce better decisions. In a manner that conceptually (only) resembles the logic of systematic reviews, available evidence needs to be screened and amalgamated in a conclusive manner. Stated differently, dependability of individual pieces of decision-related information needs to be assessed, following which potential differences need to be reconciled, all in the context of a sound foundation of probabilistic thinking.