Misrepresentative Evidence

MISREPRESENTATIVE EVIDENCE

Conditions, environments and circumstances change

People like to look for systems that have worked over the past 20 years or so. If you could make money based on what has worked the past 20 years, all of the richest people would be librarians. - Warren Buffett

Bertrand Russell said in Problems of Philosophy: "The man who has fed the chicken every day throughout its life at last wrings its neck instead." Often the past is a good guide to the future - but not always. Statistics are a record of the past, not a prediction of the future. We can't automatically assume that the future will mirror the past. Processes and circumstances change. Warren Buffett says: "Conditions relating to technology and all aspects of human behavior can make the future a lot different than the past."

We need to consider changes in conditions before using past evidence to predict likely future outcomes. For example, it's in the nature of businesses and economic conditions to change. Competition and demand changes. If there are more ways for creating competition or less demand, we have to change the equation. Ask: Why was past experience the way it was? What reason is there to suppose that the future will resemble the past? Has the environment changed? Are the conditions similar? Are the context and circumstances that caused the past still present?

We also make mistakes if we ignore that past performance may have been achieved under far different circumstances than today. As Warren Buffett says, "The same mistake that a baseball manager would were he to judge the future prospects of a 42-year-old center fielder on the basis of his lifetime batting average." Management performance may also be conditioned on environment. What makes an individual successful in one environment does not guarantee success in another. Ask: What is the company's or manager's ability to handle adversity?

"We can sell more if we market the illness rather than the drug," said the manager of TransCorp's pharmaceutical division.

Is the frequency of a disease really increasing? We need to consider other factors before we conclude that the frequency of an event has changed. For example, a disease may be more correctly diagnosed than it was in the past. Often we only see what we have names for - a disease that was formerly classified as "disease X" or "cause unknown" may now be re-classified or get a name. There may also be technological improvements in the collection and reporting of data. There may also be business incentives at work. For example, widening a market by creating a new condition, redefining a disease or exaggerating a minor one, thereby having more people labeled as having a disease.

If conditions change, we must update our assumptions to reflect the present environment. Before we use the change as evidence for what is likely to happen, ask: What has changed? Are there more ways for some undesirable event to happen? Is the change permanent or temporary?

The single case or unrepresentative samples

Four out of five doctors recommend the drug. This statement doesn't tell us anything if we don't know how many doctors were observed. Maybe it was just 10; an observation that can't be extrapolated to include all doctors. A small sample size has no predictive value. The smaller the sample is, the more statistical fluctuations and the more likely it is that we find chance events. We need a representative comparison group, large enough sample size, and long enough periods of time.

Small samples can cause us to believe a risk is lower or higher than reality. Why? A small sample increases the chance that we won't find a particular relationship where it exists or find one where it doesn't exist.

Charles Munger gives an example of the importance of getting representative data - even if it's approximate:

The water system of California was designed looking at a fairly short period of weather history. If they'd been willing to take less perfect records and look an extra hundred years back, they'd have seen that they weren't designing it right to handle drought conditions which were entirely likely.

You see that again and again - that people have some information they can count well and they have other information much harder to count. So they make the decision based only on what they can count well. And they ignore much more important information because its quality in terms of numeracy is less - even though it's very important in terms of reaching the right cognitive result. All I can tell you is that around Wesco and Berkshire, we try not to be like that. We have Lord Keynes' attitude, which Warren quotes all the time: "We'd rather be roughly right than precisely wrong." In other words, if something is terribly important, we'll guess at it rather than just make our judgment based on what happens to be easily countable.

Chance and performance

No victor believes in chance. - Friedrich Wilhelm Nietzsche

Past performance is no guarantee of future results. Consider evidence that describes what happens in most similar situations or to most people.

Sometimes a track record is not a good indicator of what is likely to happen in the future. It may show up by luck. Imagine a room filled with 1,000 monkeys. Each is trying to predict the direction (up or down) of interest rates. At the end of 10 predictions, one monkey has a perfect record of predicting the direction of interest rates. He is considered a genius and the greatest economist in history. Even if it was just by chance. As soon as we have a large population of forecasters that predict events where chance plays a role, someone will be right, get press coverage and be presented as a hero. He will hold lectures and give sensible explanations.

Sometimes we only see the good performers. Partly because winners have a tendency to show up (one monkey). Losers don't (999 monkeys). Often we aren't interested in the losers anyway. But we shouldn't be amazed to see winners if there is a large population to choose from. 10,000 monkeys and we find 10 geniuses. When we measure performance we must consider both the number of successes (one monkey), the number of failures (999 monkeys), and the size of the relevant comparison population they came from (1,000 monkeys). The more people (or monkeys) involved in something where chance plays a role, the more likely it is that some people have great performances just by chance. An exception is in a group of high performers where we can observe some common characteristics that may be a causal factor and not luck.

The same mistakes may happen when people base their conclusions on mere effects and ignore the influence of chance. Think about 100 monkeys. They each roll a die once. Select those 16 monkeys (1/6x100) who rolled a six. As a cure for their "roll-a-six" tendency we give them a new drug. After taking the drug they roll the die again. Now only 2 or 3 monkeys (1/6x16) rolled a six. The rest were "cured." Our false conclusion: "The drug obviously worked."

A con artist sets up a trap. He calls John with a tip. "Watch this stock. It will go up." After 3 correct predictions, John sends him his money. The con artist disappears. What John didn't know was that the con artist made the same call to 80 people. He told half of them the stock would go up, and the other half the stock would go down. And one of his predictions is sure to be right. 40 people were impressed. At the second call 20 people were impressed and at his third and last call he was considered a genius by 10 people who all sent him their money.

Ignoring failures

Evidence must be drawn from the frequency of both success and failure over time. Often we only consider the successful and supporting outcomes. The epidemiological literature refers to this as survival bias. Only the characteristics of the survivors of a disease or outcome under study are included in the study. Those who have died before the end of the study are excluded. If these are patients with more severe risk factors, the study reduces an apparent association between risk factors and outcomes. Survival bias is also common in all studies made after the outcomes have occurred (including back testing). They only focus on surviving cases or patients. The people who have died are not in the sampling pool. People may also select or omit certain information by only publishing positive outcomes and omitting negative ones.

If we only study successes or survivors, a performance record may look better than it really is. Charles Munger says that we give too little attention to failures:

It is assumed by many business school graduates, and by almost all consultants, that a corporation can easily improve its outcome by purchasing unrelated or tenuously related businesses. According to this widely shared view, if only the obvious steps had been taken, if the right "mission statement" had been adopted and the right "experts" hired, then each railroad, instead of remaining bound in chains by new forms of competition and obsolete and hostile laws and union rules, would have become another Federal Express, another United Parcel Service, or even another brilliant performer in the mode of Emerson Electric. Our experience, both actual and vicarious, makes us less optimistic about easy solutions through business acquisition. We think undue optimism arises because successful records draw too much attention. Many people then reason as I would if I forecasted good prospects in big-time tennis after observation limited to Ivan Lendl and Steffi Graf, or good prospects in the California lottery after limiting observations to winners. The converse is also true, only more so. Far too little attention is given to the terrible effects on shareholders (or other owners) of the worst examples of corporate acquisitions such as CBS-DuMont, Xerox-Scientific Data Systems, General Electric-Utah International, Exxon-Reliance Electric... and Avon Products.

In 1999, people said: "Internet businesses are doing great." Often we see only the businesses that do well and ignore the failures. Especially in bull markets where successes get wide publicity. Ask: What is the relevant comparison population from which we measure 5-year performance (assume 100 Internet businesses)? How many are doing well (assume 5). How many are not performing well (assume 80). How many have gone out of business (assume 15). From this we can draw the conclusion that the above statement is false.

TransCorps technical department developed a new defense system and claims a success rate of 80%. When John observed 10 tests, he witnessed 8 failures and only 2 successes. In 1992, Theodore Postal, a professor of science and national security at MIT, measured the effectiveness (not in terms of its psychological and political impact) of the Patriot anti-missile system in the Gulf War. Based on studying videotapes of 26 Patriot/Scud engagements involving 25 misses and 1 hit, he told the Committee on Government Operations that, "the video evidence makes an overwhelming circumstantial case that Patriot did not come close to achieving a 80 percent intercept rate in Saudi Arabia." The Pentagon reported firing 47 Patriot missiles at Scuds, at first claiming an 80% success rate. A congressional report later concluded that Patriots succeeded in downing only 4 Scuds.

If we assume that John's observations represent a valid random sample of tests, how likely is it that he would observe exactly 2 "hits" and 8 "misses" if the technical department's claim was true? This is an example of a binomial experiment. It describes experiments with repeated, identical trials where each trial can only have two possible outcomes (e.g. success or failure). Assuming independent engagements, the probability is only 0.007%. If the department's claim was true, then John witnessed an event of a very low probability. It is more likely that their claim was false.

Variability

You can, for example, never foretell what any man will do, but you can say with precision what an average number will be up to. Individuals vary, but percentages remain constant. So says the statistician. - Arthur Conan Doyle (from The Sign of Four)

When evaluating likely outcomes, look at the whole distribution of possible outcomes - average outcome, variability, and the probability of an extreme outcome and its consequences. What we mean by variability is how much the individual outcomes are spread out from the average outcome. The more spread out, the more variability. Ask: What has happened in the past? How much do the outcomes fluctuate around the average? What factors contribute to past variability? Have they changed?

The normal distribution curve shows a frequency distribution of outcomes and may sometimes help us find the most frequent outcomes and the variations. There are many ways a set of outcomes can be spread out or distributed. But some outcomes are more frequent than others. Many characteristics resulting from independent random factors have a bell-shaped frequency distribution. This means that the most frequent outcomes will be in the middle of the distribution, and the other outcomes will fall on either side of the middle. The further away any outcome is from the middle, the less frequent it is.

Examples of normally distributed outcomes are heights or weights of adults (which depend on factors such as genes, diet, or environment), temperatures, car accidents, mortality rates, or the length of life of a light bulb.

What is the average height of female adults? If we randomly select one thousand adult females and measure their height, we end up with a distribution of outcomes that look like a bell-shaped curve. Their heights will center around their average height, and the breadth of the curve indicates how variable around this average the heights are. Adding one extremely tall woman to a large sample doesn't really change the average height.

The same reasoning can be applied to an auto insurance company. The more of the same game the insurance company plays, the better the average becomes as a guide. Adding an extremely expensive car accident doesn't really change this. The average is representative and has predictive power.

But we can't use the normal distribution curve for those classes of insurance that involve mammoth and unusual risks. For an insurance company specializing in insuring unique events, the possible variability in outcomes is key.

TransCorp's new computer software grabbed 90% of the market. In many cases, the normal distribution curve may not give us a true picture of reality. One single favorable or unfavorable extreme event can have a large impact and dramatically change the averages. We saw examples of this uneven type of distribution earlier with size and frequency. The less number of times or the shorter the time or the more impact a single event may have on the average, the more important it is to consider the variability, and the more unpredictable some factor becomes. For example, suppose the average sales of books are $200,000. But if our sample contains one extreme best-seller (the Harry Potter books), the average will not help us predict what a new book on average may sell for. Other examples are movie sales (the movie Titanic), price changes in financial markets (a sudden large depreciation of a currency), damages from hurricanes (Katrina), or impact from innovations (TransCorp's new software). For example, there is no typical software and past averages don't mean anything. We see what has happened in the measured past. We don't know what the biggest hurricane will be or its likelihood of occurring. When we look back, we only see what the biggest hurricane was in the documented past.

We can't use past statistics to predict these rare and high-impact events. We don't know their timing, frequency, or degree of impact. We can't exactly figure out their properties or develop a formula. We only know that they happen and that they can have a huge impact. In some cases, we may have some evidence telling us if the probabilities are changing. Also, many events are not independent but are instead interconnected. Financial markets are one example; markets have memories, assets may be correlated and one bad event may cause another. The danger of using the normal distribution curve in cases where one huge event can dramatically change reality is well described at length by the Epistemologist of Randomness Nassim Nicholas Taleb in his book The Black Swan.

10 people have a total wealth of $10 million i.e. the average wealth per person is $1 million. What if one person has $10 million and nine nothing? How can the average income soar? Suppose ten middle-class people are riding on a bus. One gets off the bus and a billionaire gets on.

John has an option to play one of two games. Each game has three equally likely outcomes. The game may be one of chance, insurance, investing, etc.

Game1	30 40 50	40	1/3 1/3 1/3	40
Game2	0 20 100	40	1/3 1/3 1/3	40

If he plays each game over and over, the expected value is 40 for both, so it shouldn't matter what game he chooses to play. But what about if John only plays this game once? He should play game 1 since it has less variability.

John has been offered to invest in a private venture capital fund. The venture manager's track record is an average rate of return of 25% over the last five years. This doesn't say much if we don't look at how the underlying performance was produced. By looking closely at how this return was produced John found that the venture manager had done 10 deals. One deal had been a spectacular success and the rest failures. Had this one deal been due to luck?

Remember that some people leave out data when reporting their performance.

Mary reads in the paper that the average price of a house is $1,000,000. But this may cause her to get a false picture of reality. Assume there are 100 houses and 90 of them are priced at $500,000 and 10 "castles" at $5.5 million. We have to watch out for the variations.

A business executive tells us that his company had average earnings of $50 million over the last 3 years. But when we look closer we find great variability and a downward trend in the performance record: 1998: $100 million, 1999: $50 million, 2000: $0.

The median is the middle of a distribution where half the outcomes are above the median and half below. If 9 people have $1 million each and one has $1 billion, the average wealth is about $101 million, but the median amount is $1 million.

In 1982, Stephen Jay Gould was diagnosed with a rare and deadly form of cancer with a median mortality of 8 months after discovery. He beat the cancer for 20 years. Does an 8 month median mortality mean that a person will be dead in 8 months? Gould learned that an 8 month median mortality means half the people die before 8 months and half will live longer. But he found that there was a wider spread of outcomes after 8 months than before. This makes sense since there is a lower limit to the spread of outcomes during an 8 month period than after 8 months. No person will die before the start of the 8 month period but those who survive can live considerably longer than 8 months. We can't treat the average or the median as the most likely outcome for any single individual. Look at the variation between all outcomes. This means that a treatment ought to be determined based on whether an individual is likely to have an outcome better or worse than the median.

Effects of regression

"Regression to the mean" is a notion worked out by Sir Francis Galton (Charles Darwin's first cousin). It says that, in any series of events where chance is involved, very good or bad performances, high or low scores, extreme events, etc. tend on the average, to be followed by more average performance or less extreme events. If we do extremely well, we're likely to do worse the next time, while if we do poorly, we're likely to do better the next time. But regression to the mean is not a natural law. Merely a statistical tendency. And it may take a long time before it happens.

Dissatisfied with the new employees' performance, John put them into a skill enhancing program. He measures the employees' skills at the end of the program. Their scores are now higher than they were on the first test. John's conclusion: "The skill-enhancing program caused the improvement in skill." This isn't necessarily true. Their higher scores could be the result of regression to the mean. Since these individuals were measured as being on the low end of the scale of skill, they would have shown an improvement even if they hadn't taken the skill-enhancing program. And there could be many reasons for their earlier performance - stress, fatigue, sickness, distraction, etc. Their true ability perhaps hasn't changed.

Our performance always varies around some average true performance. Extreme performance tends to get less extreme the next time. Why? Testing measurements can never be exact. All measurements are made up of one true part and one random error part. When the measurements are extreme, they are likely to be partly caused by chance. Chance is likely to contribute less on the second time we measure performance.

If we switch from one way of doing something to another merely because we are unsuccessful, it's very likely that we do better the next time even if the new way of doing something is equal or worse.

Post Mortem

Spanish-American philosopher George Santayana once said: "Those who cannot remember the past are condemned to repeat it." How can we understand what is happening to us without any reference to the past? We conveniently forget to record our mistakes. But they should be highlighted. We should confess our errors and learn from them. We should look into their causes and take steps to prevent them from happening again. Ask:

• What was my original reason for doing something? What did I know and what were my assumptions? What were my alternatives at the time?
• How did reality work out relative to my original guess? What worked and what didn't?
• Given the information that was available, should I have been able to predict what was going to happen?
• What worked well? What should I do differently? What did I fail to do? What did I miss? What must I learn? What must I stop doing?

Why don't we do post mortems? Charles Munger says:

You tend to forget your own mistakes when reputation is threatened by remembering. For that very reason, one very wise company - Johnson & Johnson - has a system whereby two years or so after they've made some big acquisition they have a post-mortem. And they bring back the original projections and the original reasons for doing the deal. They identify the people who made the arguments and what have you. Then they compare them with how the deal worked out.

Warren Buffett says that, "Triumphs are trumpeted, but dumb decisions either get no follow-up or are rationalized." He continues:

Managers tend to be reluctant to look at the results of the capital projects or the acquisitions that they proposed with great detail only a year or two earlier to a board. And they don't want to actually stick the figures up there as to how the reality worked out relative to the projections. That's human nature. But I think you're a better doctor if you drop by the pathology department occasionally. And I think you're a better manager or investor if you look at each decision that you've made of importance and see which ones worked out and which ones didn't - and figure your batting average. Then, if your batting average gets too bad, you better hand the decision-making over to someone else.

We could also use pre-mortems to help us anticipate problems and key vulnerabilities. For example, before making an important decision, imagine a failure where things really have gone wrong and ask: What could have caused this?

What guidelines and tools are there to better thinking? Charles Munger gives us some introductory remarks for Part Four:

Berkshire is basically a very old-fashioned kind of a place and we try to exert discipline to stay that way. I don't mean old-fashioned stupid. I mean the eternal verities: basic mathematics, basic horse sense, basic fear, basic diagnosis of human nature making possible predictions regarding human behavior. If you just do that with a certain amount of discipline, I think it's likely to work out quite well.

• PART FOUR -

• • GUIDELINES TO BETTER THINKING

The brain can be developed just the same way as the muscles can be developed, if one will only take the pains to train the mind to think.

• Thomas Alva Edison (American inventor, 1847-1931)

• ONE -‌

Sun Tzu said in The Art of War: "The general who wins a battle makes many calculations in his temple before the battle is fought."

The purpose of this part is to explore tools that provide a foundation for rational thinking. Ideas that help us when achieving goals, explaining "why," preventing and reducing mistakes, solving problems, and evaluating statements.

The following 12 tools will be discussed:

1. Models of reality.
2. Meaning.
3. Simplification.
4. Rules and filters.
5. Goals.
6. Alternatives.
7. Consequences.
8. Quantification.
9. Evidence.
10. Backward thinking.
11. Risk.
12. Attitudes.