Survivorship Bias and Its Extensions

IT IS EASIER TO BUY AND SELL THAN FRY AN EGG

Some technical extensions of the survivorship bias. On the distribution of “coincidences” in life. It is preferable to be lucky than competent (but you can be caught). The birthday paradox. More charlatans (and more journalists). How the researcher with work ethics can find just about anything in data. On dogs not barking.

This afternoon I have an appointment with my dentist (it will mostly consist in the dentist picking my brain on Brazilian bonds). I can state with a certain level of comfort that he knows something about teeth, particularly if I enter his office with a toothache and exit it with some form of relief. It will be difficult for someone who knows literally nothing about teeth to provide me with such relief, except if he is particularly lucky on that day—or has been very lucky in his life to become a dentist while not knowing anything about teeth. Looking at his diploma on the wall, I determine that the odds that he repeatedly gave correct answers to the exam questions and performed satisfactorily on a few thousand cavities before his graduation—out of plain randomness—are remarkably small.

Later in the evening, I go to Carnegie Hall. I can say little about the pianist; I even forgot her unfamiliar foreign-sounding name. All I know is that she studied in some Muscovite conservatory. But I can expect to get some music out of the piano. It will be rare to have someone who performed brilliantly enough in the past to get to Carnegie Hall and now turns out to have benefited from luck alone. The expectation of having a fraud who will bang on the piano, producing only cacophonous sounds, is indeed low enough for me to rule it out completely.

I was in London last Saturday. Saturdays in London are magical; bustling but without the mechanical industry of a weekday or the sad resignation of a Sunday. Without a wristwatch or a plan I found myself in front of my favorite carvings by Canova at the Victoria and Albert Museum. My professional bent immediately made me question whether randomness played a large role in the carving of these marble statues. The bodies were realistic reproductions of human figures, except that they were more harmonious and finely balanced than anything I have seen mother nature produce on its own (Ovid’s materiam superabat opus comes to mind). Could such finesse be a product of luck?

I can practically make the same statement about anyone operating in the physical world, or in a business in which the degree of randomness is low. But there is a problem in anything related to the business world. I am bothered because tomorrow, unfortunately, I have an appointment with a fund manager seeking my help, and that of my friends, in finding investors. He has what he claims is a good track record. All I can infer is that he has learned to buy and sell. And it is harder to fry an egg than buy and sell. Well . . . the fact that he made money in the past may have some relevance, but not terribly so. This is not to say that it is always the case; there are some instances in which one can trust a track record, but, alas, there are not too many of these. As the reader now knows, the fund manager can expect to be heckled by me during the presentation, particularly if he does not exhibit the minimum of humility and self-doubt that I would expect from someone practicing randomness. I will probably bombard him with questions that he may not be prepared to answer, blinded by his past results. I will probably lecture him that Machiavelli ascribed to luck at least a 50% role in life (the rest was cunning and bravura), and that was before the creation of modern markets.

In this chapter, I discuss some well-known counterintuitive properties of performance records and historical time series. The concept presented here is well-known for some of its variations under the names survivorship bias, data mining, data snooping, over-fitting, regression to the mean, etc., basically situations where the performance is exaggerated by the observer, owing to a misperception of the importance of randomness. Clearly, this concept has rather unsettling implications. It extends to more general situations where randomness may play a share, such as the choice of a medical treatment or the interpretation of coincidental events.

When I am tempted to suggest a possible future contribution of financial research to science in general, I adduce the analysis of data mining and the study of survivorship biases. These have been refined in finance but can extend to all areas of scientific investigation. Why is finance so rich a field? Because it is one of the rare areas of investigation where we have plenty of information (in the form of abundant price series), but no ability to conduct experiments as in, say, physics. This dependence on past data brings about its salient defects.