Data Snooping Bias (The Backtester)

The Backtester

A programmer helped me build a backtester. It is a software program connected to a database of historical prices, which allows me to check the hypothetical past performance of any trading rule of average complexity. I can just apply a mechanical trading rule, like buy NASDAQ stocks if they close more than 1.83% above their average of the previous week, and immediately get an idea of its past performance. The screen will flash my hypothetical track record associated with the trading rule. If I do not like the results, I can change the percentage to, say, 1.2%. I can also make the rule more complex. I will keep trying until I find something that works well.

What am I doing? The exact same task of looking for the survivor within the set of rules that can possibly work. I am fitting the rule on the data. This activity is called data snooping. The more I try, the more I am likely, by mere luck, to find a rule that worked on past data. A random series will always present some detectable pattern. I am convinced that there exists a tradable security in the Western world that would be 100% correlated with the changes in temperature in Ulan Bator, Mongolia.

To get technical, there are even worse extensions. An outstanding paper by Sullivan, Timmerman, and White goes further and considers that the rules that may be in use successfully today may be the result of a survivorship bias.

Suppose that, over time, investors have experimented with technical trading rules drawn from a very wide universe—in principle thousands of parameterizations of a variety of types of rules. As time progresses, the rules that happen to perform well historically receive more attention and are considered “serious contenders” by the investment community, while unsuccessful trading rules are more likely to be forgotten. . . . If enough trading rules are considered over time, some rules are bound by pure luck, even in a very large sample, to produce superior performance even if they do not genuinely possess predictive power over asset returns. Of course, inference based solely on the subset of surviving trading rules may be misleading in this context since it does not account for the full set of initial trading rules, most of which are unlikely to have underperformed.

I have to decry some excesses in backtesting that I have closely witnessed in my private career. At the time of this account, Omega TradeStation offered a product designed for this, used by tens of thousands of traders. It even had its own computer language. Beset with insomnia, the computerized day traders become night testers plowing the data for some of its properties. By dint of throwing their monkeys on typewriters, without specifying what book they want their monkey to write, they will hit upon hypothetical gold somewhere. Many of them blindly believe in it.

One of my colleagues, a man with prestigious degrees, grew to believe in such a virtual world to the point of losing all sense of reality. Whether the modicum of common sense left in him might have rapidly vanished under the mounds of simulations, or whether he might have had none to engage in such pursuit, I cannot tell. By closely watching him I learned that what natural skepticism he may have had vanished under the weight of data—for he was extremely skeptical, but in the wrong area. Ah, Hume!