Survivors and Hindsight: What a Chart Doesn't Tell You

The stocks you think to test are the ones you have heard of, largely because they did well. Today's S&P 500 lists the companies that grew into it, not the ones that shrank out of it, were bought or went bankrupt along the way. A buy rule tested only on today's famous names is tested on survivors, and almost any buy rule looks good on stocks that went up.

A backtest can use information before it existed. Sometimes that is a bug; more often it is ordinary: economic figures that were revised later, a number published weeks after the period it describes (the Economy page notes that unemployment arrives about five weeks late), or a trade at a price you could only have known after the market closed.

The Scanner's Backtest tab checks your entry and exit rules at each bar's close and trades at that same close. (Stops and targets are different: they rest like real orders and fill inside the day.) In real life you see a rule's signal after the close and trade at the next open, which after news can be several percent away. Over many trades in a slow rule that barely matters; for a rule built on big one-day moves it can be most of the result.

The subtlest version is human. You open a chart, see a stock that doubled, and test a breakout rule on it. The test is now about that one stock's past, chosen BECAUSE of it. Choose the stocks first, before looking (a whole list, or a few picked at random from one), then run the rule on all of them.

Would I have chosen these stocks back then? Was every number available on the day it was used? Did the test trade at a price I could actually have got? A 'no' to any of them doesn't make the result wrong. It makes it flattered, by an amount you don't know.