Try Enough Rules and One Will Look Brilliant

Give twenty fair coins thirty flips each. One coin lands heads 20 or more times only about 5% of the time, but the BEST of the twenty typically lands heads about 20 times: a 67% 'win rate'. Nothing about that coin is special; it is the luckiest of twenty. Picking it after the fact is how most impressive backtests are found.

Nobody sets out to flip twenty coins. They test RSI below 30, then below 25, then a 10-day RSI, then add a trend filter, then move the stop from 5% to 7%. Each tweak is another coin. After twenty of them, the best version is mostly a record of which tweak was luckiest on this stock in these years. This is overfitting (or curve fitting): the rule now describes the past instead of anything that repeats.

Statisticians often call a result 'significant' when its t-statistic, a measure of how far it sits from pure chance, is above 2: about a 1-in-20 fluke for ONE test. Try many ideas and the best one gets close to that bar by luck alone. What pure noise typically produces as its best result:

At twenty tries, luck alone gets most of the way to 'significant'. This platform applied the same correction to its own research, and its once-promising results did not survive it.

Decide the rule BEFORE you see results, and write it down. Count every version you tried, including the ones you threw away. Prefer ordinary settings (a 14-day RSI, a 50-day average) to ones that happen to score best. Then test the finished rule on stocks and years it was not tuned on: a rule that only works where it was tuned is a description of the past.

The Scanner makes tweaking easy, which makes this trap easy too. A useful habit: take one of its strategy presets, change nothing, and backtest it on five stocks before touching a setting. If you then tune it, judge the tuned version on five different stocks.