Quantitative Methods & RiskBacktesting Hygiene: Why Most Backtests Lie
Overfitting & the Multiple-Testing Trap
The backtest that fit the noise
A backtest is a hypothesis test run on historical data, and like any hypothesis test it can be fooled — most often by the experimenter, not the market. Overfitting is the act of tuning a strategy so tightly to the past that it captures the random noise in that particular history rather than any durable structure. The fitted strategy looks spectacular in-sample and collapses the moment it meets new data, because the noise it learned does not repeat.
Overfitting
Fitting a model so closely to a specific dataset that it captures noise rather than signal. In trading, it shows up as a backtest with a beautiful equity curve and many tuned parameters that degrades sharply out-of-sample. The more parameters and the more variants you try, the more certainly you overfit.
The more you search, the better the lie looks
If you test enough strategy variants on the same data, one WILL look excellent by pure chance — this is the multiple-testing problem from Module 2, applied with a vengeance. Backtest 1,000 parameter combinations and the best of them is selected precisely for its luck. The reported Sharpe of the 'winner' is biased upward by the very act of choosing the best of many. The fix is not a better optimizer; it is fewer degrees of freedom and honest out-of-sample testing.
Why the best-of-N backtest is biased up
E[ max of N noisy Sharpes ] grows with N even when the TRUE Sharpe is 0
A
Signs you have overfit
- Many free parameters relative to trades
- Performance very sensitive to small parameter changes
- Beautiful in-sample, weak out-of-sample
- Complex rules with special-case carve-outs
- The strategy was the best of hundreds you tried
B
Defenses
- Keep parameters few; prefer simple, economically-motivated rules
- Hold out a true out-of-sample period you never look at while developing
- Use walk-forward, not a single train/test split
- Track how many variants you tried (the 'deflated' Sharpe)
- Demand robustness across nearby parameters, not a single sweet spot
Parameter robustness beats parameter optimization
A strategy whose performance is a broad plateau across a range of parameters is far more trustworthy than one with a single sharp peak. A sharp peak means tiny parameter changes break it — a hallmark of fitting noise. When you sweep a parameter, you want a gentle hill, not a needle.
Knowledge check
You optimize 500 parameter combinations and the best shows a 3.5 Sharpe in-sample. The most likely explanation is: