Quantitative Methods & RiskBacktesting Hygiene: Why Most Backtests Lie

Walk-Forward & Out-of-Sample Discipline

13 min read

If you only have in-sample, you have nothing

The single most common mistake in retail backtesting is reporting in-sample performance — performance on the very data the strategy was tuned on. It is almost meaningless, because the strategy was selected to look good on exactly that data. Out-of-sample testing evaluates on data the strategy never saw during development. Walk-forward analysis systematizes this: fit on a rolling window, test on the immediately following window, roll forward, and report only the stitched-together out-of-sample results.

Walk-forward analysis

1
Split into rolling windows
Choose a fit (in-sample) window and a test (out-of-sample) window — e.g. fit 1 year, test the next quarter.
2
Fit on the in-sample window
Estimate parameters using ONLY the in-sample data. No peeking at the test window.
3
Evaluate on the out-of-sample window
Apply the frozen parameters to the immediately following, unseen window and record the result.
4
Roll forward and repeat
Advance both windows by the test length and refit. The strategy re-adapts as a real one would.
5
Aggregate the OOS pieces
Concatenate the out-of-sample results into one equity curve. THAT is the number you report — never the in-sample fit.

walk_forward.py

python

1def walk_forward(df, fit_window=252, test_window=63):
2    results = []
3    step = test_window
4    last = len(df) - fit_window - test_window
5    for start in range(0, last + 1, step):
6        fit  = df.iloc[start : start + fit_window]
7        test = df.iloc[start + fit_window : start + fit_window + test_window]
8        params = fit_strategy(fit)          # tune on the PAST only
9        results.append(evaluate(test, params))  # judge on the FUTURE
10    return aggregate(results)               # report the OOS stitch

A 1-year fit / 1-quarter test rolling window is a common starting point for daily-bar strategies. The parameters are always fit on data strictly before the window they are judged on.

In-sample only (worthless)

Strategy tuned and judged on the same data
Reflects how well you fit the past, not the future
Sharpe biased upward by selection
Says nothing about live behavior

Walk-forward out-of-sample

Parameters frozen before each test window
Mimics how a real strategy adapts over time
Aggregated OOS curve is the honest estimate
Still vulnerable to look-ahead in the feature pipeline

Heads up

Even walk-forward can be cheated

Walk-forward is necessary, not sufficient. If you run many walk-forward configurations and keep the best, you have re-introduced overfitting at a higher level. If your features secretly use future data, every window is contaminated. And if you tweak the strategy after seeing the out-of-sample result, that result is now in-sample. The discipline is to look at the true hold-out exactly once.

Tip

The locked drawer

Carve off the most recent slice of history — say the last 1–2 years — and lock it away. Develop, tune, and walk-forward on everything before it. Only when you are fully done do you run the strategy once on the locked slice. If it holds up, you have a real out-of-sample read. If you peek early or iterate against it, you have burned it and must find fresh data.

Knowledge check

Why must the result you report from a backtest come from out-of-sample windows rather than the in-sample fit?