Quantitative Methods & RiskBacktesting Hygiene: Why Most Backtests Lie

Look-Ahead & Survivorship Bias

13 min read

Two biases that fabricate edge from thin air

Beyond overfitting, two structural biases quietly inflate backtests, and both can survive even a careful walk-forward if the data pipeline is wrong. Look-ahead bias uses information that would not have been available at the moment of the decision. Survivorship bias tests only on the assets that survived to today, silently deleting the ones that went to zero. Each can turn a worthless strategy into a backtest that looks like a money machine.

Look-ahead bias

Term

Using data in a backtest that would not have been known at the decision time. Classic forms: trading on a closing price you couldn't have had until after the close, computing an indicator over the full sample (including the future), or using restated/revised data that wasn't available in real time.

Survivorship bias

Term

Testing only on assets that exist today, having dropped those that were delisted, defaulted, or went bankrupt. A stock-picking backtest run on the current index constituents never 'buys' the companies that went to zero — making any strategy look better than it would have been live.

Heads up

Look-ahead hides in the most innocent feature

The deadliest look-ahead is subtle: a z-score normalized over the WHOLE dataset (using future mean and standard deviation), a 'next-day return' joined on the wrong date, a universe built from today's index members (which quietly drops the names that later failed), a fundamental data point timestamped at its reporting date rather than its release date. Audit every feature for the exact moment each input was actually available. If you can't prove a value was knowable at decision time, it wasn't.

Bias	How it sneaks in	How it inflates results	Guard
Look-ahead	Indicator/normalization computed over full sample	Uses future info as if known	Compute features causally, point-in-time only
Survivorship	Universe = today's surviving assets	Deletes the losers that went to zero	Use a point-in-time, delisting-inclusive universe
Data restatement	Revised fundamentals dated to period end	Trades on numbers not yet released	Use as-released vintages with real release timestamps
Selection / p-hacking	Report only the best run	Best-of-N noise looks like edge	Pre-register the test; count the variants tried

Example

A concrete survivorship trap

Backtest 'buy the 10 worst-performing S&P 500 stocks each year and hold' using the CURRENT 500 members. It looks like a brilliant mean-reversion strategy — because every company that actually went bankrupt after a bad year was removed from the index and never appears in your universe. Re-run it on the point-in-time membership, including delisted names, and the apparent edge can vanish or reverse entirely.

causal_feature.py

python

1# WRONG: normalizes using the whole series (future included)
2z_bad = (price - price.mean()) / price.std()
3 
4# RIGHT: rolling, point-in-time only — uses data up to t-1
5roll = price.shift(1).rolling(252)        # exclude today, look back 1y
6z_ok = (price.shift(1) - roll.mean()) / roll.std()
7 
8# Rule of thumb: a feature used to decide a trade at time t may only
9# depend on information with a timestamp strictly < t.

The shift(1) and rolling window enforce causality: the signal at time t sees only the past. A single missing shift is enough to fabricate an edge that evaporates live.

Knowledge check

A backtest of 'buy beaten-down stocks' uses the index's CURRENT members over the last 20 years and shows a strong edge. The most likely culprit is: