Quantitative Methods & RiskExpectancy, Edge & Sample Size

Variance, Sample Size & the Law of Small Numbers

14 min read

A positive expectancy still loses sometimes — for a long time

Expectancy tells you the long-run average. It says nothing about the road there, and the road is paved with variance. A strategy with a genuine edge can — and routinely does — produce losing months, even losing years, purely from the randomness of a finite sample. The flip side is more dangerous: a strategy with no edge at all can produce a long, convincing winning streak by luck. Telling these apart requires understanding sample size.

Law of small numbers

Term

The cognitive error of expecting small samples to reflect the true probabilities as reliably as large samples do. A trader who concludes after 15 winning trades that the strategy is 'proven' has fallen for it. Small samples are dominated by noise; the true expectancy only emerges over hundreds of trades.

Formula

Standard error of the mean

SE = σ / √n

Heads up

How long a drawdown can a real edge survive?

Even a strategy with a Sharpe of 1.0 — a genuinely good number — has roughly a 1-in-6 chance of being underwater over any given year, and multi-year drawdowns are entirely normal. If you abandon a positive-expectancy system during a statistically ordinary drawdown, you converted a winning edge into a realized loss. Variance, not the edge, is what tests your discipline.

Formula

Probability of a losing streak

P(k consecutive losses) = (1 − W)ᵏ

is_the_edge_real.py

python

1import numpy as np
2 
3def t_stat(per_trade_pnl):
4    """How many standard errors is the mean edge above zero?"""
5    x = np.asarray(per_trade_pnl, dtype=float)
6    n = len(x)
7    se = x.std(ddof=1) / np.sqrt(n)
8    return x.mean() / se, n
9 
10t, n = t_stat(my_trade_results)
11print(f"n = {n} trades, t = {t:.2f}")
12# |t| < 2  -> result is consistent with NO edge (noise)
13# |t| > 2  -> edge is ~2 standard errors from zero, but beware
14#            multiple-testing: if you tried 20 strategies, one will
15#            cross t=2 by chance alone.

A t-statistic above ~2 is the usual rough threshold for 'distinguishable from zero,' but it is necessary, not sufficient — and it is meaningless if you cherry-picked the best of many strategies. Sample size and multiple-testing both bite here.

Note

Multiple testing: the more you try, the more you fool yourself

If you test 20 random strategies with zero true edge, on average one will look 'significant' at the 5% level by pure chance. Backtest 1,000 parameter combinations and dozens will look brilliant. This is the statistical root of overfitting, and Module 5 is devoted to defending against it.

Equity curves: same +edge, 80 random paths80 paths · 250 steps · μ=0.0006 σ=0.014

Terminal distribution

Eighty simulated one-year equity paths from the SAME positive-expectancy process. Some finish far ahead, some finish underwater — all from identical odds. A single realized equity curve tells you very little; the distribution of paths is the real picture. This is why one good year proves almost nothing.

Knowledge check

Your strategy has a true positive edge. After 20 trades it is down money. The statistically literate response is: