The ways a backtest lies

Most backtests that fail live were not unlucky — they were biased. These pages catalogue the systematic errors that make a backtest look better than the strategy really is: using information you would not have had, testing on a survivor-only universe, torturing the data until something fits, and mistaking noise for signal. Each page explains why the bias happens and the concrete disciplines that reduce it.

Backtesting Biases: Backtesting biases are systematic errors that make simulated performance look better than a strategy's real edge. The major ones are look-ahead bias (using future information), survivorship bias (testing only on instruments that survived), selection and sample bias (an unrepresentative universe or period), data snooping and curve fitting (searching until something fits by chance), overfitting (modelling noise) and its opposite underfitting, and confirmation bias (seeing the result you want). Each is reduced by point-in-time data, out-of-sample discipline and honest process.

Look-Ahead Bias

Bias

Look-ahead bias is the error of letting a backtest use information that would not actually have been known at the moment a decision was made, which m…

→

Survivorship Bias

Bias

Survivorship bias is the distortion that arises when a backtest uses only the assets that survived to the present, silently excluding those that were…

→

Selection Bias

Bias

Selection bias is the distortion that arises when the assets, periods or trades used in a backtest are chosen in a way that is not representative of …

→

Data Snooping (Data Dredging)

Bias

Data snooping, also called data dredging, is the practice of trying many strategies, parameters or variants on the same data and reporting the best o…

→

Curve Fitting

Bias

Curve fitting is the practice of tuning a strategy's rules and parameters so closely to a specific historical dataset that it captures the noise of t…

→

Overfitting

Bias

Overfitting is the condition in which a strategy or model captures the noise of its training data instead of the underlying signal, so it performs su…

→

Underfitting

Bias

Underfitting is the condition in which a strategy or model is too simple or too constrained to capture the genuine structure in the data, so it perfo…

→

Sample Bias

Bias

Sample bias is the distortion that arises when the historical data used to build and test a strategy is not representative of the conditions the stra…

→

Confirmation Bias

Bias

Confirmation bias is the human tendency to seek, favour and remember evidence that supports a belief you already hold, which in backtesting leads res…

→

Frequently asked questions

What are the main backtesting biases?

The main ones are look-ahead bias, survivorship bias, selection bias, sample bias, data snooping, curve fitting, overfitting, underfitting and confirmation bias. Most share a root cause: information or choices that would not have been available or made at the time leak into the test, inflating apparent performance.

What is the difference between overfitting and curve fitting?

They are closely related. Curve fitting usually refers to tuning a strategy's parameters until it fits the historical price curve almost perfectly. Overfitting is the general statistical result: a model that captures the noise of the sample rather than the signal, so it performs well in-sample and poorly out-of-sample. Curve fitting is a common way to overfit.

How do I reduce look-ahead bias?

Use point-in-time data that reflects only what was known at each moment, lag every signal so a decision uses only already-closed bars, never fill an order at a price your rule needed to compute after the fact, and account for reporting and adjustment delays. Event-driven backtests that process one bar at a time make look-ahead far harder to introduce than vectorised ones.