Backtesting: The Curve Is the Seduction
Backtesting runs a set of rules over historical data to produce the equity curve those rules would have generated. The output is only as good as the assumptions about fills, costs and data quality, and six specific errors reliably make a backtest look better than the live result.
How it works
A backtest applies rules to past data and records the results. Entry, exit, size and cost are specified, the software walks the history, and an equity curve comes out.
The curve is persuasive out of proportion to its evidence. A rising line looks like proof, and it is really a statement about how the rules interacted with one particular history under a set of assumptions you chose.
The six errors
Sample size is the first and most common. Twenty trades is not enough to distinguish a method with an edge from one without; two hundred begins to be. Most published backtests run on far fewer.
Cost understatement is the second. On this site’s shared history the round trip is 2% of a median bar’s range and 45% of the smallest bar. A test applying half the real figure turns a losing method into a winning one on paper.
Fill assumptions are the third. A resting limit order fills in a backtest whenever price touches its level; in reality it joins a queue. A stop fills at its trigger; in reality it becomes a market order.
Lookahead bias is the fourth and the most damaging. Using a bar’s close to make a decision at that bar’s open, or a revised data point that was published later, produces results that were never available.
Survivorship bias is the fifth. A test on today’s index members excludes every company that was removed, which is precisely the set that would have hurt. And the sixth is overfitting — tuning parameters until the curve looks right, which deserves its own page.
In practice
Model costs before anything else. A method that survives a pessimistic cost assumption is worth testing further; one that only works at optimistic costs has already failed.
Market impact is assumed away entirely. In thin volume a real order moves price and a simulated one does not, which flatters every result taken in quiet conditions.
Slow methods are the hardest to test honestly. Two hundred daily-chart trades can require a decade of data, across regimes that no longer resemble each other.
Many engines fill a stop at its level across a gap. That single modelling error removes the largest risk in overnight trading from the record.
A stop needs slippage attached. Adding a fixed pessimistic amount to every stop fill is crude and far closer to reality than assuming none.
High-frequency results are the most fragile. The more trades a method takes, the more of its result is determined by the cost assumption rather than by the rules.
Test against the base rate
A backtest that beats zero has proved nothing. The comparison that matters is against what would have happened anyway, and on this site’s shared 576-bar history 52% of bars closed higher than the one before across 571 observations.
That number is the bar every entry rule has to clear. A pattern with a 55% next-bar hit rate sounds useful and is three percentage points above doing nothing at all — before costs, which on a per-bar edge that small will usually consume the entire difference. Compute the base rate first and put it beside every result; it converts most impressive-looking hit rates into unremarkable ones, which is exactly what it should do.
One structural habit prevents most of these errors at once: hold data back. Fit on the first two thirds of the history, then run once on the final third and accept whatever comes out. The reserved section is only useful while it remains untouched, so it can be used once — a second look at it turns it into part of the training set.
The discipline is harder than it sounds because a poor out-of-sample result invites one more adjustment. That adjustment is the moment the test stops being a test. Write down the result before changing anything, so the record shows what the method did rather than what the third version of it did.
What a backtest is not
It is not evidence of a future edge. It describes one history.
It is not a live result. Fills and costs are modelled, not paid.
It is not objective. Every assumption in it was chosen.
And it is not enough on its own. Forward testing is the check.
When it fails
A period dominated by one regime produces a misleading verdict either way. A trend method tested across a range looks broken; the same method tested across a trending year looks excellent. Neither is a property of the method.
The second failure is testing on the data you tuned on. The result is a description of the tuning.
A third is a survivorship-contaminated universe. The failures are the missing evidence.
A fourth is ignoring the sequence. Two curves with the same total return and different orderings are very different to hold.
And a fifth is stopping at the backtest. It is the cheap first filter, not the decision.
The original data
On this site’s shared 576-bar history the base rate for a higher close is 52% over one bar across 571
observations and 54% over ten across 566. A breakout above a 20-bar high closed back below the level within
ten bars in 85% of 39 events, and above a 55-bar high in 100% of 11. The round-trip cost is 2% of a median
bar’s range and 45% of the smallest bar. The figures are in research/series-measurements.json, produced
by site/measure_series.py.
The 39-event and 11-event breakout counts are the honest illustration of the sample problem. Those are all the 20-bar and 55-bar breakouts a 576-bar history contains, and no conclusion drawn from eleven events is worth acting on. Before reading any backtest result, find the number of trades behind it — if it is under a hundred, the curve is a story about a handful of events, however convincing the line looks.
Related
Forward testing is the check a backtest cannot perform on itself. Overfitting is the error that survives every other fix. And strategy tester covers the software and what its defaults assume.
The backtest that taught me most was one I threw away. It looked excellent until I realised my exit was using the bar’s closing price to decide something at the open. One line, and the whole result was fiction. I now write the data-availability check before I write the strategy.
— Michael Whitman
This page is educational, not financial advice. Test every idea on your own charts before risking money.