Overfitting: Describing the Past
Overfitting is a result that fits the particular history it was tested on, including that history's random noise, rather than any repeatable relationship. Each adjustable parameter, and each variant tried, adds another way to fit the past. A fitted system did not stop working; it never worked.
How it works
Overfitting is a result that describes the past and nothing else. A system is tuned until it fits one stretch of history, noise included, rather than anything repeatable. It never worked; it did not stop working.
Every parameter is a dial, and dials get turned. A length, a threshold, a stop loss distance: each is a degree of freedom, another way to bend a system around its data.
Turn enough of them and the old curve looks perfect. With enough degrees of freedom and enough attempts, some combination fits almost any dataset, including pure noise.
Because you fitted the noise, not the signal. Price history holds a little structure and a great deal of accident. Optimisation cannot tell them apart.
Which is why it dies the moment it meets new data. The accidents do not recur, so the fitted trading rules have nothing to work on. The drawdown arrives at once.
Counting the ways you can fit
Three parameters are far harder to fit than twelve. The number of dials sets how closely a history can be traced. Fewer means a worse tested result and a more believable one.
And a rule added to fix a bad year is the classic case. It always improves the backtesting result, because it was derived from the outcome it removes. It rarely improves the future.
Costs left out of a test hide the problem completely. Omitted commission, look-ahead bias (using information the moment did not have) and survivorship bias (testing only the survivors) all flatter the result.
More inputs mean more ways to fit the same history. Adding volume filters, correlation screens or session windows feels like rigour. Each is another dial.
A slow system has too few trades to judge at all. A rule trading twice a year yields a few outcomes per decade, too few to separate skill from luck in any backtesting software.
In practice
And a backtest fills a gapped stop that never filled. An opening gap jumps past the level, yet the strategy tester books the exit at the price you asked for.
Move the stop until the test improves and you have done it. Stop distance is a dial like any other. Sliding it through twenty values and keeping the best is twenty tests.
Every round trip costs 2% of a bar. A round trip on this site’s shared 576-bar history is 0.0098 price units: 2% of a median bar’s range and 45% of the smallest bar.
The tests that help are the ones you cannot tune. Hold a block of data back and never touch it. Run forward testing on unseen periods, and prefer fewer parameters.
Stability is the most useful single check. A system that works at one setting and fails at the settings either side of it has found an accident. A genuine relationship degrades gradually.
Trades per parameter
The arithmetic is simple and unforgiving. Every parameter you fit consumes information from the sample it was fitted on, so the real question is how much sample there is per dial. Sixty trades and twelve parameters leaves five outcomes behind each decision.
Five outcomes cannot separate an effect from a run of luck. The same twelve parameters against several thousand trades is a different proposition, because no single accident can move that result far.
Treat the ratio as a gate before you read the result. Count the trades, count the dials, and count the variants you tried. Where that figure is small, the number in front of you is work for probability, not intuition.
It is the quiet cause behind much of why traders lose money. The sample was always too thin for the confidence placed on it.
What overfitting is not
- Not a system that decayed. A decayed system worked first; a fitted one never did.
- Not simply a losing system. Plenty of honest systems lose without any fitting involved.
- Not the same as optimisation. Choosing sensible values is normal; searching until the past is perfect is not.
- Not fixed by more history. More data helps only if you stop tuning against it.
When it fails
The diagnosis can be wrong in both directions. These are the cases where the label misleads:
- A quiet trading range punishes a sound system. A flat period flattens an honest edge too, and calling that overfitting throws away something that works.
- A short live sample proves nothing either. Twenty live trades cannot confirm a failure any more than sixty backtested trades confirmed a success.
- Fitting hides inside simple systems. Two parameters chosen after four hundred attempts is a heavily searched result wearing a modest disguise.
- Held-back data stops being held back once you use it. Look at your reserved block a second time and it has quietly joined the training sample.
- Some markets genuinely change. Rule changes, new venues and shifting participants can end a real edge, and that is decay rather than fitting.
- A result can be honest and still useless. A stable, few-parameter system with a thin edge is not fitted; it simply does not survive costs.
The original data
The scan is in research/broker-coverage.json. Across the 31,760 trading and investing videos in
research/search-study-corpus.jsonl, 812 titles mention “backtest” and 2 mention “overfitting”. “Walk
forward” appears once and “forward test” three times, at a median of 683 views. “Expectancy”, “risk of
ruin” and “monte carlo” appear zero times each.
The audience numbers widen the gap. Those two overfitting videos come from 2 channels, median 299 views, maximum 515, against a median of 11,527 views for the 211 videos about win rate.
So the failure that invalidates a backtest is covered in roughly one video in four hundred. The published curriculum teaches how to produce a backtest, and almost nothing about how one lies to you.
research/series-measurements.json shows what that costs you. On this site’s 576-bar history a
breakout above a 20-bar high closed back below the level within ten bars in 85% of 39 events, against a
54% base rate across 566 observations.
Thirty-nine events is already thin, and a system tuned on a sample that thin is describing 39 accidents. Before your next optimisation run, write the test down first, record how many variants you tried, and reserve a block of data you will look at exactly once.
Related
Backtesting is the process this failure contaminates, which is why a strong tested result deserves suspicion. Forward testing runs on data your tuning could not reach, which is what makes it worth more than any in-sample number. Systematic trading is where the problem lives, because no discretionary trader runs two hundred variants overnight.
I had a system that tested beautifully across years of history, and I kept tuning it until every ugly stretch of the equity curve smoothed out. It went live and behaved like a different system entirely, because it was. What I had actually built was a very detailed description of the years I happened to test on. The tuning was the problem, not the fix.
— Michael Whitman
This page is educational, not financial advice. Test every idea on your own charts before risking money.