Strategy Tester: The Optimiser Lies
A strategy tester is the testing environment built into a trading platform, which replays a strategy over historical data and reports the result. It also runs an optimiser, which ranks parameter combinations. The top row is not an edge; only a later untouched period tells you anything.
How it works
A strategy tester is the testing environment built into a trading platform. It replays a strategy over stored history and produces a report and an equity curve — backtesting with a graphical interface. The term belongs to the MetaTrader platforms, whose tester also runs an optimiser.
Modelling quality is the first setting and it decides everything after it. The options run from the opening price of each bar, through interpolation inside bars, up to modelling every tick.
Only every tick resolves what happened inside a bar. A strategy with a stop loss and a target inside the same bar cannot be judged without it — the tester would simply assume which was touched first.
The optimiser and the forward test
The optimiser is the most dangerous button in it. It will search hundreds of parameter combinations and hand you the best one, and the best of hundreds of random results always looks like an edge.
A grid search will always find a winning setting, even in data with no edge in it. That is what searching means: it is a probability problem before it is a trading one.
So split the history and test forward on the half you hid. Optimise on the earlier period, then evaluate once on a later one the optimiser never saw — forward testing, run inside the tester.
Parameter stability is the other check, and it costs nothing. Read the rows either side of the winner: a lone peak is an accident, a broad plateau is a candidate.
And the headline number is the least useful line on the report. The drawdown, the trade count, the longest losing run and the spread of results carry the information.
In practice
Set the spread, commission and swap before any run. On many platforms the default bid-ask spread is not the one your account trades, and in forex it widens at the hours a fast strategy likes.
It cannot model a market too thin to fill you. The tester fills your order at the price it chose, whatever the volume on that bar was.
A longer sample beats a better optimisation every time. More history means more regimes, and surviving several of them beats being tuned perfectly to one year.
Weekend gaps are where its assumptions break. An opening gap means the market never traded at the levels in between, so a resting order was never fillable.
And it will fill a stop the market jumped straight past. The record then shows a controlled loss rather than the larger one you would have taken.
Costs are not a rounding error at short holding periods. On this site’s shared 576-bar history a round trip costs 2% of a median bar’s range, paid on every trade.
An honest run, in order
An honest run has a fixed order, and the order is the method. Set the costs to what your account actually charges, then set modelling quality to every tick and leave it there.
Split the history before you optimise, not after. Hold back the most recent third and do not look at it. Optimise on the older portion alone, and treat its table as a list of suspects.
Check stability next, because it is free. If a small change to a length or a threshold turns profit into loss, the setting is fitted to noise.
Then evaluate once on the untouched period, and once only. Re-optimising because the forward result disappointed turns that period into training data. Systematic trading lives on that discipline.
What a strategy tester is not
It is not a prediction. It describes one history under assumptions you chose.
It is not proof of an edge. The top row of an optimisation table is a selection, not a finding.
It is not a live account. Slippage, requotes, outages and partial fills all sit outside it.
And it is not automated trading. Running an expert advisor in the tester is rehearsal, not trading.
When it fails
In a trading range the optimiser simply finds a range setting. Every parameter it picks is tuned to whichever regime dominated the sample, so the verdict describes the period, not the strategy.
The second failure is a short history. A few hundred bars holds too few independent events to mean much, and the optimiser will fit all of them.
A third is the broker’s own price data. The tester uses whatever history the platform supplies — short, patched across feeds, or quoted on a spread nobody traded.
A fourth is a strategy that trades rarely. Twenty trades cannot separate method from luck, and optimising over twenty trades is fitting noise with a progress bar.
A fifth is re-running it after seeing the forward result. That single act converts test data into training data, and nothing short of new history undoes it.
And the sixth is generous fill assumptions. Limits that fill on a touch, stops that fill at their level and no slippage turn a losing method into a profitable one on paper.
The original data
Nine videos carry “strategy tester” in a title, at a median of 52,868 views across four channels and a maximum of 228,801. “Expert advisor” returns nine at a median of 11,637, “backtest” returns 812 at a median of 2,115 across 400 channels, “overfitting” two at 299, and “walk forward” one, at 3,796.
Those counts come from research/broker-coverage.json, a scan of the 31,760 trading and investing
videos in research/search-study-corpus.jsonl. The cost figures are in
research/series-measurements.json, built by site/measure_series.py: on the shared 576-bar history a
round trip costs 0.0098 price units, which is 2% of a median bar’s range and 45% of the smallest.
That median is the highest of any testing-related term here, roughly twenty-five times the 812 backtesting videos. Very few people teach the tester and a great many want it taught. One video in the whole corpus covers walk-forward testing, the technique that makes an optimiser result mean anything.
The cost point is smaller on the page than it is in practice. The spread is one field in a dialogue box, and here a round trip is 45% of the smallest bar, so leaving it on a default decides the run. Set the real costs before the first run, and judge a strategy by the untouched period, never by the best row in the table.
Related
Backtesting software is the wider category, and a dedicated package gives more control over data quality and costs. Overfitting is the failure the optimiser produces by construction, which is why the top row means so little. An expert advisor is what the MetaTrader tester runs, so its code limits what a report can show.
I once left an optimisation running overnight and woke up to a row that looked like the answer to everything. It fell apart on the months I had held back, and the settings either side of it lost money. What I had found was the luckiest combination in the search, not a rule. Now I look at the neighbours of a result before I look at the result.
— Michael Whitman
This page is educational, not financial advice. Test every idea on your own charts before risking money.