There are other reasons a backtest can fail live, such as trading costs like spreads, slippage and commissions that differ from the backtest (see counting your trading costs), and changes in how the market behaves. This article looks at one of them: overfitting.
What overfitting is
A backtest calculates what would have happened if you had traded a set of rules on past prices. If you keep changing the period, stop distance, indicator settings and so on, some versions will happen to fit the past prices very well.
Those settings may only fit the random quirks of that past period. If so, they may not do as well in another period or in live trading.
Every rule in the chart decides buy or sell by flipping a coin, so none has a real advantage. Even so, if you pick the best of 200, it looks like a "good rule" when you only look at the first half.
What research shows
A paper that studied backtest overfitting mathematically shows the following.[1]
- Even with only 10 tries: assuming no rule has a real edge (average result of zero), if you try 10 variations on one year of data, the best one is expected to show a Sharpe ratio (return divided by how much it swings) of 1.57. In the period not used to pick it, its expected result is still zero
- Data length and number of tries: under the same assumptions, with only 5 years of data, trying more than 45 independent variations makes it almost certain you will find a rule with a backtest Sharpe ratio of 1 (annualized) but an expected Sharpe ratio of zero in the unused period
- Even on random prices: in the paper's example, choosing the entry day, holding period, stop and buy-or-sell side for 1,000 days (about 4 years) of randomly generated prices produced a rule with a Sharpe ratio of 1.27
- Report the number of tries: the authors say that if you do not report how many variations you tried, the risk of overfitting cannot be judged
These results use a simplified model of prices, and real markets may not follow those assumptions. Still, it is worth remembering that trying many variations and picking the best, by itself, pushes results to look better.
How you can check in MT5
The MT5 Strategy Tester has "optimization", which runs a strategy many times with different settings, and "forward testing", which runs it again on a different period. The MT5 help says forward testing helps you avoid fitting settings to certain parts of the historical data. In the "Forward" tester setting, you can use half, a third or a quarter of the period for the forward test, or set the forward start date yourself.[2] Other testing tools also let you check on a separate period; the names and steps differ.
Even if the forward period looks good, it is one check, not a promise of future results. If you adjust your settings after seeing the forward results and test again, over and over, you start fitting to the forward period too.
What you can do (you decide)
This article does not decide which strategy or settings to use. When you look at backtest results, logging these makes overfitting easier to spot:
- Number of tries: how many settings or rules you tested
- Split periods: results for both the period used to pick and the period not used
- Number of trades: with few trades, results swing a lot by chance
- Live results: put results after going live next to the backtest
The trading stats calculator turns a profit and loss list into win rate, average P&L and longest losing streak. The losing streak calculator shows how often losing streaks happen even when your win rate stays the same. For mistakes in measuring that can make a backtest look profitable, such as leaving out costs or guessing the order of prices inside a bar, see why a backtest can show an edge that isn’t there.
Sources
- Bailey, D. H., Borwein, J. M., López de Prado, M., & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society, 61(5), 458. https://doi.org/10.1090/noti1105 (authors' copy: https://www.davidhbailey.com/dhbpapers/backtest-pseudo.pdf)
- MetaQuotes. Strategy Testing (MetaTrader 5 Help). https://www.metatrader5.com/en/terminal/help/algotrading/testing