One of the downsides of back testing is that it can lure the trader into overconfidence about systems and the way they have performed in the past.
There is always a real danger of curve fitting and data mining to find a perfect system that would have worked with an exact set of market conditions in the past that can never occur again in the future. This is the problem of over reliance upon the power of computation. It neglects the reality of the market as a complex adaptive system which never shows you the same face twice.
Even if a backtest is performed rigorously and with full knowledge of the limits of its ability to forecast into the future, it is not unusual to see a large discrepancy between backtest results and actual results from live trading.
Some of the causes of this discrepancy in results have to do with some backtests being conducted in isolation and not as part of a full portfolio of strategies. Often backtests may use unrealistic values for slippage and commissions. Many times they will not take into account the human dimension of executing a set of trading rules. Experience shows that this is one of the most important aspects overlooked by most back testing.
Perhaps the biggest challenge with back testing is understanding the limits of its forecasting power. Some traders place so much reliance on their back testing that they cannot be convinced by evidence of actual results that they have missed something important in their backtest. They might persist in trading a system that will simply not work in the real world. This can be attributed to the overconfidence bias combined with the need to be right.
Professional engineers and doctors are especially prone to this problem because of their belief systems and testing from their previous professions. Those professions place a premium on being right to be successful and yet in the trading profession, it is the ability to act with incomplete knowledge and a willingness to be wrong that often leads to the best results.
While back testing offers many powerful advantages to the professional trader, it must be undertaken with full recognition of the limits of its usefulness as well. The professional trader will take back testing results into consideration, as a way to select a system to prototype with real money, in real markets with the human factors fully engaged to see what the real world results look like before committing to full production system risk.