Short version: We've changed the benchmark used in backtesting. When you re-run an algorithm through the backtester, you should expect different results for the benchmark and risk calculations.
Long version: A few months ago it was pointed out in this thread that we were using a price-based benchmark rather than a returns-based benchmark. When we thought about it, it was pretty clear we'd chosen the wrong one. If you're comparing your algorithm to a benchmark, you need to look at the returns of that benchmark. This evening we changed the benchmark being used in the backtester.
The effect of this is that most backtest results will look a little different than they did before. The benchmark will look better, which means the algorithm results won't look comparatively as good, and the risk metrics driven by the benchmark will similarly be different. For short backtests, or backtests that didn't include a dividend date, there won't be a change. For an 11-year test, the difference is pretty significant.
We know that having a stable backtester is important, and having an accurate backtester is also important. This is one of those times where the goals are in conflict, and being correct is more important than being consistent. We work very hard to make changes like this as unusual as possible. We're sorry for any inconvenience this change causes.
Example: The backtest below buys a bunch of SPY and holds it. You see the returns match pretty closely. The tiny differences in the returns are driven by the real-world effects of a strategy v. a benchmark.
- The benchmark is assumed to be 100% invested at the market open, while the algorithm has to place an order in the first bar and be filled in the second bar.
- The algorithm can't be 100% invested because it can't hold partial shares. It always has a small cash position, and that cash has 0% return.
Still to Come: One of our highly-requested features is to make the benchmark customizeable. This change we shipped today gets us a long way towards that - when we put in the new benchmark, we laid the foundation for future changes on the fly. I don't have a delivery target yet but I'm hoping we can finish it sooner than later.
Edit 31-Jan-14: Shipped another version of the benchmark - benchmark now even closer to the fully-invested SPY return.
|Returns||1 Month||3 Month||6 Month||12 Month|
|Alpha||1 Month||3 Month||6 Month||12 Month|
|Beta||1 Month||3 Month||6 Month||12 Month|
|Sharpe||1 Month||3 Month||6 Month||12 Month|
|Sortino||1 Month||3 Month||6 Month||12 Month|
|Volatility||1 Month||3 Month||6 Month||12 Month|
|Max Drawdown||1 Month||3 Month||6 Month||12 Month|
def initialize(context): context.spy = sid(8554) def handle_data(context, data): # put all my money in SPY. rebalance as dividends come in. order_target_percent(context.spy,1.0) if context.portfolio.cash < 10000: record(cash=context.portfolio.cash)