I said I'd post some of the links to other issues I've raised about the backtester by this morning so here they are:
The data feed that Quantopian uses is compiled from all trades within all exchanges. For example, XLY trades on AMEX in New York, Chicago, etc. The exchange reports an "official" closing price but this may not necessarily be the very last price of the day. As a result, there are discrepancies between Quantopian's datafeed and, for example, IQ Feed (a popular subscriber datafeed for algo trading) or Yahoo Finance. The caveats here are 1) small differences in cost basis between backtesters and 2) small differences in momentum values for given securities - https://quantopian.com/posts/problems-with-data-feeds-prices
The benchmark is purportedly an investment in SPY. However, if you generate an algo to actually buy and hold SPY it will always outperform. The reason is that the benchmark is the price return of SPY and does not take into account dividend distributions. The caveat here is that you will frequently outperform the benchmark but in reality will underperform SPY. - https://quantopian.com/posts/question-regarding-the-benchmark
I had an issue with forward filling in fetcher that caused some false buy signals. The caveat here is to make sure that you have data for each ticker and for each day of the backtest. Otherwise the most recent signal is forward filled. That sort of forward filling may be desirable depending on the algo. - https://quantopian.com/posts/help-with-fetcher-and-filling-nan-values
I occasionally get runtime errors that I can not explain. These frequently occur with the internal functions available. Because the compiler catches the error and produces and error message, there is no false data generated but this does present an obstacle to generating certain algos. - https://quantopian.com/posts/runtime-error-with-vwap
I have derived and implemented a basic sector rotation strategy within my firm that basically rotates between the sector SPDRs every quarter. It is not open source at the moment but we may open it up at some point. I have posted the "rankings" for the SPDRs from 2001 to 2013 in a csv and read them using Quantopian. The transactions in Quantopian appear to be consist witht he transactions I observe in my backtester (the one I coded in extended basic) but the final balance is different. The difference was initially substantial but I am finding small differences between assumptions made in each backtester and, as I resolve those differences, the answers appear to converge. I am still in the process of working on it. - https://quantopian.com/posts/sector-rotation-strategy-dot-dot-dot
Overall, there aren't necessarily errors in either Quantopian or in my in-house backtester (I'd provide my backtester but it is NOT user friendly at all). However, I have found that certain assumptions made within these backtesters have an impact on the final result, which can become compounded over a long backtest. I'm still trying to figure out all the nuance behind both backtesters, understand the caveats of the results generated by each, and work on providing the best asset management given all the tools I have.