Back to Community
[BUG REPORT] Large errors in the backtesting system

I created a simple buy and hold trade to test the robustness of the current backtesting system.

From 2017-05-30 to 2017-10-20 with $120,761.00 initial capital.
500 shares of SPY at $241.52 were bought at the market close on 2017-05-30. No more trades afterwards.
The position value on 2017-05-30 was 500*$241.52 = $120,760.00, with less than $1.00 cash value.

I supposed the cumulative performance should be very similar to benchmark SPY, since I have already eliminated most of the possible error terms. Thus, a discrepancy between the actual performance and benchmark is expected be to less than 0.05% which could be reasonable. However, there was a big jump with larger than 0.5% errors between these two from someday in June then disappeared in August, then happened again on someday in September----in the context that the SPY itself had only 0.0%-2.5% returns during this period, and the 0.5% error could be HUGE!

I believe that the benchmark SPY used the daily adjusted closing price eliminating the influences of dividends, and I assume such discrepancy is not due to the after-hours or pre-market price fluctuation.

Some other user had similar issues as follows, there is no official response though.
https://www.quantopian.com/posts/difference-between-spy-and-benchmark

So...I would wonder where does this error come from. If it is due to some specific methods calculating the performance data, I would be happy to know the math behind it; it there is a bug, then the performance numbers for all the products or algorithm based on this backtesting system might be doubtful, in consequence the parameter optimization and so on would be further influenced and unreliable.

Thanks,
Tiger.S.

Clone Algorithm
4
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5a10e16f0ecc844200a312e6
There was a runtime error.
3 responses

tiger,

This may help you see the difference between EX-DATE and PAYABLE DATE.

Thanks Vladimir for the explanation !

I have identified the source of the errors---
For benchmark, the backtest system always uses the ADJUSTED closing price;
For the algorithm backtesting, it uses the ADJUSTED closing price, but EXCEPT the period between EX-DATE and PAYABLE-DATE (usually 10 - 50 days). During such period, the cash value in the account was not accrued or did not have compensation of the dividend amounts in advance to match up the true lookahead net capital value.

If the algorithm bought the equity before the EX-DATE, and 'unfortunately' sold it during the days between EX-DATE and PAYABLE-DATE, the backtest will show PERMANENT LOSS of the dividend value ! This could considered as a Bug that influences any algorithm that holds ETF or stocks which have many times of dividend or large amounts of dividend.

For example, Microsoft (MSFT) had dividend of $3.08 in 2004 Q4, which was traded in the $28-$30 range during that time, the EX-DATE and PAYABLE-DATE was 2004-11-17 and 2004-12-02, let's see what would happen if the algorithm bought MSFT before the EX-DATE and hold it through this period. As shown in the attached backtest (Buy&Hold MSFT vs. Benchmark MSFT), there was a huge loss of 10% suddenly after the EX-DATE, which might trigger StopLoss orders for many algorithms.

While this bug exists, many algorithms will be appeared to be underperformed and unstable due to the sudden change or jump in the backtest. There are two suggestions could be considered: (1) ALWAYS uses ADJUSTED price for the algorithm as the same as the benchmark; (2) During the days between EX-DATE and PAYABLE-DATE, uses UNADJUSTED price with compensation for the cash value in the amount of dividend values, but this solution may also trigger StopLoss orders for some algorithms.

Quantopian should take notice of this issue, or the backtesting system might give inaccurate numbers calculating many parameters like performance, sharp ratio, volatility, etc., and further mislead the users when they take effort to design and optimize the algorithms.

Thanks.
Tiger. S.

Clone Algorithm
4
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5a11d45291035d41c677e09c
There was a runtime error.

Hi Tiger -

This is behaving as expected. I'll walk you through some of the design decisions and let's see what you think. Also check out the help section on dividends which covers this in additional detail.

I added some record() statements to your algorithm to add custom variables to the graph. You can see that cash starts at $2974, drops to $.90, and then up to $300.90 when the dividend is paid out. Then, look at the price and returns lines (Tip: if you click "cash" in the legend, it's a toggle that removes the cash line and lets you zoom into the remaining custom lines). In the price you can see it drop by almost $3 on the dividend date. But the percent change line doesn't show a 10% drop - the returns for that day are about 1.5%.

Unpacking what that means:

  • The cash of the dividend is added back to the portfolio on payment date (the cash jump)
  • The simulation is always using the as-traded price of MSFT. That's why we see the price of MSFT drop.
  • The simulation is always using fully adjusted historical prices. That's why even though the MSFT price dropped, the percentage change from the previous day doesn't reflect the price drop - it's using an adjusted price to do the calculation.

If you have built a sensible stop-loss for your MSFT position, it isn't going to be triggered. The MSFT price changes are fully adjusted.

I think what is throwing you off is that your day-to-day portfolio returns are dropping temporarily. The good news is that the simulation is correctly telling you what your cash position is. The bad news is the non-intuitive drop in portfolio value. I think a future improvement might be to include some accounting for a pending dividend payment. You point about calculations like Sharpe being affected by this portfolio value change are valid. In most cases the effect of dividends is negligible; this example is the exception to that.

As for the benchmark, it does indeed use adjusted prices. The backtester is simulating your portfolio, with all the ugly sausage making of cash payments and slippage costs. The benchmark is idealized and hypothetical - it is not constrained by transaction costs or liquidity or other real-world constraints.

Clone Algorithm
4
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5a146a80b39f32407f68eb87
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.