Back to Community
Nondeterministic backtesting

Has anyone else encountered a situation when identical code yields different backtest results? It's just a simple Pipeline algo in my case.

12 responses

It seems that the Q500US filter returns different assets in different runs, maybe someone from the Q team knows why?

Hello Albert!

That is by design: the Q500US and Q1500US are Pipeline filters that select only the top 500/1500 most easily tradeable assets every day. I suggest you read more about these universes on this forum post.

if the dates and the code are the same it should be deterministic though

I've ran and re-ran numerous algorithms and backtests (2418 backtests according to my Q profile) and have never seen or suspected an instance where the same backtest yields different results.

Could you give a specific example of code and the different results you are seeing?

Here is a simple algo that doesnt trade but just prints out when a certain security is in the Q500US universe.
When you do several backtests, you get different results and therefore different universes...

2016-01-19 15:31  PRINT 2016-05-02 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-03 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-04 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-01-19 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-01-20 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-01-21 00:00:00+00:00  
Clone Algorithm
8
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5962bc6c8339f4579f1024a7
There was a runtime error.

I was doing more tests and it's the quantopian.pipeline.filters.default_us_equity_universe_mask which returns a different set of assets already.

@Albert R

Hmmm. I ran the above algorithm 20 times and each time got the exact same results (partial output shown below). It shows SID 42251 moving in and out of the Q500US universe but always on the same dates from what I see. It moves in on 2016-01-09 then out on 2016-01-29. It moves in again on 2016-05-02 and out again on 2016-05-18, and so on. On all the backtests I ran this was repeated exactly.

What behavior are you seeing?

2016-01-19 08:31  PRINT 2016-01-19 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-01-20 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-01-21 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-01-22 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-01-25 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-01-26 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-01-27 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-01-28 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-01-29 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-02 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-03 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-04 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-05 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-06 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-09 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-10 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-11 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-12 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-13 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-16 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-17 00:00:00+00:00  
2016-01-19 08:31  PRINT 2016-05-18 00:00:00+00:00  
2016-01-19 08:31 WARN Logging limit exceeded; some messages discarded  
2016-07-20 08:31  PRINT 2016-07-20 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-07-21 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-07-22 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-07-25 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-07-26 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-07-27 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-07-28 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-07-29 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-01 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-02 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-03 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-04 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-05 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-08 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-09 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-10 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-11 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-12 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-15 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-16 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-17 00:00:00+00:00  
2016-07-20 08:31  PRINT 2016-08-18 00:00:00+00:00  

@Dan I reran the algo again and these are the two different logs I'm getting: (Note the first line)

2016-01-19 15:31  PRINT 2016-05-02 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-03 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-04 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-05 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-06 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-09 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-10 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-11 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-12 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-13 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-16 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-17 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-18 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-19 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-20 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-23 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-24 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-25 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-26 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-27 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-31 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-06-01 00:00:00+00:00  
2016-01-19 15:31 WARN Logging limit exceeded; some messages discarded  
2016-07-20 15:31  PRINT 2016-07-20 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-21 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-22 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-25 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-26 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-27 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-28 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-29 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-01 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-02 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-03 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-04 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-05 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-08 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-09 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-10 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-11 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-12 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-15 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-16 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-17 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-18 00:00:00+00:00  
2016-07-20 15:31 WARN Logging limit exceeded; some messages discarded  
End of logs.  
2016-01-19 15:31  PRINT 2016-01-19 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-01-20 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-01-21 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-01-22 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-01-25 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-01-26 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-01-27 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-01-28 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-01-29 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-02 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-03 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-04 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-05 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-06 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-09 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-10 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-11 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-12 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-13 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-16 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-17 00:00:00+00:00  
2016-01-19 15:31  PRINT 2016-05-18 00:00:00+00:00  
2016-01-19 15:31 WARN Logging limit exceeded; some messages discarded  
2016-07-20 15:31  PRINT 2016-07-20 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-21 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-22 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-25 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-26 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-27 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-28 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-07-29 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-01 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-02 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-03 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-04 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-05 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-08 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-09 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-10 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-11 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-12 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-15 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-16 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-17 00:00:00+00:00  
2016-07-20 15:31  PRINT 2016-08-18 00:00:00+00:00  
2016-07-20 15:31 WARN Logging limit exceeded; some messages discarded  
End of logs.  

OK, I saw it. Sometimes SID 42251 is in the Q500US from 2016-01-09 thru 2016-01-29 and sometimes it's not. Just re-running the same backtest with same dates and same code. Interesting.

Hmm? My understanding is that for paper/live trading, Q requires backtests to be repeatable (although I never quite understood the reason). And I don't know what happens if they are not repeatable. I've sent a note to Q support, in case they need to have a look.

@Grant already did that, they are investigating the issue

Hey guys, thanks for reporting this issue. We're looking into it.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.