Back to Community
Data change?

Was there any large change in the universe of stocks or any other part of the data?
I have two backtests from 2017-07-31 and 2017-08-04 with identical code but very different results...

6 responses

I have the same issue. Sent a note to support but I have not heard back. Here is a compare below. The same exact code. One backtest
ran on 29th of July one ran today! Someone at Quantopian please resppond.

DATE RUN    DATE RANGE  FREQUENCY   CAPITAL BASE    RETURNS  

Backtest 225 2017-08-07 2016-12-27 to 2017-07-28 Minute $2,620,000 -0.51%
Backtest 214 2017-07-29 2016-12-27 to 2017-07-28 Minute $2,620,000 2.46%

I was sorting stocks using AverageDollarVolume and another set of securities is returned by the pipeline now.

I am assuming you mean in the same time window? Otherwise ADV is suppose to return different lists at different time windows.

Of course, all parameters of the backtests were also identical.

Hello Kamran, Albert

It's pretty common for community members to dig into backtests and try to understand why the result is different. When people ask us to help, our first step is to verify that the code and parameters (capital base, date range, algorithm parameters) for both backtests are exactly identical using the "Compare" backtests feature. Sometimes we might think that a small change in the code has no effect, but the slightest difference can cause performance/results to fluctuate. Often people have modified the code, or the date range, or the capital, and the change in results was unexpected.

If you have confirmed that both backtests have identical code and parameters, please contact us using the menu option or email [email protected] and include both backtest IDs in your message. This would help us investigate further.

The Quantopian platform does change sometimes - bug fixes, improvements, data updates (missing splits, dividend or spinoff events, etc), but generally we make an announcement to the community when this is the case like we did when we updated Limited Partnership Filtering in the Quantopian Universes.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Ernesto

The following is the output from my comparison:

            Date Run    Date Range                Frequency Capital Base  Returns  
Backtest 2  2017-08-04  2010-01-01 to 2017-01-01  Minute    $1,000,000    574.23%  
Backtest 1  2017-07-31  2010-01-01 to 2017-01-01  Minute    $1,000,000    945.75%  
The two backtests have identical code.  

Your team should probably know what has changed between those dates.
I'm not comfortable sharing the backtest IDs, orders or the code but if it helps: it's not related to the previous issue I reported with the Q500US and universe masks being nondeterministic, as I am not using those in this algorithm.
Thanks in advance for looking into this issue!

Albert