Back to Community
Upgrading to Python 3

Hey everyone,

As many of you know, Quantopian currently runs on Python 2.7. In the coming months, we will be rolling out upgrades to the Quantopian platform that will move us from Python 2.7 to Python 3.5. Python 2 will no longer be officially maintained as of Jan 1, 2020 (referred to as the “end of life” or “EOL” date). As a result, we are planning to cut off of Py2.7 and upgrade to Python 3.5 prior to that date.

Upgrading libraries on Quantopian is a complex task. Millions of Quantopian backtests and notebooks are currently written in Python 2.7. While some of these will be able to run in Python 3.5 without any changes, we are expecting a significant number to require at least minor syntax changes. In an effort to ease the burden of making these changes, we have come up with an upgrade plan that we would like to share with you.


Why Python 3.5?

When we set out to upgrade Quantopian to Python 3, the first thing we had to decide was the minor version of Python 3 we wanted to target. We ultimately decided on Python 3.5 (for context, the newest version is Python 3.8). The biggest reason why we chose Python 3.5 over a newer version was to cut down on the scope of the project and the magnitude of the change. Upgrading to a newer version of Python 3 would have required us to upgrade pandas, numpy and other libraries, which are core dependencies of the Quantopian API. Upgrading these libraries would have required us to make API changes that would be likely to break a lot of user code. At some point, we would like to upgrade to a newer version of Python 3 and newer versions of pandas and numpy. But for now, we wanted to start with an incremental upgrade to Python 3.5 to ease the transition and to allow us to shift our focus back to adding data to Pipeline. For the time being, we are staying on our current versions of pandas and numpy (0.18.1 and 1.11.3, respectively). It’s also worth noting that the whitelist of supported libraries (and their versions) on Q will remain unchanged.


Steps to Python 2.7 —> 3.5 Upgrade

  1. We plan to update the algorithm IDE to allow running backtests in either Python 2.7 or Python 3.5. We are currently testing this feature, and we hope to make it available to you soon. We plan to release to a small group of alpha users in the coming days, with a public release coming soon after.
  2. We plan to update our Jupyter-based "Research" environment to allow running notebooks in either Python 2.7 or Python 3.5. We are currently working on this feature, and we aim to roll it out as soon as possible.
  3. In addition to supporting Python 3.5 in the IDE and Research, we plan to release tools to make it easier to migrate existing Python 2 code to Python 3. In particular, we are aiming to release two tools to help smooth the transition:
    • An "auto-conversion" tool based on 2to3 that will allow you to automatically migrate existing Python 2 code to equivalent Python 3 code. We have been testing a prototype of this tool internally, and we've found that the vast majority of algorithms on the platform can be migrated successfully using the conversion tool. We are still exploring the best way to expose the conversion tool in the IDE interface, but we are leaning toward making an interface where you can click a button and get a Python 3 copy of your code in the IDE.
    • A "Python 2 Compatibility" extension for Research that will enable running many notebooks that were written for Python 2, in Python 3. Most notably, the extension will allow Python 2-style print statements in Python 3.
  4. Finally, we will phase out the Python 2.7 environment from Quantopian. In expectation, this will happen in mid to late December. Prior to the Py2.7 EOL date (Jan 1, 2020), we plan to remove the Py2.7 options from the IDE and Research. We plan to leave the auto-conversion tool available for you to continue using indefinitely so you can easily convert any old algorithms to Python 3 beyond the Python 2 EOL date.

Contest

At a certain point, we will stop accepting Py2.7 submissions and require all new submissions to be written in Py3.5. Currently, there are hundreds of submissions running in Python 2.7. To avoid resetting everyone's contest score, we are planning to use the auto-converter to allow any submissions that were written in Python 2.7 to stay running in the contest beyond the end of Python 2.7 on Quantopian. When the time comes to drop Python 2.7 support from Quantopian, any previously submitted contest entries that were written in Python 2.7 will continue to run. Our plan is to use the auto-conversion tool to upgrade Python 2.7 contest algorithms in memory (as opposed to storing the Py3 version persistently) prior to running the nightly simulation. This way, your originally submitted code will stay the same and we can run your algorithm in Python 3.5.

The hope is that no one will have to do any work to keep existing submissions running. We will provide more info on this at a later date.


More Info Coming

Currently, we have a version of the Python 3.5 backtesting working in the IDE. If you would like to help test this feature, please let me know (email [email protected] and mention that you'd like to be added to the Python 3 alpha) and we can add the feature to your account. Soon, we will open up the feature to everyone.

The goal of sharing this plan is to keep you informed of upcoming updates as we further develop our plan. As noted earlier in this post, some steps in the plan are still in progress. We plan to provide updates as we make progress on the work.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

18 responses

That's great news! Looking forward to use the upgraded version.

Based upon the release schedules for Python 3.2, 3.3 and 3.4 and end-of-life timetable with those releases, you've got about 3 months until Python 3.5 goes end-of-life, is frozen and receives no further bugfixes.

Do you have any news on upgrading Scikit-Learn to 0.17.1 from 0.16.1? This older package has pretty limited and outdated processes. I hope you can make the change!

Great to hear that you are finally switching. I was wondering how you planned to handle the sunsetting of Python 2. By the way, the end-of-life date for Python 3.5 is only ten months away: Sept. 13th, 2020. https://devguide.python.org/#branchstatus

Hey everyone,

We just opened up Python 3.5 backtesting to everyone. Now, you can select a dropdown in the IDE to choose whether to run a backtest in Python 3.5 or Python 2.7.

We are still working on making a Python 3.5 kernel available in Research, and we still need to make the code converter accessible to you, but in the meantime, you can manually convert your code to Python 3.5 and run it in the IDE. Please note that the contest doesn't yet accept Python 3 submissions. We hope to start accepting Python 3.5 submissions in the near future and we will post here when that becomes an option.


Regarding some of the recent questions:

@John, Colin: Yes, Python 3.5 EOL is later in 2020. We will announce our plans for the next upgrade closer to that time.

@mike: We don't have any immediate plans to upgrade scikit-learn, but we will consider upgrading it along with other libraries like pandas and numpy (and possibly others) when we upgrade the next Python version.

Suggest you skip straight to 3.7 (with a view to being compatible with 3.8) and latest numpy + python + others. Is it that hard? Anything we can do to help?

Hey everyone,

Another update: you can now use a Python 3.5 kernel in Research. To select your Kernel, go to Run --> Change kernel --> Python 3.5. Similar to backtesting, this change does not update any other libraries.

Once you change the Kernel, you will be able to see the currently running Kernel in the upper right corner of the notebook interface:

To make the change a little easier, we added a feature that attempts to handle any print statements written in Python 2 syntax. In most cases, if you have a notebook that has something like print 'hello', and you run it on the Python 3.5 kernel, it should print the message (normally this would raise a SyntaxError in Python 3).

If you run into any unexpected errors when switching to the Python 3 kernel, please let us know! At this point, we are aware of one issue in full tearsheets where the risk factor plot raises an exception instead of displaying the plot. It's on our list to fix and we will post here with any further bugs that we discover (or fix).


@John: Unfortunately, the upgrade to 3.7 is a lot more complex than the upgrade to 3.5. Per the original post:

The biggest reason why we chose Python 3.5 over a newer version was to cut down on the scope of the project and the magnitude of the change. Upgrading to a newer version of Python 3 would have required us to upgrade pandas, numpy and other libraries, which are core dependencies of the Quantopian API. Upgrading these libraries would have required us to make API changes that would be likely to break a lot of user code.

It's definitely on our list. We will update the community when that project comes to the forefront.

I'm surprised that most user's code is that fragile to have such a dependency.

Perhaps you could say something like:
Python 3.5 + Pandas 0.18.1
Python 3.7 + Pandas 0.24 (at a minimum) or 0.25.x
Python 3.8 + Pandas latest

If "user code" doesn't work in later versions - that's a user issue, not yours!

@John: The issue is actually that the Quantopian APIs depend on features that are removed in newer versions of pandas, like Panel. For example, data.history will return a panel if given a list of asset and a list of fields as input. Another example is in the way that categoricals are handled, which affects Classifiers in pipeline. We'll have to update the internals of our API to use newer features, and doing so will likely cause breaking changes in user code. I like your argument, it would mean less work for us! But in this case, we'll have to do some work to keep the API working with newer versions of pandas. It's on our list. I'm not sure exactly which versions we'll be targeting yet, but we will update the community as soon as we have more information to share.

How is the transition going? There is still reference to in lesson material to code that only runs on Python 2.7. Example: https://www.quantopian.com/tutorials/pipeline#lesson12 - intervalues is not compatible.

Please upgrade statsmodels too!

We just updated the backtester's default environment to be Python 3.5 and officially deprecated the Python 2.7 environment. Python 3 submissions are also now accepted in the contest and we will be disallowing new Python 2.7 submissions soon. We are working on updating some of our content still - thank you for pointing out that example in the Pipeline tutorial that needs to be updated!

Thanks Jamie, and as you are working on updating on contents, it reminds me of a Pandas function to calculate rolling beta coefficient in Lecture 44: Introduction to Pairs Trading but no longer supported in Python 3:

rolling_beta = pd.ols(y=S1, x=S2, window_type='rolling', window=30)  

Brad Solomon posted PandasRollingOLS() on Github to replicate the pd.ols model - I tried to use it in the form as he proposed in stackoverflow for example:

model = PandasRollingOLS(y=S1, x=S2, window=30)  

Would the PandasRollingOLS() workaround be ok for Quantopian use? Thanks

Found this useful for conversion: https://www.python.org/dev/peps/pep-0469/

The default Research kernel is now Python 3.5, and the Python 2.7 kernel has been deprecated.

@Karl: Thank you for pointing that out. We'll have to look into it, but we will consider your suggestion when we update lecture 44. Thank you!

@Jamie where is this 2to3 tool "click a button and get a Python 3 copy of your code in the IDE". I could not find it currently and it's close to 2020 1 Jan. thanks

do we know if sklearn upgrade is a part of this?

Update: We've added the 'Convert to Python 3.5' button in the IDE which allows you to get a 'diff' between your algorithm's current syntax and the expected Python3.5-equivalent syntax. The conversion tool should make it easier to quickly convert Python 2 algorithms to Python 3. Let us know if you have any questions or issues with the conversion tool!