Back to Community
Upgrading to Python 3

Hey everyone,

As many of you know, Quantopian currently runs on Python 2.7. In the coming months, we will be rolling out upgrades to the Quantopian platform that will move us from Python 2.7 to Python 3.5. Python 2 will no longer be officially maintained as of Jan 1, 2020 (referred to as the “end of life” or “EOL” date). As a result, we are planning to cut off of Py2.7 and upgrade to Python 3.5 prior to that date.

Upgrading libraries on Quantopian is a complex task. Millions of Quantopian backtests and notebooks are currently written in Python 2.7. While some of these will be able to run in Python 3.5 without any changes, we are expecting a significant number to require at least minor syntax changes. In an effort to ease the burden of making these changes, we have come up with an upgrade plan that we would like to share with you.


Why Python 3.5?

When we set out to upgrade Quantopian to Python 3, the first thing we had to decide was the minor version of Python 3 we wanted to target. We ultimately decided on Python 3.5 (for context, the newest version is Python 3.8). The biggest reason why we chose Python 3.5 over a newer version was to cut down on the scope of the project and the magnitude of the change. Upgrading to a newer version of Python 3 would have required us to upgrade pandas, numpy and other libraries, which are core dependencies of the Quantopian API. Upgrading these libraries would have required us to make API changes that would be likely to break a lot of user code. At some point, we would like to upgrade to a newer version of Python 3 and newer versions of pandas and numpy. But for now, we wanted to start with an incremental upgrade to Python 3.5 to ease the transition and to allow us to shift our focus back to adding data to Pipeline. For the time being, we are staying on our current versions of pandas and numpy (0.18.1 and 1.11.3, respectively). It’s also worth noting that the whitelist of supported libraries (and their versions) on Q will remain unchanged.


Steps to Python 2.7 —> 3.5 Upgrade

  1. We plan to update the algorithm IDE to allow running backtests in either Python 2.7 or Python 3.5. We are currently testing this feature, and we hope to make it available to you soon. We plan to release to a small group of alpha users in the coming days, with a public release coming soon after.
  2. We plan to update our Jupyter-based "Research" environment to allow running notebooks in either Python 2.7 or Python 3.5. We are currently working on this feature, and we aim to roll it out as soon as possible.
  3. In addition to supporting Python 3.5 in the IDE and Research, we plan to release tools to make it easier to migrate existing Python 2 code to Python 3. In particular, we are aiming to release two tools to help smooth the transition:
    • An "auto-conversion" tool based on 2to3 that will allow you to automatically migrate existing Python 2 code to equivalent Python 3 code. We have been testing a prototype of this tool internally, and we've found that the vast majority of algorithms on the platform can be migrated successfully using the conversion tool. We are still exploring the best way to expose the conversion tool in the IDE interface, but we are leaning toward making an interface where you can click a button and get a Python 3 copy of your code in the IDE.
    • A "Python 2 Compatibility" extension for Research that will enable running many notebooks that were written for Python 2, in Python 3. Most notably, the extension will allow Python 2-style print statements in Python 3.
  4. Finally, we will phase out the Python 2.7 environment from Quantopian. In expectation, this will happen in mid to late December. Prior to the Py2.7 EOL date (Jan 1, 2020), we plan to remove the Py2.7 options from the IDE and Research. We plan to leave the auto-conversion tool available for you to continue using indefinitely so you can easily convert any old algorithms to Python 3 beyond the Python 2 EOL date.

Contest

At a certain point, we will stop accepting Py2.7 submissions and require all new submissions to be written in Py3.5. Currently, there are hundreds of submissions running in Python 2.7. To avoid resetting everyone's contest score, we are planning to use the auto-converter to allow any submissions that were written in Python 2.7 to stay running in the contest beyond the end of Python 2.7 on Quantopian. When the time comes to drop Python 2.7 support from Quantopian, any previously submitted contest entries that were written in Python 2.7 will continue to run. Our plan is to use the auto-conversion tool to upgrade Python 2.7 contest algorithms in memory (as opposed to storing the Py3 version persistently) prior to running the nightly simulation. This way, your originally submitted code will stay the same and we can run your algorithm in Python 3.5.

The hope is that no one will have to do any work to keep existing submissions running. We will provide more info on this at a later date.


More Info Coming

Currently, we have a version of the Python 3.5 backtesting working in the IDE. If you would like to help test this feature, please let me know (email [email protected] and mention that you'd like to be added to the Python 3 alpha) and we can add the feature to your account. Soon, we will open up the feature to everyone.

The goal of sharing this plan is to keep you informed of upcoming updates as we further develop our plan. As noted earlier in this post, some steps in the plan are still in progress. We plan to provide updates as we make progress on the work.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

4 responses

That's great news! Looking forward to use the upgraded version.

Based upon the release schedules for Python 3.2, 3.3 and 3.4 and end-of-life timetable with those releases, you've got about 3 months until Python 3.5 goes end-of-life, is frozen and receives no further bugfixes.

Do you have any news on upgrading Scikit-Learn to 0.17.1 from 0.16.1? This older package has pretty limited and outdated processes. I hope you can make the change!

Great to hear that you are finally switching. I was wondering how you planned to handle the sunsetting of Python 2. By the way, the end-of-life date for Python 3.5 is only ten months away: Sept. 13th, 2020. https://devguide.python.org/#branchstatus