Back to Community
Testing on a larger scale

I'm in the process of learning the Quantopian backtesting system and have studied all of the sample algorithms in the help area though I am far from being proficient. Here's what I've missed despite looking a the multiple securities and set universe samples. Let's say I have an algo to buy (or sell) a security under a specified condition. It's easy to test that on 1 security or to build a portfolio of securities (I think). I'd like something a bit different. I'd like to run the algorithm on every security one at a time and analyze the results. That is I want to test 1 algo on 500 (or more) stocks individually. What was did the average or median return v a buy and hold? What's the batting average? Information ratio etc. Is there a way to do this more massive testing?

To add to this, but to put it in a different dimension, let me add the following. Let's say I have an algo that depends on 2 parameters such as a window length (20 day) and a threshold (3% e.g., buy if 3% above a moving average). If I want to test the algo varying both the window length and the threshold so that I can examine the results, is there a way to loop through the backtests recording the results?

13 responses


For a variety of reasons, this sort of data exploration/screening is kinda challenging on Quantopian, but it could be done offline using zipline and your own trade data.

No time to elaborate now, but I figured I'd provide some initial guidance. Others can chime in.


I would appreciate an elaboration when you have time. All I'm really asking for is a bunch of loops which computers are good at. I realize this could be time consuming, but I'm a guy who learned to program with punch cards. We were thrilled with a 300 baud acoustic modem. Patience I have. Email me in a week when the analysis is done. I don't mean to complain or be rude. Quantopian is fascinating and an amazing tool (and it's free to boot). I'm just curious.

Let me add that I've never been a trader, just an investor (the distinction is rather arbitrary, but I mean my holding periods are long and I have focused more on fundamentals than technicals). For me to actually trade on an algo would require extensive testing for robustness. I wouldn't be happy saying it worked on Apple and Google. I'd prefer to know it worked on many securities across many markets. That's why the breadth of the testing is important to me.

It takes computation power to do backtests. Right now Quantopian is providing this for anybody to do here. The backtests are running on Quantoptian provided computing power. If everybody were allowed to run hundreds of stocks across multiple parameters the amount of power necessary to provide such a service would grow exponentially.

Suppose a test is run over two parameters each varied over 10 different values. To test that on a single algorithm is 2^10 = 100 iterations. Now running that over 500 stocks/portfolios is 50,000 iterations. Add another similar parameter (3 parameters) that is 500,000 iterations. Multiply that over the number of users and the number of different algorithms a single user will test.

On one hand I get it. On the other hand, the daily algos run amazingly quickly...if it takes another hour/day/week I don't care. As I said, I wouldn't mind a batch process. Or if the community became kind of like the SETI project with distributed processing. My PC/CPU is idle most of the time. I haven't quite figured out the Quantopian business model (their privacy policy precludes them from capitalizing on my amazing algorithms). Seriously, I'd pay. The value of a better algo is well worth whatever the computational power costs. Perhaps quick tests are free and there's a charge for the long ones. One could quick ones with just a few stocks/parameters. Once it looks quite promising, a person might pay for the longer one.

I started another thread asking what would it take for someone to actually invest according to an algo. In my mind, it would probably have to be robust (work across a lot of securities) depending on the idea. If that's true for many people, then more breadth is going to be a requirement.

I use the QSTK package for this task. It is a free python2.7 based set of scripts developed by Tucker Bulch of Georgia tech. While challenging to install and learn it can do many things.
For example do the following:
Automated download of daily data for sp500
Event studies with graphic results.
Market simulations against benchmark of your choice
Analysis of market simulation to determine Sharpe ratio, and return.

Many other possibilities.
I am currently writing a library of technical indicators to use in my event profiler.

Bottom line is you can do everything you can do in quantipian but you have to write the supporting code.

Quantopian is much e easier to use and is better documented. To my knowledge there is no free source for minute data either. While qstk is capable of running minute data or any time interval for that matter only daily data are available for free. For example I download daily data from yahoo finance into csv. They have daily data back to the 1950's.
Google QSTK for more info.

Hello Doug,

I used QSTK on the first instance of Tucker's "Computational Investing, Part 1" course on Coursera. I haven't used it since but I should revisit it.

Have you thought of using using the talib Python wrapper for the TA-Lib indicator library rather than writing your own?


Hello Rex,

Presently, Quantopian has no limit to the number of backtests you can have running concurrently, in separate browser tabs. There is no way to launch them automatically (e.g. via a batch file); you'll need to initiate each manually.

Based on previous feedback, Quantopian is aware of the gap in capability, and have more flexible research/screening/exploration tools on their long-term roadmap. In some respects, they've already covered this case by offering zipline as a free, open source tool; you just need to provide the data. I have not tried it yet, but there have been reports of it being useful, including a cloud computing example by Thomas Wiecki.



The dependencies looked a bit daunting.

Hi Doug,

Would you mind explaining how you would like Quantopian to better expose parallel backtesting and optimization?



The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

I was able to get zipline doing parallel backtests in a local cluster as well as picloud last year. For quantopian to expose this sort of thing, they'd probably want to create "parameters" in the context, including universe selection parameters.

All that said, grinding out thousands of backtests looking for a set of parameters that are profitable in-sample is a pretty dangerous way to build a system, especially without any cross-validation.

Exactly anony.

I know Tucker and his partner Erez Katz at Lucena Research. Small world. Any chance any of ya'll are in Atlanta?

Re: Hi Doug,

Would you mind explaining how you would like Quantopian to better expose parallel backtesting and optimization?


The only item on my wish list at the moment would be for the display of backtest results.
I'd like to see multiple windows for record() so scaling does not mute results.
Say record(), record_sub1()....etc would open 2 windows referencing the same time series.
This way I could visualize movement of say a $500 stock as well as an indicator that for example would move in the -1 to 1 range.