Survey on Backtesting Time

Hello All,

I'm a Quantopian beginner.
I'm writing an algorithm that currently takes about 1 minute to finish one day's backtesting, which means it will take about 4 hours to finish 1 year's backtesting and 40 hours for 10 years.

Is this normal to you? I've tried my best to optimize my computing time but it just has a lot of computing.

Anybody experienced such long backtesting time?
Does Quantopian even allow such long time?

Thanks.
Wei

7 responses

I'm new too. I have one algorithm that is fairly intensive (aka: inefficient) , as it grows it gets worse and I can see the slowdown over time.

How long is your backtesting? In hours?

Backtesting time isn't great. I can't really backtest any descent machine learning-based hypotheses. Anything more than a few features with a training window of more than 50 days or so makes the backtest unapproachable. In particular, any data transformations from original feature space to a transformed space before classification are not doable. Today, I tried one on just a few months of calendar time and after ~6 hours it was still only ~3% done.

Mine's pretty simple compared to Ian, but over the course of 10yrs backtesting I can typically get through about 4-5 years in a couple hours then years 6-10 take overnight.

Thank you all for your inputs.
Looks like I'm not alone...mine even got timed out if the dataframe I created is too big.
I tried to eliminate loops if possible. I think we can work harder to optimize the code but there is always limit.
Quantopian has to support all the coders so no way we can get super fast computing resource.

It's taking me about 5 minutes to get through a year of backtesting. The platform works fine for what I'm doing, but losing the internet connection even for a few seconds will stop the backtest, which is the biggest issue. Pipeline-based algorithms seem a lot slower than older algorithms but the pipeline is much more flexible and the older algorithms may stop working at some point. I haven't tried to use machine learning yet.

Thanks for everyone's feedback on this thread. It pains me to read any report that our backtester is too slow. We are actively working on speeding up some of the data sources that are bottlenecks to many backtests.

We have a bunch of best practices here: https://www.quantopian.com/quantopian2/best_practices. They boil down to accessing data only when you need it (for example, accessing context.portfolio requires recomputing your portfolio, including fetching the last known prices of any assets you hold, so it's relatively expensive to do every minute of a backtest).

If would be great if you guys could send me a version of your algorithm with any proprietary ideas removed, or some other example algorithm that lets me see the bottleneck you are experiencing. We can then help identify any optimizations that can be made, or otherwise learn about a new bottleneck in our system. My email address is my first name at quantopian.com.

thanks,
Jean

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.