Back to Community
Digging Deeper Into Backtest Performance

Last week, we shipped a new feature where you can easily jump into a detailed tear sheet analysis of your backtest. On the backtest results page, you'll see a new button available:

For those already familiar with tear sheets, you can now make one at the click of a button. No more copy-pasting code or hunting around for backtest IDs! For those wondering 'what the heck is a tear sheet?', read on.

Creating a Tear Sheet

Clicking the button above will bring you to a notebook with the following cell pre-populated. The code below is all you need to get started with analyzing your backtest. It gets the ID of your backtest, loads your positions, transactions and other exhaust into an object, and then runs a series of analyses on your backtest.

Click the cell, and then press Shift + Enter to run it. Running it will emit several tables and graphs that give you insight into your backtest's performance (in Quantopian parlance, these tables and graphs are known as a 'tear sheet').

The tear sheet code is all part of pyfolio, our open source performance analysis library. Head over here if you'd like to see the code for create_full_tear_sheet in particular. (Some changes in pyfolio haven't yet made it to the Quantopian platform, so there may be a few analyses there that aren't yet in your notebook).

Reading a Tear Sheet

The tear sheet is a great way to learn things about your backtest beyond the details available on the backtest results page. For example, my backtest had an in-sample average Sharpe of 1.44, but when I look at its rolling Sharpe in the tear sheet, I can see that my Sharpe jumped around all the way between 4 and -2:

Beta, by contrast, is much more stable:

The tear sheet provides a whole host of other metrics in addition to the ones shown above. For example, you could:

  • Find out your Fama-French factor exposure
  • View your most concentrated long and short positions of all time
  • Track your net dollar exposure, to verify you remain hedged over the duration of the backtest
  • Plot your turnover to see how actively the backtest trades

The BacktestResult object (i.e the output of get_backtest) also allows you to programmatically access attributes of the backtest, so you can create your own analyses outside of those that create_full_tear_sheet provides. For example, with the code cell above, you could run bt.positions to get a pandas DataFrame of your positions over time. The full list of attributes you can access is available here. You can also run individual analyses, if you don't want to get the entire tear sheet in one go - check out the pyfolio docs for more details, or take a look at this notebook for even more things you can do with pyfolio.

One thing that is important to note is that all the metrics provided in the tear sheet are in-sample. It's good to look at them, but be careful of over-optimizing! Overfit backtests can have disappointing out-of-sample results, which can hurt an algorithm's chances of winning the contest or getting an allocation.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

7 responses

Please don't take this the wrong way, but I just wanted to say that this hardly seems to work for a typical long-short algo using e.g. 300 positions, with daily rebalance and volume slippage enabled. There are too many transactions in the backtest (> ~10k per day) to analyze.

I already contacted support about this and I fully understand why it would hit certain resource limits, and I'm not being negative here, it's just that if you're aiming for long-short algos the current backtest analysis doesn't cut it for longer periods.

Hi @Ivory Ant,

We're going to investigate this further to identify what the bottlenecks are (# of transactions, backtest length, etc). If anyone's running into similar issues, the feedback would be very helpful as well. If possible, could you share an example backtest that you're running into problems with?

As a workaround while we investigate the bottlenecks, it might be possible to run a shorter backtest (say a 2-year backtest, for example) - the tear sheet will still be informative, even though it won't have as much data.

Sorry I haven't got time to prepare something right now, I was just about to log off. A simple example would be the long-short tutorial algorithm, just run it from 2004 to 2017 with volume slippage set to 0.025, with a starting capital of 10M.

This also locks up the backtest results in the browser after a while BTW.

I've had the same issue for a long time. Run any long/short dollar neutral/beta neutral model and resource limits trip when the book size exceeds $1B. The browser lock up issue has more to do with local ram than research memory. My suggestion is to start your backtest and then shut down your browser while the test runs. This is only for longer duration tests on high performance models with substantial capacity. Everyone else needn't worry.

Hi @Abhijeet Kalyan;

Try creating a tear sheet for this backtest: 59c96026ed42ff507a91b6e1

@Pej Hamidi that's indeed a good trick which I already used, but I meant that the page also locks up after loading backtest results when it's done running. Mostly just after loading the transaction data. It's more a JS/cpu issue than a RAM issue, I have 32GB ram in this thing.

Hi @Abhijeet Kalyan;

I think this Analyze Backtest Tearsheet is good, although occasionally i'm also having similar problems to Ivory Ant & Azure Tiger. In the case of Equity LongShort & lengthy runs, it starts up but then freezes with no results displayed at all.

I also have some questions / comments / suggestions regarding several of the individual output plots. I will get back to you with those shortly.
Thanks & best wishes, Tony.