Back to Community
bug: risk metrics are miscalculated (overall metrics)

EDIT: the "overall metrics" of Risk Metrics are miscalculated.

here is a repeat of my last post. see the last post for the algorithm with the invalid metrics

In the following, every month except the first (which is only a couple
days) has a Beta of 0.997 or above. Yet the final Beta displayed in
overall metrics is 0.76

8 responses

ok, so I did a test, ordered 100% of capital into SPY on day 1. I don't at all understand the alpha/beta values given by the results.

i would expect the alpha to be slightly negative, and the beta to be almost 1.0

Clone Algorithm
13
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 531e8642d5c80f074676ad16
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

Dan posted a good explanation of why your algo gives slight differences to benchmark returns here.

If you want to dig a little deeper, jump into the alpha and beta source code!

i don't mind getting different returns, what I am asking about is how the quantopian risk metrics are not documented and don't seem to follow industry conventions.

for example, beta, is a measurment of risk related to some industry "standard index". a value of 1.0 would imply "as risky". a value less than that implies "less risky". so I would assume that an investment into SPY would provide a Beta value similar to that of SPY (which is 1.0) but instead quantopian provides a beta of 0.75,

for alpha, a value of 0.0 means the same return as the standard index, with a value of less than zero meaning the returns are lower than the standard index. So I'm wondering why the Alpha for my example is greater than zero? that implies my returns are greater than the standard index.

thanks for posting the source code links, though skimming it doesn't really help me understand the difference between quantopians values and that of other finance websites.

ok it looks like if I click on the details for Alpha/Beta it looks more realistic. Does that mean that the "overall Metrics" values are being miscalculated?

For example, Beta only shows 0.705 for Feb 2008. Every other month is above 0.92, so the overvall Beta of 0.75 seems in error.

Beta will only be equal to 1 if the algo returns are equivalent to the benchmark returns.

Remove the if statement around order_target_percent to enable portfolio rebalancing for dividends and beta will be closer to 1.

ok, so in that case, my last statement about the overall Metrics being miscalculated looks even more evident.

In the following, every month except the first (which is only a couple days) has a Beta of 0.997 or above. Yet the final Beta displayed in overall metrics is 0.76

Clone Algorithm
13
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 531eaf3e645ae60751cd4274
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

That is a bit strange, you may well have found a bug! Alpha doesn't look right either.

Hi Jason,

Thanks for pointing this out to us! I'm going to investigate this issue and you (or anyone else) are welcome to dig in as well. Take a look at this thread here: https://www.quantopian.com/posts/sharpe-ratio-other-risk-metrics-change-slippage-in-live-trading

In that post, we shared the excel file answer key against which we test our Zipline code logic. If our calculations match the answer key, we consider the code to be correct in the backtester. I'm going to run some local calculations and check the results against the Zipline code base and answer key. I'll report back my findings.

You may be right that there's a bug, and I'm curious to find out. I agree with you that we can do a better job documenting the risk metrics; it's something I've been thinking about but haven't gotten my hands on. Thanks for the encouragement to make it a priority! We want to be as transparent as possible about our documentation, calculations, and logic.

Best,
Alisa

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.