Sharpe ratio of 36.69 on Leaderboard!!!!!!!

It was suggested by a very nice person on here to look at the statistics on the leaderboard to understand the contest statistics and so I did. The leader, Naoki Nagai, has a paper trading Sharpe ratio of 36.69. Whoa! What little I know about Sharpe ratios suggest this is fantastic. It is going to be hard to beat an algorithm that paper trades with a 36.69 Sharpe ratio.

In a former life I was an assistant to a powerful guy who managed money. I answered his phone, ran errands, got coffee and lunch, kept his schedule that sort of thing. I overheard him express disbelief when he was told that Steven A. Cohen - the founder and manager of the SAC hedge fund - had a Sharpe ratio of 7 in his first few years. Now we know that SAC was doing some pretty illegal stuff to generate returns like that. Even so, a Sharpe ratio of 7 is probably as high as it gets.

Imagine a Sharpe ratio of 37?

Perhaps there should be a reality screen on scoring algorithms.

8 responses

That score is based on 2 days of trading data. There's nobody who thinks that a Sharpe of 36.69 is the last word on that algorithm. You'll find that the contest scoreboard is very noisy in the beginning. Lots of people have a good day.

We don't give out prizes for having one good day. That would be like declaring the winner of a marathon based on their first 1/8 mile. We give out prizes after you've put together a lot of good days. The question will be, what is his Sharpe after trading out of sample for 6 months?

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Dan,

This what you said:

We give out prizes after you've put together a lot of good days.

But I can not find any metrics in scoring system which counts good days.

May be there is a reason to add something like this

good days/(good days+bad days) = portion of good days


or probability weighted ratio of gains versus losses for some threshold return target

SUM(IF(Returns > RiskFreeReturn, Returns - RiskFreeReturn,"")) / -SUM(IF(Returns < RiskFreeReturn, Returns - RiskFreeReturn,""))


to make scoring system more balanced?

First thank you Dan for your reply but this just leads to more confusion for me about what the "Leaderboard" is all about. How can I see who is actually leading in the standings, you know determine who the next contest winner will be? On Kaggle, the leaderboard indicates who is leading, often using a public testing set and while there are changes in rankings after the private test set is applied, the metrics of the rankings are the same.

And to Vladimir's observation, the scoring system used to construct the leaderboard may need updating. From what you wrote, perhaps the "leaderboard" should be renamed, "The Newest Algorithm with Positive Returns Board".

Would you be so kind and send me a pointer to documentation or give a brief overview that helps me understand how algorithms are ranked? Many heartfelt thank you's in advance.

Hi Sally,

I would urge you to read the contest page available at https://www.quantopian.com/open

You will find that the seven performance criteria that generate the final score for each entry are neatly mentioned there. I believe that the contest is constantly improving, and rules are updated every month to ensure that. Have a look at https://www.quantopian.com/posts/august-contest-rules-update-new-prizes-staying-hedged-going-longer

And yes, the Leaderboard will be volatile during the first few days. The 6-month contest has been kicked off to address that.

Thank you AD for your helpful comments. I read and now re-read the contest page you referenced and while it was helpful, it brings up all sorts of questions most of which I will have to think through for myself.

Correct me if I am wrong, and I'll admit my maths are a bit rusty, but the one month contest uses paper trades for a one month period or approximately 20 trading days. Doesn't that mean that my sampling error at the 95% confidence level is something like 22%? So each stat listed should have a +- 22% range?

A second question is on how the 7 criteria were chosen. One might expect many of them to be correlated since Sharpe ratio, annualized returns and volatility and Sortino ratio all use returns while beta and return stability are quite similar. I've seen these metrics used in marketing and comparing funds, but worry about using them for investment decisions, which is what the contest is - an investment decision on selecting an algorithm. Would anyone use these seven criteria for selecting stocks? If not why use them for selecting algorithms?

The six month contest sounds interesting. 120 trading days still leaves a 10% sampling error. No wonder many investors demand a three year track record using real money before considering an investment.

Justin Lent in https://www.quantopian.com/posts/bug-in-consistency-score explains this briefly, "It's important to think about the metrics used for the contest ranking in a more holistic manner. No single metric can define a great algo from a poor algo and it's important to analyze algos across a breadth of different metrics in order to get a 360-degree view of its previous performance, and hopefully future performance.

Perhaps in a sports analogy, consider the individual sports such as golf or tennis, where most of the time the best #1 ranked player in the world are above average across the breadth of the shots required in the sport, but many other lower ranked competitors are best at any of the 1 single strokes (serving, forehand, backhand, driving off the tee, putting on the green, etc). To be the best in investing is analogous, and in particular building a portfolio, is about selecting the components of your portfolio that are all above average across a breadth of characteristics rather than getting any single metric perfect."

This, I believe gives some insight on why Q insists on using all these metrics, in spite of them being correlated.

And your maths is spot on, of course while using 0.98/sqrt(n) to get the margin of error at 95% confidence with sample size n, the assumption of simple random sampling is crucial. Now its up to you to believe whether every one of those 20 sample points is just as likely as any other in the population, and if you subscribe to the random walk hypothesis :-)

Personally, if you look at how U.S. stocks suffered their worst day in 18 months yesterday, I'd say that confidence level ought to be worse than 22%. But what can you do, just too few data-points.

There are funds which do have Sharpe Ratios of 6.

Hey Sam, certainly a Sharpe Ratio of 6 is impressive, but you also have to factor in survivorship bias. If 100 funds are started all making random trades, then after a year or two there will certainly be a few that have happened to get lucky. These funds are indistinguishable looking at past performance from those that are actually making intelligent trades. Many funds go bankrupt every year, so the funds that survive have a hefty deal of survivorship bias associated with their returns.

If you chase Sharpe Ratios or returns, then you stand a big risk of falling prey to this bias. I would say a first step would be to pick a fund that looks good based on past performance. Then observe it for 3-12 months and check for consistency in performance between the historical numbers and those over the out of sample 3-12 month period.. This is similar to what we have done in the contest and will let you start to get a better sense of whether a fund is actually as good as they say.

You should also try to look at the beta exposures of a fund if possible. It is very easy to get huge returns if you take out leveraged bets during a good market. These funds, of course, are incredibly vulnerable to down markets.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.