Back to Community
Quantopian Credibility: What is the formula used for Scoring the Contest. I'm not understanding what I'm seeing. Can someone provide some insight to the below?

Trying to understand how the contest scores are calculated. See image below.

http://www.titlelight.com/images/seriously.jpg

14 responses

Beta, volatility and drawdown seem to weight heavily and returns not as much.
That beta could go to 0 and volatility could improve over time however the drawdown at 27 could only go further south.
At the bottom of links like https://www.quantopian.com/leaderboard/29, take a look in the CSV to see what the values were for those at top in ranking. In the headers, pt stands for papertrading and bt for backtest.
For an algo submitted at deadline, after one month I understand the paper trading portion of the score would be 1/6th of the overall score and will gradually become 100%.

The scoring is outlined in the judging section of the rules.

In summary, every algo is ranked on 7 criteria, and your score is an average of rank, and then inverted so higher scores are better. Then, the entries that have all three badges are ranked from highest to lowest, and then the entries with 0, 1, or 2 badges are ranked highest to lowest. The overall effect is that the three badges are a filter - entries need to have all three badges in order to rank highly.

Looking at your screenshot, I'm going to guess that the lower-ranked entry isn't hedged, and didn't get the hedging badge (I'd need s specific entry ID to be sure).

In answer to your rhetorical question, many investors would prefer a lower-performing, predictable, hedged strategy over one that is long-only and thereby exposed to market shocks. Of course, different investors have different desires. The contest scoring system is designed to reward low-volatility, low-risk, well-hedged strategies.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Ah. I see. So, I could make 10m on 10m and have a 14% draw down, with a Sharpe above 8, and a beta below 0.15, year after year, and it wouldn't matter. It wouldn't matter if you made a gazillion dollars. A well hedged algorithm, even if it under performs, with respect to the SPY benchmark, irrespective of inflation, will always win, because the point of investing, in this contest, is not to beat the market, but triumphant risk management irrespective of returns.

Wait .... One sec. I just got a call from a hedge fund manager who threatened to shoot me if I ever uttered that paragraph again. Seriously though Dan, is that really the point of this contest? May the best hedge win? I want you to answer me with a straight face and tell me that you would pick the algorithm on the left if you had 10 million to invest, but before you answer, I would like to reaffirm that I have a great degree of respect for you, being the wise and candid individual that you are. And I'm not being sarcastic here, as I often am.

I don't know about the competition scoring, but here are my thoughts. Forget the backtest which is just "theoretical" and might have been curve-fit, whereas the live paper-trading is about as "real" as it gets, although 8 days is nowhere near enough to conclude anything meaningful at all. However after the whole 8 days, take a look at the most important number, the max DD, and remember that "worse is ALWAYS still ahead .... and maybe just around the corner". The brown case on the left has maxDD = 0.0079% whereas the green one on the right has maxDD = 0.27% or 34x worse. And if you look at volatility, the green one is worse by a factor of 28.8/0.11 = more than 260x. There's lots of scope to leverage up the brown case to increase the return if desired, but not the green one. "Gazillions " in theoretical profit is useless if the account gets bankrupted on the way there. If you don't believe it, the best way to convince yourself is just to try trading it live with your own account. Yes, seriously.

I would be inclined to agree if bankruptcy occurred at any point during the entire history of back testing. However, it did not, and cannot due to one simple fundamental principle that all sensible traders employ. It uses a "Stop Loss." I back tested as far as you can go, and it never went bankrupt. You can see those stats on the right. In fact, it performed best in the worst market conditions, as indicated by the long term beta. I'm curious as to what tests are being run for this "Paper Trading." I say that because I don't see a 26% drop and this thing has been running every day. It's at 15.44% in eight days, and has not had any such dip. Is there somewhere we can observe this "paper trading" as to how it differs from what the algorithm is presently doing. Come to think of it, why would it be doing anything else? I was under the impression that paper trading is what I'm looking at in my console while the algorithm is running. Where is the 26% drop? How is it that I don't see that?

www.titlelight.com/images/Ismellarat.jpg

Why waste your precious time "smelling rats"? There are better things to do! ;-))

Just be careful about:

... bankruptcy .....during the entire history of back testing ...... cannot due to one simple fundamental principle that all sensible traders employ. It uses a "Stop Loss."

Back-testing is not reality. In reality stuff happens. Stops don't always work. In the markets, there is no such thing as "cannot". Just ask LTCM.

However, if you believe in your system, try it live, with real money. Yours. Its the only way to find out.

Tony. I trade the markets every single day with real money. My point here is not to debate what if scenarios. My question is simple. How does Back Testing differ from Paper Trading? Are they plugging into a live paper trading account and and running your algorithm against it? I'm simply trying to understand.

@Aniken $kywalker: I agree, i also think, that Q is not doing a good job when it comes to evaluating algorithms...
I understand, that Q is leveraging its allocated algorithms and therefore lower volatility and DD are preferred, but what if leveraged ALGO is still doing worse in all aspects than ALGO with leverage 1?

I've started similar topic here (if you are interested): https://www.quantopian.com/posts/strange-contest-scoring

No investor with half a brain is going to invest in something that's going to make him/her less money than the SPY, while at the same time failing to factor in inflation. Even Buffet factors inflation into his investments, while looking for a return of at least 10%. All this .00000001% risk hedge, 1% return buffoonery is only exalted in the bizzaro world of Quantopian contest assessment. To make something like that work, you would need really good sales people, straight from hell, completely devoid of conscience, and a nation of Muppets. The point of investing is to make money. In my opinion, the baseline for an algorithm should be at least twice the return of the SPY, or you should not even be considered a participant. Quantopian should be encouraging people to be creative, taking risks within rational boundaries, creating algorithms that set themselves apart from the herd, not encouraging people to mount armored bicycles in hopes of winning the Indy 500.

https://ibb.co/ik8a2b

Interesting fact (image), algo2 is nmb1 in contest and even with leverage quite useless for the investors... What do you think?
True, algo1 has bad beta, but this will even out when the bear market comes...

Is that link an image? What is there no suffix? For instance .png, jpg, gif, jpeg ... etc?

It appears that it's starting to even out as Dan alluded to earlier. As the Beta falls over time and approaches the .141 as indicated by the back test, the score should climb. It's been less than a month. So, I'll keep watch. I'm currently at 2.3 million on 10 million. At this rate, I will have doubled the investment. If this actually happens, and I have a Sharpe over 5, currently at 8.9 right now, I will be interested to see where the algorithm finishes. If Quantopian isn't interested after this, and awards Orange Green Snail an award for risk management, we can come to our own conclusions, as to whether or not we should invest any more time. Point blank, If I can double 10 million in a contest and lose to a guy who makes less than 5 percent, well that scenario will emphatically speak for itself.

https://www.quantopian.com/leaderboard/35/59e2f50a6bf94500102cb7be

Yes, that link is an image, there is no suffix, because it is hosted on image hosting service...

Thanks. I assume you already know why I asked.