Back to Community
The Process of Naming the Contest 14 Prizes

We spent the last several days reviewing the leaderboard results at the end of Contest 14. Our process is pretty straightforward. First, we run the leaderboard-generating code and review the resulting file. We look at the file to make sure that there are no obvious unexpected outcomes and investigate anything that looks suspicious. Then we run tearsheets on the leaders. We review each tearsheet to see if there is anything obviously wrong. (Just about everything also gets evaluated for an allocation, but that process is separate from the way we declare a winner).

What we found this month in the tearsheet was unexpected. There were a couple of entries (not the leader) that weren't following the rules as we intended them to be applied. We found that a few entries were not hedged - they were holding only a nominal short position. Take a look at this snippet from one of the tearsheets. You can see that the algorithm is a long-only algorithm. If I didn't tell you that the short position was slightly greater than zero, you wouldn't be able to tell from this graph:

The relevant section of the rules says: "Your algorithm must be hedged to the market. It should hold both long and short positions simultaneously, or be entirely in cash. Hedged strategies reduce their market risk and correlation risk to individual positions." The entry you see in the tearhsheet certainly isn't consistent with the spirit and intent of the rules. Everyone who has read our commentary about the contest knows that we are looking for long/short algorithms with low exposure to the market. The nominal position is a fig leaf, and that fig leaf doesn't cover much. The algorithm is short "enough" to get a badge on the leaderboard, but it is not hedged by any reasonable measure.

We talked about the situation for a while. On one hand, there are a couple of contestants who think they had valid entries and think that they should get a prize. If we disqualify them, they will be understandably frustrated. On the other hand, we have a couple of contestants who built the kind of high-quality algorithms that we are looking for; if they don't get a prize, they would also be understandably frustrated. We've come up with a solution that we think solves the problem most equitably.

Going forward:

  • We will award prizes to two sets of winners this month. The entries with a nominal short position will get prizes. And, the next-best entries follow the spirit of the rules will also get a prize.
  • Going forward in Contest 15 and beyond, these unhedged entries, the entries with only a nominal short position, will be disqualified.
    I believe that everyone will see the care and consideration we've put into this solution. We apologize for any frustration or confusion that it may cause. We think that this path is the best, balanced outcome that is available to us.

Edit: I wanted to clarify that we were concerned with the 2nd and 3rd place entries. The 1st place entry, Frederik's, was well-hedged.


The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

42 responses

Eyeballing shorts doesn't seems very deterministic. Have you considered introducing net_leverage as a metric and requiring

abs (net_leverage) <.75

or some % that has the desired effect?

Sorry, perhaps it's written somewhere else, but what specifically are the new requirement for hedging? Are the contest entries now getting the badge only at some specific level of simultaneous notional exposure long and short?

Or are the future disqualifications going to be a post hoc judgement call...

Dan, I think this is fair borderline on generous, so thumbs up.

The contest entry list does feel like a bit of a bazaar at times. The more it is aligned with the fund's requirements the more motivating in my view. This would include downgrading entries with:

  • Insufficient hedging (as you describe above)
  • Concentration risk (e.g. >10% in a single stock/etf)
  • Passive algos (i.e. not Active by Jonathan Larkin's criteria in his blog post)
  • High correlation to strategies already in the fund (although this might be difficult to make transparent in practice)

I also note the contest rankings are often a few days out of date. I've seen it a few times, but today is a good example. The rankings didn't run on Monday morning, so they are showing "last updated: 4 days".

Maybe the rule should read delta neutral, or (near) neutral and give some range or neutrality i.e +/- 20%.

The specific rule is important, because there are instruments with internal leverage that might provide an effective hedge even if only 1/3 of the capital is allocated to them. This problem will be worse when futures are allowed, though presumably that will herald a complete overhaul of the margin and notional exposure calculations.

I totally agree with Simon, rules should be extremely clear.

Why isn't this rule already captured in the requirement for low beta?

From Simon: there are instruments with internal leverage that might provide an effective hedge even if only 1/3 of the capital is allocated to them

Sure (e.g. SPXS, as noted here: Why is this a bad thing though? The use of "internal leverage"?

I think explicitly geared ETPs are already forbidden. I was talking more about whether they intend to mandate some kind of dollar neutrality or something else. Attached is a silly strategy which would probably pass all the automated tests, yet if I understand correctly, would be rejected if it won.

EDIT: if anyone is feeling adventurous, you could probably turn this into a conforming strategy by trading the top 100 stocks of the consumer cyclical vs consumer staples directly, rather than using ETFs. In fact, you could hedge with the top 100 in the S&P 500 by market cap too.

Clone Algorithm
Backtest from to with initial capital
Total Returns
Max Drawdown
Benchmark Returns
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 57aa6ca517b4f6106e99eb0c
There was a runtime error.

Eyeballing shorts doesn't seems very deterministic.

Could also look at correlation between market and algorithm performance under samples from the time-series model used in the Bayesian tear-sheet.

Why do you have to short to hedge? In most cases, a long T-bond positions is actually a better hedge.

Quantopian - "We want low beta, high sharpe, perfect hedging and low correlation to the rest of the algos. We want a unicorn!" Good luck with that, guys. Meanwhile, back in the real world, where quant pay is higher than $10,000 for six months... At least have the integrity to acknowledge that hedging can be accomplished in a long-only portfolio.

Why do you have to short to hedge? In most cases, a long T-bond positions is actually a better hedge.

Totally agree with this as well.

Dan comments above:

Just about everything also gets evaluated for an allocation, but that process is separate from the way we declare a winner.

Maybe the two processes should be reconciled? One would like to think that high ranking in the contest would indicate a high probability of getting an allocation, right? Presumably, if Quantopian aims to fund a decent fraction of its 80,000+ users, both the contest judging and allocation evaluation processes will need to be largely automated (unless the plan is to fund only 30 or so algos, in which case, a manual process might suffice). In the end, it would seem that there would be enough data to identify and rank promising algos automatically. Can the HAL 9000 be programmed to sort it out? "Just what do you think you're doing, Dave? Your algorithm is grossly overfit!"


Thank you very much for sharing the algo. Could you please describe shortly what it does?

Why do you have to short to hedge? In most cases, a long T-bond positions is actually a better hedge.

Um yeah...why bother?
Dual Momentum

See "my" 50/50 system:

1m LB  
CAGR    Max DD  Vol  
mean    6.86    12.73   7.61  
median  6.88    12.95   7.63  
min 4.84    8.79    5.35  
max 9.26    16.34   9.15  

Thank you very much for sharing the algo. Could you please describe shortly what it does?

It just switches between consumer staples (defensive) and consumer discretionary (aggressive) ETFs based on the slope of the VIX term structure, and hedges with S&P 500. All static.

Here is simple question. Let's assume I have a magical algo that uses sp500 and bonds to balance excellent score and I'll win the competition with it.

If I use spy + some bond etf to hedge it will be disqualified because it's "all long".
If I use spy but instead of going long some bond etf I go short inverse bond etf. The algo will be long/short algo and not be disqualified.

Why? What is the difference in risk between these algos?

It was my understanding that those would both be disqualified for not having enough assets presenting some sort of concentration risk. But that is neither here nor there.

It would be interesting to hear from the Q powers-that-be why the long-short hedging jazz is so special? Why is it such a big deal? Is there some proven magic in it? I keep hearing folks say it ain't all that special. Well, why is it special? I kinda see that if one can get a nice smooth long-term return and then apply 6X leverage, as Q has announced as part of their plan, it could work out nicely. Is that why it is so exciting? There's been very little information on why the contest rules and fund criteria are what they it just that it is what Point72 is asking for? For those in the industry, maybe it is obvious, but it is not so clear to everyone here.

By the way, I sorta have to congratulate anyone who managed to game the contest a bit. It would seem to be in the spirit of hacking Wall Street (as with its raised fist would imply--power to the people!). Kinda like a trick play in football--if it is within the rules and it works--awesome! In the end, just a game.

When the Q fund concept was first kicking around, I thought the idea was that user's retail trading strategies would somehow align with the needs of the fund. In fact, I recall that Q was looking for users to share their real-money trading results (with the money coming from their own pockets). The Q open was meant to be a means for folks without lots of cash to still be able to be in the running for an allocation. Now we are at $10M minimum in capital, and strategies dealing in large portfolios that are probably not in alignment with the user's retail trading practices. Any thoughts on going back to the idea of supporting low-capital retail trading? There is a gross mis-alignment between the Quantopian retail side of the business (Robinhood?) and your push into institutional-style scalable strategies. Was it simply concluded (hopefully with data) that your user's retail algos wouldn't work? You'd never reach your $10B AUM goal, even if you could scale the user base to the sky?

Correct me if I'm wrong, but:

  1. Beta is cheap to buy. So why impose a) beta -0.3 to +0.3 limitation, and b) short position requirement, when Quantopian as a fund can hedge out market exposure itself from any algo to meet the desired beta target? In the end, Quantopian will pool a certain number of algos into a portoflio, each of those algos will have a certain beta, and portfolio of algos will have its own overall beta. So why can't Quantopian itself hedge out beta (with short market position or a long negatively correlated asset) on a portfolio level?

  2. If beta -0.3 to +0.3 limitation and overall market neutrality is so important for Quantopian, why SPY is used as a benchmark by default? Why cash is not used for benchmarking then?

  3. If we are here to beat the market, why the contest rules do not take into account performance-per-unit-of-beta type of measures? For example, why Information Ratio is not a criteria for ranking algos?

My speculation is that at some point over the past year, Point72 said "Show me X and we'll provide Y." I ain't no business guy, but if a customer shows up with $250M, it is a no-brainer that their need becomes your need, and by extension, the need of anyone participating in the contest and trying to get a fund allocation. The risk is that what the customer thinks he needs is not really what he needs, but everyone falls in line since he has money. One way to mitigate this lemming-effect risk would be for Quantopian/Point72 to support a channel for non-conforming algos, under the assumption that this whole formula of scalable, institutional-grade long-short equity could end up a disaster (or at least under-perform, relative to alternatives). The thing that is not discussed in any of the press releases is what Point72 expects to pay for work by Quantopian quants, relative to full-time ones. The cost of having an on-staff, full-time experienced guy with an advanced degree sitting in an office in the U.S. is very high ($300K per year? Maybe more in expensive locales). Why not outsource globally? Hey, how about global temp workers that we don't even have to pay? Quantopian to the rescue! In fact, if the workers are also users and do lots of retail trading, they'll be paying to work, effectively (since I have to think that Quantopian will need to profit in some fashion from its "free" trading, if it isn't already). So, it is basically a cost-benefit-risk analysis. The algos are black-box from unknowns around the world. Their performance may not be super-duper, but they are cheap and their are lots of them, so in a relative sense, it is a great deal for "alpha."

One risk, presumably covered by paperwork signed by Quantopian managers, is that they have not utilized information proprietary to others (e.g. covered under NDAs, terms of their present employment, etc.). Or insider information? Eventually, there may be a manager who gets sued/investigated, along with Quantopian/Point72. But I'm guessing the lawyers have thought this all through, and have means to ensure that managers are on the right side of the law, or at least that Quantopian/Point72 aren't to blame.

It would be interesting to hear from Fawce and Steve Cohen on how they are approaching the business and legal end of things, but I'm not gonna hold my breath. Am I on the right track? Or totally off-base?


" Everyone who has read our commentary about the contest knows that we are looking for long/short algorithms with low exposure to the market. The nominal position is a fig leaf, and that fig leaf doesn't cover much. The algorithm is short "enough" to get a badge on the leaderboard, but it is not hedged by any reasonable measure."

Then you exclude 95% of retail quants.

What do you mean by::

Short enough
hedged by any reasonable measure

The only effective hedge is via options or futures. Do you offer that? Long-short is not a hedge.

Anthony FJ Garner and Mikko

The 50/50 or 60/40 strategic allocation in stocks/bonds is the most stable hedge historically. Most other long/short allocations suffer from non-stationarity of statistical properties.

Limiting research to long/short algos with low beta motivated by contest rules severely limits the scope of Q platform.

In my opinion, the only restriction should be low beta. Of course, Q may have its own objectives and this is to be respected.

The so called "Dual Momentum" strategy or the "binary" and 50/50 strategy outlined on my website.

Binary = all bonds / all equity depending on momentum indicator, monthly re-balance. 50/50 same thing but 100% bonds or a max 50% equity/50% bonds.

Tests well since 1753 in the UK and 1870 in the US. Also tests well more recently on iShares Country funds.

But I suspect you mean the more common simple monthly re-allocation to achieve re-balancing back to the relevant proportions?

I share your suspicion of all such strategies but equally as I have often said, the stock index itself is a systematic strategy in its own right. And if the stock index does not work, then I'm not at all sure what one has left?

Dual Momentum

And if you are not a believer in deep learning, ANNs etc take a look at Q "rival" Numerai

I'm certainly going to submit some forecasts however sceptical I might be.


The problem with momentum backtests is that they reflect gains from periods when momentum was not employed and investing was mostly value-based. Therefore, the backtests are unrealistic because they do not account for the impact of momentum strategies on prices but only extract post hoc alpha from value investing effects on prices, which are now basically absent. I am not sure if this is clear.

I looked at Numerai but I have objections about their use of probability since I do not understand to which of the four definitions of probability this corresponds to. See for details:

When I use something along the lines of probability in my p-indicator, that is a deterministic calculation I call a directional bias. Machine learning gives a stochastic calculation with high variance based on training set, algo structure,properties, etc. Each time you run a learning algo you will get a different probability or model for it for same training set. The high variance means that the actual probability is 50%, i.e., a coin toss. This is no different than the odds of classical TA with chart patterns, MACD, etc. It is an exercise in futility. If trading was a matter of probabilities only, all math professors would be very rich. On the contrary, professors have failed, often with a big bang (LTCM) The main problem is the variance and the fact that although probability exists as a limit, we may not have enough data to know its value and actually we do not. Therefore, probability measures, as with my p-indicator, can only be useful in the context of discretionary trading and when considered in specific context, rather than a form of systematic trading. Again, I am not sure this is entirely clear.

Out of sample metrics of disqualified algo:

annRet_pt 20.29%
annVol_pt 7.58%
sharpe_pt 2.48
maxDD_pt -3.37%
stability_pt 0.93
sortino_pt 3.66
beta_spy_pt 3.16%

corr_pt 0.07

Is it not hedged enough by any reasonable measure?

Vladimir, stats are good but it is my understanding from email exchange I had today that they are now looking for long-short algos only, possibly due to requirements from investors. How long was your out-of-sample and how many trades were placed?

Fair point Michael re lack of TF back a few hundred years. However my more recent tests on iShares seem to suggest the idea of switching risk on and off still works well to achieve better risk adjusted returns than the indices. And for investment purposes what are the alternatives? Active fund managers are always long and so DD is every bit as bad as B and H of the indices. Of course if it ceases to work, as it might, you will be faced with the inevitable whipsaw losses which can be devastating. No easy answers other than, perhaps, very wide diversification over asset classes and modes of investment.


How long was your out-of-sample and how many trades were placed?

It was in contest paper trading out-of-sample for 6 month.
There were 54 trades.

Quantopian open contest 22 results

83 (18% ) contestants outperformed SPY, but their average rank is 234.
From top 50 only 4 (8%) beat SPY by return.

Here they are:

☢ ☣ ☢ ☣ ☢ ☣ ☢ ☣ ☢ ☣ ☢ ☣ 33

score_contest_rank 33 score_contest 67.48
Paper Trading
score_pt_rank 74 score_pt 67.48

rankAnnRet_pt 56 annRet_pt 35.13%
rankAnnVol_pt 311 annVol_pt 16.74%
rankSharpeRatio_pt 82 sharpe_pt 1.89
rankMaxDD_pt 250 maxDD_pt -8.11%
rankStability_pt 40 stability_pt 0.88
rankSortino_pt 92 sortino_pt 3.01
rankBetaSPY_pt 180 beta_spy_pt 15.90%

corr_pt 0.01

Richard F 28

score_contest_rank 28 score_contest 69.39
Paper Trading
score_pt_rank 67 score_pt 69.39

rankAnnRet_pt 69 annRet_pt 27.59%
rankAnnVol_pt 296 annVol_pt 15.27%
rankSharpeRatio_pt 102 sharpe_pt 1.68
rankMaxDD_pt 166 maxDD_pt -5.34%
rankStability_pt 103 stability_pt 0.77
rankSortino_pt 83 sortino_pt 3.15
rankBetaSPY_pt 133 beta_spy_pt 9.43%

corr_pt -0.08

Joseph Harwood 18

score_contest_rank 18 score_contest 74.47
Paper Trading
score_pt_rank 44 score_pt 74.47

rankAnnRet_pt 77 annRet_pt 23.18%
rankAnnVol_pt 175 annVol_pt 8.37%
rankSharpeRatio_pt 39 sharpe_pt 2.55
rankMaxDD_pt 93 maxDD_pt -3.46%
rankStability_pt 54 stability_pt 0.85
rankSortino_pt 49 sortino_pt 4.50
rankBetaSPY_pt 308 beta_spy_pt 56.94%

corr_pt 0.11

tamir kogot 24

score_contest_rank 24 score_contest 72.92
Paper Trading
score_pt_rank 55 score_pt 72.92

rankAnnRet_pt 83 annRet_pt 22.12%
rankAnnVol_pt 197 annVol_pt 9.08%
rankSharpeRatio_pt 60 sharpe_pt 2.26
rankMaxDD_pt 135 maxDD_pt -4.50%
rankStability_pt 171 stability_pt 0.62
rankSortino_pt 69 sortino_pt 3.60
rankBetaSPY_pt 128 beta_spy_pt 8.82%

corr_pt 0.07

The third prize got tom yitav with 10.02% return

tom yitav 3

score_contest_rank 3 score_contest 82.12
Paper Trading
score_pt_rank 9 score_pt 82.12

rankAnnRet_pt 139 annRet_pt 10.02%
rankAnnVol_pt 62 annVol_pt 4.07%
rankSharpeRatio_pt 48 sharpe_pt 2.39
rankMaxDD_pt 35 maxDD_pt -1.33%
rankStability_pt 29 stability_pt 0.90
rankSortino_pt 62 sortino_pt 3.85
rankBetaSPY_pt 184 beta_spy_pt 16.37%

corr_pt 0.12

The second prize got Ryan Parker with 6.37% return

Ryan Parker 2

score_contest_rank 2 score_contest 84.42
Paper Trading
score_pt_rank 6 score_pt 84.42

rankAnnRet_pt 175 annRet_pt 6.37%
rankAnnVol_pt 37 annVol_pt 2.68%
rankSharpeRatio_pt 56 sharpe_pt 2.34
rankMaxDD_pt 19 maxDD_pt -0.67%
rankStability_pt 7 stability_pt 0.93
rankSortino_pt 53 sortino_pt 4.17
rankBetaSPY_pt 141 beta_spy_pt 10.73%

corr_pt 0.09

The first prize got Wenkai Zhang with amazing 0.86% return.

Wenkai Zhang 1

score_contest_rank 1 score_contest 86.30
Paper Trading
score_pt_rank 2 score_pt 86.30

rankAnnRet_pt 267 annRet_pt 0.86%
rankAnnVol_pt 10 annVol_pt 0.32%
rankSharpeRatio_pt 33 sharpe_pt 2.67
rankMaxDD_pt 7 maxDD_pt -0.07%
rankStability_pt 84 stability_pt 0.79
rankSortino_pt 14 sortino_pt 6.28
rankBetaSPY_pt 15 beta_spy_pt 0.30%

corr_pt 0.04

Don’t you think that contestants are running in opposite direction?

I agree that they should focus on improving their ranking criteria. 0.86% is close to zero returns.

Antony Jackson, I don't think so. My algo ranked 6 has a sharp ratio = 3.399 , wich is bigger than the 5 algos ahead of me.

From the contest results, I guess the most important things for Quantopian are the low Beta, low Volatility, and the Stability. but what's the point
of your algo if it doesn't make money.

A has 100% CAGR and 50% volatility. B has 10% CAGR and 5% volatility. A and B have the same Sharpe ratio of 2.

There is a widespread confusion about performance metrics vs. metrics useful in statistical inference. Sharpe is not a very useful performance metric. SPY buy and hold total return since inception has 0.5 Sharpe yet a few not data-mined strategies can beat that.

Sigh... The contest is an opportunity for Quantopian to reflect back to the community quantitatively which algos are attractive for the fund. Without this sort of feedback, it is kind of a dud, in my opinion. Presently I'm ranked 53 in out of 590 entries, but what does this tell me? Am I anywhere close to getting anything, or not at all on the path to the land of milk and honey?

It seems that the fund team should chime in, and re-rank according to what they see as potentially (not necessarily) fund-able. Sounds kinda high-touch, but it would only be a monthly exercise.

Antony Jackson,

The highest sharpe ratio has

Philip Waku

score_contest_rank 222 score_contest 63.85
Paper Trading
score_pt_rank 98 score_pt 63.85

rankAnnRet_pt 27 annRet_pt 88.55%
rankAnnVol_pt 330 annVol_pt 17.77%
rankSharpeRatio_pt 1 sharpe_pt 3.69
rankMaxDD_pt 293 maxDD_pt -10.14%
rankStability_pt 115 stability_pt 0.74
rankSortino_pt 4 sortino_pt 7.68
rankBetaSPY_pt 353 beta_spy_pt 78.92%

corr_pt 0.16

The highest sortino ratio has

Andrew Curran

score_contest_rank 244 score_contest 59.83
Paper Trading
score_pt_rank 133 score_pt 59.83

rankAnnRet_pt 4 annRet_pt 1125.43%
rankAnnVol_pt 409 annVol_pt 76.59%
rankSharpeRatio_pt 2 sharpe_pt 3.63
rankMaxDD_pt 316 maxDD_pt -11.44%
rankStability_pt 99 stability_pt 0.77
rankSortino_pt 1 sortino_pt 13.60
rankBetaSPY_pt 416 beta_spy_pt 167.15%

corr_pt 0.09
it is all about beta

The best return has

Yusup Jorayev

score_contest_rank 418 score_contest 23.70
Paper Trading
score_pt_rank 415 score_pt 23.70

rankAnnRet_pt 1 annRet_pt 2789624.36%
rankAnnVol_pt 442 annVol_pt 7943.68%
rankSharpeRatio_pt 300 sharpe_pt 0.03
rankMaxDD_pt 441 maxDD_pt -939.29%
rankStability_pt 440 stability_pt NaN
rankSortino_pt 300 sortino_pt 0.04
rankBetaSPY_pt 439 beta_spy_pt 1391.05%

corr_pt -0.02

The more contest score number the more return.

"it is all about beta"

Maybe you are correct but this tell me nothing. MOM, the neutral momentum ETF lost 18.3% in 2016.

I'd be interested to know how Q look at long/short strategies that keep most, say 90%, of assets in cash-like short maturity bond ETFs in order to decrease volatility while still utilizing the excess "cash" to generate stable returns, utilize long positions with 9% to generate alpha, and hedge the market risk with 1% by shorting SPY. This kind of strategy could have everything they're looking for, ie. it's low volatility and highly stable, generates high sharpe and low drawdown, and has a beta of 0.00. However, it might have large exposure to a single asset (short-term bond ETF), and looks like the short position is only nominal (1%).

Also, instead of shorting SPY, the market risk could be negated by longing an inverse S&P 500 ETF or implied volatility ETF such as VXX. I assume such strategies would not be welcome since they appear to not contain any short positions, despite being hedged?