We spent the last several days reviewing the leaderboard results at the end of Contest 14. Our process is pretty straightforward. First, we run the leaderboard-generating code and review the resulting file. We look at the file to make sure that there are no obvious unexpected outcomes and investigate anything that looks suspicious. Then we run tearsheets on the leaders. We review each tearsheet to see if there is anything obviously wrong. (Just about everything also gets evaluated for an allocation, but that process is separate from the way we declare a winner).
What we found this month in the tearsheet was unexpected. There were a couple of entries (not the leader) that weren't following the rules as we intended them to be applied. We found that a few entries were not hedged - they were holding only a nominal short position. Take a look at this snippet from one of the tearsheets. You can see that the algorithm is a long-only algorithm. If I didn't tell you that the short position was slightly greater than zero, you wouldn't be able to tell from this graph:
The relevant section of the rules says: "Your algorithm must be hedged to the market. It should hold both long and short positions simultaneously, or be entirely in cash. Hedged strategies reduce their market risk and correlation risk to individual positions." The entry you see in the tearhsheet certainly isn't consistent with the spirit and intent of the rules. Everyone who has read our commentary about the contest knows that we are looking for long/short algorithms with low exposure to the market. The nominal position is a fig leaf, and that fig leaf doesn't cover much. The algorithm is short "enough" to get a badge on the leaderboard, but it is not hedged by any reasonable measure.
We talked about the situation for a while. On one hand, there are a couple of contestants who think they had valid entries and think that they should get a prize. If we disqualify them, they will be understandably frustrated. On the other hand, we have a couple of contestants who built the kind of high-quality algorithms that we are looking for; if they don't get a prize, they would also be understandably frustrated. We've come up with a solution that we think solves the problem most equitably.
- We will award prizes to two sets of winners this month. The entries with a nominal short position will get prizes. And, the next-best entries follow the spirit of the rules will also get a prize.
- Going forward in Contest 15 and beyond, these unhedged entries, the entries with only a nominal short position, will be disqualified.
I believe that everyone will see the care and consideration we've put into this solution. We apologize for any frustration or confusion that it may cause. We think that this path is the best, balanced outcome that is available to us.
Edit: I wanted to clarify that we were concerned with the 2nd and 3rd place entries. The 1st place entry, Frederik's, was well-hedged.