Grant - on the first one you posted, the biggest point to note is that it hasn't made any money during it's out-of-sample period. That could be just a natural drawdown period, in which case it will recover and becoming interesting. Or, it could be that the algorithm had some overfitting, and it's never going to make it. There's no way to know without waiting. Kayden and Frank's analysis was similar.
The second note on that one is the large position in QQQ that reverses repeatedly. That's not disqualifying, but it raises some questions that we'd be asking if you were in the due diligence process. There are times when filling in a hedge with a large ETF is a reasonable move, but it's also a warning sign of overfitting. When there is a single equity with a large position in the porfolio, the risk is that during development it was inadvertently "tuned" into being overfit.
Harrison's point is good - 250 is better than 100 is better than 50 - but a good algo with 50 is perfectly fine.
Anthony - We're not changing our business model. We're looking to our community to write high quality algorithms, to attract investor capital with those algorithms, and to compensate the algorithm authors. It's the core of our business, and we try to repeat that everywhere we talk about our business.
Michael, I understand that some quant shops use the division of labor that you describe. We aren't doing that exact division, but we do have one. At Quantopian, on a simplified level, we have one team that is building the platform, one team that is testing and making allocations of capital, and one team that is doing the trading. We're asking the community to do the idea generation, research of the ideas, and the coding of the ideas.
Steve Cohen: There have been several mentions of Steve Cohen in the thread, so I'll say a little bit about our relationship. He agreed to be our first customer, which we are very excited about. Through Point72 Ventures he purchased a relatively small fraction of Quantopian's stock. His organization has made some introductions for us to vendors and reporters and the like, for which we are also quite grateful. On the other hand, neither he nor his organization have a role in which algorithms are selected for allocations.
Grant, we don't have a great way, post-hoc, to figure out your sector exposure. It's something that we can do, and will in the future. It is possible to control your sector exposure in the algorithm using the fundamentals database. It defines sector codes. All of the early techniques were pretty manual, but the optimization API makes it a lot easier.
Grant, you added a phrase to Fawce's quote that does not need to be there. The original quote is "As long as we find algos that are marginally better than random, we can combine them into something compelling." It is correct as written, and does not need to be qualified with "constructed by a team of professionals" or anything else. Writing an algorithm that is consistently better than random is very difficult, but possible. It doesn't require a team of professionals. If one can combine many uncorrelated strategies with a positive sharpe ratio, the resulting portfolio will have a higher sharpe ratio. That's the point that Fawce is making.
Grant, on your 2nd tear sheet, a few notes:
- two months of out of sample look good - more is needed,
- a lot of the total return is from 2008, and that is presumably idiosyncratic. 1) if we take that out, how does it look? Is it still interesting, or did it lose too much of the returns? 2) If there is an idiosyncratic good thing, it presumably is equally possible to have an idiosyncratic bad month in the future. That would need to be investigated.
- a 658-day drawdown is a loooooooong time to wait for a recovery. Hopefully that would also be resolved by whatever investigation happens into 2008.
- There is a large SPY (50%) position which flips back and forth. Same concern I mentioned above.
- There are many large positions in other stocks (20%), like NFLX and TSLA. Again, we'd need to understand the rational for that. Our null hypothesis with a tearsheet like that is that there is overfitting going on (unintentional, I'm sure), and we'd need to be convinced there is an underlying reason driving those positions.
I think Antony's questions are well-stated. If one starts with a hypothesis, a factor, one then needs to be sure it's not actually a proxy for some other risk. If you tackle it from that direction, you aren't stuck at the end, wondering if you have something real. At the end you'll have a well-tested hypothesis and the code to back it up.