Attached, please find a tear sheet for one of my contest algos. Per https://www.quantopian.com/contest, it was "Created at 2018-03-09 6:24:19 am" so results after this date are out-of-sample (I think there is a way to show this on the tear sheet--if anyone knows, please advise).
- If the performance continues, might it be attractive for an allocation? Or is it a dud based on current, or anticipated new requirements (we've seen some dramatic changes over the years)?
- If the answer to #1 is "yes" then would a 2-year backtest plus 6 months of out-of-sample data be sufficient to make a decision? Or would a longer backtest, plus possibly more out-of-sample data be required? Or would I be done?
- Are there any results on the tear sheet that indicate the need, or opportunities for improvement?
- Aside from the tear sheet, are there other analyses that could be applied at this point to provide actionable feedback?
- I do not have a "Strategic Intent" for the algorithm (per the requirement on https://www.quantopian.com/allocation). The algorithm has more than one factor, and I have not attempted to suss out the relative contributions of each, nor do I have a "hypothesis" for each that I have tested. If the algo has legs, I could try to piece together a story, but I'm unclear what would be required (e.g. a slide pack? report?). Perhaps an example could be provided of the expected deliverable?
- I would permit a limited number of Quantopian employees to view the code, and make specific recommendations for minor changes. One benefit of this approach would be that perhaps treating the algo as completely new could be avoided; if changes are understood and minor, then the risk of introducing bias ("over-fitting") could be mitigated, and an additional 6 months of out-of-sample testing could be avoided.
- To what extent might the limitations of the backtester give misleading results for this algo? For example, in light of the problem raised on https://www.quantopian.com/posts/short-selling-in-backtester-time-for-improvement-1, and the stocks traded, might there be problems when going live with real money? Aside from shorting, is there anything else that might lead to lower performance when trading, relative to the simulation (other than "alpha decay" which is not a problem with the accuracy of the simulation)? Generally, are simulation inaccuracies taken into account when assessing algos for an allocation? If so, what are the risks for this algo, and what penalty would be assessed in determining its allocation-worthiness?
- Is there a representative licensing agreement available for review? Putting a lot of effort into this, and then finding out I would not want to sign (or could not sign, for some legal reason) would be a bummer.