Has anyone ever had such a MASSIVE discrepancy between in sample and out of sample results? (click on view notebook)
Annual Returns: ALL = 669%, IS = 531.6% OOS = 3,463.7%
This is a dollar/market neutral strategy that returns 12% - 18% annually un-levered. It is designed for substantial capacity in excess of several yards, and leverage more reflective of what hedge funds typically get for this type of portfolio construction.
I checked stocks that may or may not have been available to verify data_can_trade screen.
This one's a real head scratcher. The purpose of validating using out of sample data, specifically as it applies to machine learning, is a sort of sanity check. With this model, the AI dynamically tunes factor exposures (among other things), based on a forward prediction. So for me, because I use supervised machine learning, validation can be accomplished by taking a slice of the data set and not allowing the AI to use it for training and model development. Alternatively, validation can be achieved by running a live forward test. In this specific case, I have used both methods. The forward test (simulation results) are in line with the OOS results.
I see some discrepancy sometimes, but generally, I'm looking for results b/w IS and OOS to be somewhat close, NOT on different planets.