In other words, if the strategy is developed using data from 2002-2014, and then I backtest it over 2014-2016, wouldn't that be sufficient?
Let's say you hold out 2014-16 as out of sample. Then the strategy doesn't work when tested out of sample on 14-16? In this case 14-16 is now no longer out of sample, and you'll need to wait for more data. In practice ideas almost never work on the first try, and so it might be smarter to hold out 4 x 6 month periods to use as independent out of sample tests. Of course issues with sample size and time periods being heterogenous prevent a perfect solution here.
Additionally, how many strategies have you developed that use data from 01-16? What is the probability due to multiple comparisons bias that one of these strategies looks good by chance historically? In practice waiting for new data to come in, much like doing a new experiment in a clinical trial, is the best way to validate your model. Like I said, if you keep a rotation of 6 models or so, you'll usually have one to test that's just popped up as passing the out of sample 6 month mark, and the time in between can be spent trying to improve the model and using techniques such as cross validation to minimize the likelihood it's overfit to your historical window.
Also, if trading is in slow motion (e.g. weekly), as would be dictated by using daily bars, then I'll only have a handful of trades in 3-6 months. So, it seems that I would still be relying quite heavily on an out-of-sample period that includes prior years.
This is basically small sample size. Infrequently trading strategies take way more out of sample to evaluate. In practice you want to have a mix of frequencies in your models so that you're not always waiting forever for a low frequency strategy to trade enough you can be sure it's working.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.