I have been reading about the methods of parameter optimization here on Q Forum and using the links people provided. For example, this discussion is very instructive
Among other things, Dr Ernest Chan made a very good point that testing parameters of trading signals using a limited number of real time series invites overfitting
But his way around it - simulate unlimited number of time series and do parameter optimization using theme - in my mind invites another problem - do we really know how to simulate time series? And even if we do a good job for modelling the mechanisms which were driving prices say in the last 20 years would these models (eg AR, GARCH etc) do a good job in explaining the future time series for the next 20 years? For example if the demography is changing and affects the market an economy in nonlinear fashion. Also Chan's idea seems to be designed primarily for price based, non fundamental signal based, trading signals.
Here is the question for the forum folks:
do you think that the best we could do with backtesting and a cautious approach (in-out of sample) to overfitting is to discard obviously crappy trading strategies and keep the ones which may or may not work but we would never be able to really solve the issue of parameter optimization for the future time series due to unpredictable mechanisms driving the future price series? Or we can do better than that?
Any thoughts and references are welcome in the thread :) I guess it could be a good collection of ideas on the subject for the future