Thanks for your comments, and I think you have done a great job in your preceding posts, both in answering some questions and also in raising some interesting new ones! I will now try to link my interjection to your previous train of thought.
In your first notebooks you have certainly done exceptionally well in eliminating systematic risk (or the risk associated with "common returns" as it is called in Q's model). As you say, it was very well received by Q and this implicitly answers a question that I raised in a separate post asking if the goal is intended to be the maximization of "specific (i.e. non-systematic) returns" ? Although Q has not responded to my question (yet), apparently the answer seems to be yes, as you have effectively demonstrated, and I agree with your comment about systematic risk and its impact for a fund running multiple portfolios.
With regard to high Sharpe ratios (or any other quality metrics), I guess all we can say is that high values are "probably a necessary but almost certainly not a sufficient" condition for a good system. What we get matters less than how we got there!
As algo developers, Q has given us a precise set of constraints, a single objective function to maximize (namely one specific function of Risk Adj Rtn), some good general guidelines, but then Q seems consistently to avoid answering various other questions. Sometimes I find that Q's lack of direct answers to direct questions seems frustrating, but maybe there is an underlying reasoning behind it. Although there are other possible explanations, presumably Q's intention is simply to try to encourage as much algo diversity as possible, even though it can be frustrating to so frequently have to "tease out" information by inference.
For example, your comment: " What I have since learned ... is that they would like us to use as much data as possible". Just FYI at the Quantcons in Singapore for the last 2 years, Delaney offered (approximately) the following comment to people who aspire to win an allocation, saying that he had THREE hints for them, namely:
- Don't only consider price data,
- Look at Alternative data, and
- Use non-pricing data.
With regard to data hold-back, splitting data sets, and avoiding over-fitting, there are lots of different ways to do it (e.g. your Baysean approach, the webinar & elsewhere), and also in acknowledgement to @LeoM 's excellent comment: "I wanted to elaborate on that because I think that goes to the core of strategy development. Are we accounting for risk exposures in a way that the strategy is balanced in all market regimes without knowing which market regime it is currently operating in".
Sometimes we don't even know what regime the market is currently in. And of course we don't know what the market will throw at us in future. Will it be anything like what we have already seen at some time in the past? Will it be nothing like what we have ever seen before in THIS market but nevertheless maybe something similar to what has been seen in some other completely different market? Is data from other unrelated markets just a useless diversion, or is it in fact a plausible analogue for what MIGHT happen in a possible future market regime in our market, even if never seen before in our specific data set(s)? And of course on the other hand we always have the question of how really can we avoid, or at least minimize, the adverse impact of over-fitting or data-mining bias, or whatever else we call it?
I have 2 ideas that I come back to in my own personal system development outside of the Q context, namely:
1) Consider possible use of ALL data, from all real markets, everywhere, over all timescales and all time periods. Why? Because this is the only way to get a look at the full spectrum of possible market regimes that our system might need to be prepared for in an always-unknown future.
2) Use NO actual historical data series at all. Why? Because this is the only way that our system can avoid all the various forms of over-fitting. Base the system entirely on logic only, and avoid anything that looks like data mining in any way.
Neither of these are "conventional" approaches, and perhaps you might be skeptical, but personally I have benefited from at least considering the key aspects of both of these rather extreme ideas.
Cheers, all the best from TonyM
Looking forward to more interesting & practical discussions with you.