Regarding the multiple comparisons bias, I don't yet have an intuitive feel for it. Say I have a box of 1000 toothpicks and I want to find pairs that have nearly the same weight and length, to within specified tolerances. I have a weighing scale and a measurement microscope. I roll up my sleeves and start making random pairwise comparisons. After 1000*999/2 = 499,500 comparisons, I'm done, and have a list of pairs of toothpicks (I forgot to mention, prior to my evaluation, I had each toothpick laser-engraved with a serial number). So where does the bias come in?
I guess the idea here is that if I have some additional information about the toothpicks, aside from weight and length, then I might be able to use it to improve my assessment. For example, if the toothpicks are colored (for the holidays), red and green, then I might hypothesize that the red toothpicks were made in a red toothpick factory, and the green ones in separate factory. In this case, clustering by color might help improve my assessment, and reduce the number of spurious pairs. But then, if the toothpicks are all made in the same factory, and the coloring process does not affect the weight and/or length differently by color, then I could be claiming I'd discovered an improved technique for toothpick pairing, but done nothing substantive.
For Jonathan's analysis, I'm thinking that the hypothesis is something like "All of this fancy analysis works, and reduces the number of spurious cointegrated pairs, over a brute-force approach." So, it would seem that one would actually need to do the brute-force analysis, and then figure out how to compare it to the proposed technique, to determine if it is beneficial. It may be "the bomb" but to me the advantage is unclear (other than perhaps reducing the problem down to being more computationally tractable with the resources available).
A pairs algo is very well suited to a Q fund allocation.
Intuitively, like other well-known quant techniques using readily available data (e.g. price mean reversion), I'd think that there would be pretty slim pickings out there. My assumption is that for decades now, hedge funds have had big honkin' computers churning away at the problem, squeezing out the alpha from pairs trading. I don't have any industry experience, but it doesn't seem like Quantopian would have any edge in this area. That said, perhaps as an incremental alpha factor in the multi-factor grand scheme described in Jonathan's blog post, one could roll it into the mix, just for yucks. Would it be feasible to formulate pairs trading as a general pipeline alpha factor?
As a general comment, I'd concur that the messaging on what is most likely to get an allocation is muddled. On the one hand, we have the directive to use "alternate data sets" and on the other, guidance that pairs trading, potentially only using company fundamentals and price-volume data would be attractive. And also guidance that multi-factor algos are desirable (but perhaps only if all of the factors are based on alternate data sets?). Personally, I don't want to spend the next 6 months developing a pairs trading algo (or alpha factor, if that is feasible), and then another 6 months paper trading it, only to hear that my odds of getting an allocation are slim due to the strategic intent not matching the requirements (this is the message I think I got regarding price mean reversion strategies and possibly even price mean reversion alpha factors...I'm not sure). In some sense, an advantage Quantopian has over traditional hedge funds is that they could ignore the strategic intent altogether, and just base their assessments and allocations on the black-box algo performance. This would eliminate the risk of herd mentality or personal biases on the part of the the fund team. Just go with the data.