When to combine factors, and when not to

There has been a lot of confusion on the platform, specifically relating to the challenges, on how and when to combine factors. Tl;dr: a non-additive combination of factors of different styles or data sources is highly encouraged. However, we prefer to do additive combinations of factors of different styles ourselves.

Let's say you are building a factor for our insiders challenge. It's okay to submit a factor that is the aggregate of individual factors. But, as the rules state, all individual factors need to be based on insider data. However, this does not mean that insiders is the only data source you are allowed to use for your factor.

So then, could you have a factor that uses multiple data sources — insider and estimates, for example? The answer to that question depends on how you combine them. If the combination is additive (like an average or weighted average of the two), then no, because that's like combining two different factors with assigned weights — it would not be "pure" anymore. In the extreme case, you could just set the weight of the insider contribution to 0.001, fulfill the technical criteria of having an insider-based factor but its contribution is effectively zero. If, however, your combination of these data sources is non-additive with an economic rationale (like an if-else statement, or an interaction) then combination is strongly encouraged. For example, if you think that if insider and estimate sentiments disagree it is predictive, then that is an interaction with economic rationale because you would multiply (rather than add) the positive insider and negative estimate sentiments to measure the divergence between insider and estimate sentiments. This is a type of legitimate alpha that does not come from diversification but from an insight you had — it would be difficult to recreate your factor by just combining an insider and estimates factor together.

In the end, we would like to emphasize that this post is not to suggest you brute-force non-additive ways to combine data sources without economic rationale. How to combine different data sources should always serve your alpha ideas.

If you then have multiple of these insider factors (that are either based on insiders data alone or on non-linear combinations of insiders and other data sources) it is fine to combine these additively, because they are all insider-based (same style) and you'll have one, diversified insider factor.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

11 responses

What about factors with different information horizon? How would you go about combining them? For example, say I have a factor that is designed for 5 day forward prediction and another is 10 days. To trade say the first one in production, I would need to divide my equity in 5 subportfolios and just take trades in each portfolio and hold for 5 days . The tricky part is what if I want to introduce a factor that does 10 day forward predictions? Any have any materials or ideas to tackle this problem?

Are linearly combined factors always just the sum of their parts? For example, will a factor1.zscore() - factor2.zscore() factor return the average between what a factor1.zscore() factor by itself and a -factor2.zscore() factor by itself would return?

Lets say for the sake of example that factor1 is beta and factor2 is volatility. If you had an market hypothesis that a low volatility, high beta portfolio would generate alpha, this would only work if the individual factors are also predictive enough on their own to survive being averaged together? In other words, if a beta factor generates no return and a volatility factor generates no return, then it's impossible that a linearly combined "low volatility, high beta" factor will generate a return?

For some reason I've always assumed that novel combinations of factors could produce alpha not present in the individual factors. Just trying to wrap my head around this one.

@Mike --

combined_factor = (
SimpleMovingAverage(inputs=[factor1.zscore()], window_length=5)
+ SimpleMovingAverage(inputs=[factor2.zscore()], window_length=10)
)


But it seems like that's the type of thing they're discouraging. Just submit them as two separate algorithms.

@Mike: If you apply a moving average to the 5-day factor it will have the effect of the subportfolios. Viridian Hawk provided the right code.

@Viridian: Not necessarily. The key point is the style of the factor, that should be somewhat "pure" (it will never be completely even with non-additive combinations). But combining factors of similar styles together is a great way to make your factor more robust. For example, you could have the same factor idea but with 3 different look-back windows. Combining those three (likely very correlated ones) together could very well increase the robustness of your final factor, but it's not something we are looking to do necessarily. We are mostly looking for factors that are uncorrelated to each other, but in order to get a good sense of the actual correlation the factors need to be "pure". For example, say we have an estimates and an insiders factor. Then you take those very same factors and combine them. To us that might look uncorrelated to what we already have while it's actually just more of the same.

Also, I changed my language from earlier to talk about additive vs non-additive, because technically interactions (multiplicative) is still linear.

For some reason I've always assumed that novel combinations of factors could produce alpha not present in the individual factors. Just trying to wrap my head around this one.

No, that's a key point: If you just combine two things additively together there is no alpha in that combination, the only improvement comes from diversification. But with access to all algorithms it's much more effective for us to pick the most diversifying algorithms than it is for you. And once you do it we can't undo it.

the only improvement comes from diversification

I think noise-cancellation is another improvement:

Factor 1 weights:

TSLA 0.10
AAPL -0.90

Factor 2 weights:

AAPL 0.90
OSTK -0.10

Combined (Factor 1 + Factor 2) weights (normalized):

TSLA 0.5
OSTK -0.5

Returns:

TSLA +12%
AAPL -0%
OSTK -10%

Combined Factor Performance:

Factor 1: +1.2%
Factor 2: +1%
Combined factor: +11%


So, you mean you can never generate more alpha by combining factors in an additive way like this?

    factors = make_factors()
combined_alpha = None
for name, (f,w) in factors.items():
if combined_alpha == None:
combined_alpha = w*f(mask=universe)
else:
combined_alpha += w*f(mask=universe)

# make factors would create different CustomFactors and each one would be normalized inside the compute function.
# we assume weight (w) is 1 for all factors


I guess neither multiplying then? I'm not sure I'm understanding it properly.

I was actually asking something similar in this post

Thanks in advance.

@Marc: Correct, the only benefit you get from addition/averaging will be due to diversification. Multiplication, however, can provide alpha beyond that as it's possible that you have two factors that by themselves don't have alpha but once you multiply them they actually become predictive. For example, a factor where only if two indicators are both either positive or negative, the stock goes up, but if they disagree (one positive, the other negative), the stock goes down. You can't infer anything about the stock using only one of the two indicators, they are dependent on each other.

Hi @Thomas,

I understand then that you would multiply before zscoring or normalizing right? Otherwise you would end up messing with the signs...

Best,
Marc

@Marc: No, you want the sign flip like in the example I outlined so you want to do the multiplication after z-scoring.

@Thomas, wouldn’t this be dangerously close to data mining and data snooping? I would find it difficult to rationalise (after having looked at the stats) why two factors that aren’t predictive on their own, but when multiplied together (and sign flipped) the product is all of a sudden predictive. That to me sounds like I’m just ‘torturing the data long enough to make it comply.’

@Joakim: Yes, thanks for pointing that out, I would definitely not advise doing this without a strong economic rational. I think there could be legit cases where an interaction effect could be at play, however. But yes, not advising doing this blindly on any old unpredictive set of factors.