Back to Community
Good IC but poor quantiles? When to move from research to backtest?

I have created a simple composite factor consisting of the ROIC, debt/equity, and FCF yield (each ranked then equal-weighted). Using Alphalens, the IC is about 0.02 on a 1-2 week time horizon. Based on some of the Quantopian videos I have seen, this is a "decent" IC. The quantile plot looks alright as well; there seems to be a monotonic relationship. I have a couple of questions:

Firstly, how does this reconcile with the sector quantile plots, which look pretty bad? Is it just the case that although the sector quantile plots don't seem to show any discriminatory power, the signal is predictive in aggregate?

Related to the above, is it generally the case that signals with ICs of more than 0.01 (or whatever number you consider "good") also have "good" (i.e monotonic) quantile plots?

Lastly, and this may be much more subjective: what do you normally have to see in an Alphalens tear sheet to encourage you to move into the IDE and start building a backtest? Is it a simple "if IC > 0.01 and quantile plot looks good" decision? In the interests of both time and avoiding overfitting, I don't want to be too trigger-happy with the backtesting.

I have a couple of factors which produce ICs of 0.01-0.03 in the research environment and wonder how I should be proceeding.

Cheers,
Robert

Loading notebook preview...
8 responses

I'm still new to Quantopian and I am still looking at Alphalens to find answers to the same questions you asked in the post.

Here's what I think:

1) There is no much alpha in ROIC. Before combining alphas, you need to make sure that each individual alpha is positive. If you split all companies into 5 quantiles according to ROIC, you will see that ROIC levels are not that different. 70% of companies have ROIC between 0 % and 5%.

2) I'm not sure that ranks are the best way to combine factors. Let's say that not all companies have ROIC due to negative equity. The number of companies is less. As a result, the rank of other factors is higher and factors are not equally weighted. Debt/equity has also the same problem. It would be better to reverse the rank and give a higher score to the companies with a lower rank. For example, like that (-roic).rank(mask=mask). But still, some companies don't have ROIC and Debt/equity due to negative equity. You should probably exclude the same companies from FCF factor. Otherwise, FCF factor is actually overweight.

3) In this strategy, I think alpha comes mainly from FCF factor.

4) Another question is the following - what type of companies do you aim at? Growth stocks or value? I think factors contradict each other. Because FCF is a value factor and high debt/high roic is a growth indicator.

@Vagrams Thanks for your input, these are all excellent points and I completely agree!

However, this factor is not mine – I just copy-pasted from another thread to illustrate my question. Some of my other alpha factors have a similar characteristics, i.e IC ~ 0.02-0.04, alpha ~ 0.05, and decent-looking quantile plots (in aggregate – not so much in the sector grouping). These are the raw factors before any kind of optimisation, e.g tuning z-scores or weightings.

So I'm just trying to get some insight on how people decide when to take a rough factor that seems to contain some signal to the next stage.

@Robert

Regarding sectors, the stocks are not equally split across sectors. So, it makes sense to look at sectors only to make sure if the factor persists across all sectors. And perhaps, some sectors must be excluded.

Nevertheless, I do have the same questions on how to evaluate alphabets further. I hope other members of the community will collaborate on this.

Nice thread, same boat as you guys.
Starting my Quantopian journey and will be extremely interested by feedbacks of experienced members.
Btw do you guys know why quantiles cumulative return charts do not show for periods longer than 1D in the full tear sheet?

@Smallfishmacro

Have you set your periods parameter?

merged_data = get_clean_factor_and_forward_returns(  
  factor=factor_data,  
  prices=prices,  
  groupby=sector,  
  periods=(5, 10, 20)  
)

@Robert

Yes I do, and like you, I do not get long/short portfolio cumulative return chart for periods longer than 1D.
It is really strange because I can see other people's full tear sheet getting them...

@Smallfishmacro

I think it got broken after the last update of alphalens to v0.4.0.
I had them too before the update.

Posted the issue on Github
https://github.com/quantopian/alphalens/issues/380

@Vagrams

Thanks a lot!
Finally, I managed to plot them with the below code if that can help:

from alphalens.performance import mean_return_by_quantile  
from alphalens.plotting import plot_cumulative_returns_by_quantile  
mean_return_by_q_daily, std_err = mean_return_by_quantile(merged_data, by_date = True)  
plot_cumulative_returns_by_quantile(mean_return_by_q_daily['5D'], period='5D')  
plot_cumulative_returns_by_quantile(mean_return_by_q_daily['10D'], period='10D')