What's a good combined alpha IC mean score?

Hi,

I'd like to know when it would be a good point to stop doing research and start testing my alphas using the MaximizeAlpha API.
Right now, I've got 5 alphas, with a combined IC mean score of 0.05.

Is this good enough?

7 responses

Is this a 1 day mean IC? If so, I'd say that's quite good, at least in my book. Also is this in-sample, or out of sample? If it's in sample mean IC, I'd check to see how well it holds up on data that your model hasn't seen when you trained it. If it's 1D OOS mean IC, I'd say that's quite good. Others might disagree.

Yup, it's 1-day IC mean.

How do I test for OOS?

This is my pipeline

factor_data = run_pipeline(make_pipeline(), '2013-01-01', '2016-01-01')


and this is how I get prices

pricing_data = get_pricing(
symbols=factor_data.index.levels[1], # Finds all assets that appear at least once in "factor_data"
start_date='2013-01-01',
end_date='2019-01-01', # must be after run_pipeline()'s end date. Explained more in lesson 4
fields='open_price' # Generally, you should use open pricing. Explained more in lesson 4
)


How do I know if it's OOS? Sorry if it's a silly question.

@ Nikhil, based on the information you've provided above one would consider '2013-01-01', '2016-01-01' to be their in-sample test.

If you were to run it for any period(s) before '2013-01-01' or after '2016-01-01' this would be your OSS (out-of-sample) test. Essentially, you'd want to validate or test your IC stats on multiple periods to limit or avoid over-fitting on that one range of dates.

I would recommend segregating your periods into two buckets.
1) '2013-01-01', '2016-01-01' is where you can build / train your factors on (In Sample Testing)
2) '2009-01-01', '2012-12-31' and/or '2016-01-01' , '2018-10-01' is where you would validate/ test (Out of Sample Testing)

Thanks, Daniel. I'll do the second bucket, too.

Slightly unrelated, but for get_pricing, do I have to have the end date all the way up to 2019? Or can it just be a few days after my pipeline end date?

I believe you can set the get_pricing date to a day after your pipeline end date and be okay.

Okay. Thanks, both Joakim and Daniel.

If you want to observe IC decay over N days I think you should get pricing data for N days beyond the pipeline.