I'll be honest that I'm new to Quantopian and wanted to (shamelessly) use it to teach myself Python with no real regard for trying to find a winning algo. I work in credit risk modelling so know little to nothing about equities. However I got the bug, a single idea, so thought I'd give it a go. But now I think I'm stuck.
Within Research I've written code to build a dataset of history, which is then periodically updated (say every week, month, etc - not really decided yet), and then "mined" to find correlated, cointegrated, and multivariately normally distributed groups of securities that appear mispriced. The idea is that they will converge. I was hoping to copy this approach to the Algo API where there would be one big bit of pre-processing at the outset, appending data each week/month (as correlation, cointegration and multivariate normality testing results will change periodically), and mining for mispricing each day. Given the periodic appending and reprocessing I was hoping it would update itself ala Machine Learning.
However, from a look in tutorials, it looks like my Research pipelines employing start and end dates won't work in Algo API so I'm a bit stuck. Have I missed something quite simple or have I properly snookered myself (for those who don't like snooker read - screwed :)).
I've attached my research code (note at the end long=short and short=long - it was late at night [I've got a newborn and a 1 year old who seldom sleep] and I forgot to reverse the labels). Sorry it isn't pretty but hopefully highlights how there is a lot of data mining that I hope to automatically refresh (so machine learning I think) that I want to copy to ALGO.
Any help would be greatly appreciated.