Back to Community
How to get the time series output in pipeline

Hi,

I've got a question on how to get the time series output in the pipeline. Attached is my pipeline notebook which i hope to replicate in IDE. One fundamental difference I've found comparing pipeline in research mode and IDE mode is the pipeline output in production mode must be invoked through "initialize" --> "attach_pipeline" --> "pipeline_output". Since "initialize" only take "context" as an input, it won't be possible to use "data.history" in the subsequent "attach or pipeline output" definition. One way I think might solve this problem is by using Custom Factor but looking at definition of "out" as a single dimension array (i.e. out is an empty array of length N), following pseudo code won't work since there is a dimension mismatch. Anyone there might be able to help me. Appreciate it.

class Hist_Close(CustomFactor):
inputs = [USEquityPricing.close,Returns(window_length=2),Sector()]
window_length = 1
def compute(self,today,assets,out,close,returns,sector):
out[:] = [close,returns,sector]

Loading notebook preview...
Notebook previews are currently unavailable.
4 responses

So there are two main differences in Research pipeline and algorithm pipeline.
This is the import in research, run_pipeline doesn't work in algo mode.

from quantopian.research import run_pipeline

So I commented that out, and imported
from quantopian.algorithm import attach_pipeline, pipeline_output

Usually I make pipeline a separate function, and return that function to attach pipeline. like
attach_pipeline(make_pipeline(), 'myPipe')

Then I use pipeline_output in, before trading start
myPipe=pipeline_output('myPipe').dropna()

This loads up all the stocks for ordering.

Clone Algorithm
5
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 592f17242d059f6ac81a0186
There was a runtime error.

hi Eric thanks for your reply. Looking from your example the pipeline output will only output 1 day information on close, return, and sector. Though what i want is to have a time series for this information (e.g. for the past 4 days up until the current backtest date). Perhaps following notebook will clarify what i mean though it won't compile because i can't append a Hist_Close Custom Factor object to the dataframe object

Loading notebook preview...
Notebook previews are currently unavailable.

Finally I've figured out how to get the historical data. The idea is to output any one day information from pipeline and using this information (e.g. the stock universe, sector, etc) to feed into the data.history function. The drawback is you will need to do a bit munging to the resulting frame. Hope this will be helpful for everyone and might be a consideration for a new feature addition as well

Clone Algorithm
10
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 59338c4ff917a2699b62f8fa
There was a runtime error.

I am just curious. What are you trying to do? Why do you need a time series? The data is in a time series for pipeline. I would just change the window length, and make all your computations in pipeline.

For example this will compute the returns over the last 21 days. It uses the times series of the last 21 days to make the computation.

class Returns(CustomFactor):
inputs = [USEquityPricing.close]
window_length=21
def compute(self, today, assets, out, price):
out[:]=np.log(price[0]/price[-1])

Am I missing something?