Back to Community
Questions of getting the current data from pipeline

Hello community, I have a self-serve data that I want to use in my algo. Below is the top of the data:

date             ticker          score  
12/3/15 MTSC    0.538719768  
12/4/15 WAFD    -0.501787895  
12/9/15 HZO 0.501683899  

I made a pipeline to pass the latest data (current day) in before_trading_start function and assign that to context.output, and I have a print statement in my rebalance function which happens every day.

Start from 12/3/15, it's supposed to be 1 record, 1 record, and 0 record until 12/9/15. However, what I got from the print is as following:

2015-12-03 09:35  PRINT                        score  
Equity(5107 [MTSC])  0.53872  
2015-12-04 09:35  PRINT                         score  
Equity(5107 [MTSC])  0.538720  
Equity(8162 [WAFD]) -0.501788  
2015-12-07 09:35  PRINT                         score  
Equity(5107 [MTSC])  0.538720  
Equity(8162 [WAFD]) -0.501788  
2015-12-08 09:35  PRINT                         score  
Equity(5107 [MTSC])  0.538720  
Equity(8162 [WAFD]) -0.501788  
2015-12-09 09:35  PRINT                         score  
Equity(5107 [MTSC])  0.538720  
Equity(8162 [WAFD]) -0.501788  
Equity(18917 [HZO])  0.501684  

Looks like the pipeline output is a cumulated instead of what I described above. Did I do something wrong?

This is the code that I used:

import quantopian.algorithm as algo  
import pandas as pd  
import numpy as np  
from quantopian.pipeline import Pipeline  
from quantopian.pipeline.data.builtin import USEquityPricing  
from quantopian.pipeline.filters import QTradableStocksUS  
from quantopian.pipeline.data import placeholder as sample

def initialize(context):  
    schedule_function(my_rebalance,  
                      date_rules.every_day(),  
                      time_rules.market_open(hours = 0, minutes = 5))  
    algo.attach_pipeline(make_pipeline(), 'my_pipeline')  
def make_pipeline():  
    base_universe = QTradableStocksUS()  
    pipe = Pipeline(  
        columns = {  
            'score': sample.score.latest  
        },  
        screen = (base_universe & sample.score.latest.notnull())  
    )  
    return pipe

def before_trading_start(context, data):  
    context.output = algo.pipeline_output('my_pipeline')  
def my_rebalance(context, data):  
    # Get current date data  
    df = context.output.copy()  
    print(df)  
2 responses

Nobody has the same issue has me?

The output you are seeing is what would be expect. Two things to keep in mind with self serve data:

  • Data is nan before it is assigned a value
  • Data is forward filled after it has been assigned a value

Any security which hasn't been assigned a value in the self serve data file will have a value of nan for the associated field. On 2015-12-03 the only security that is assigned a value for 'score' is MTSC. All other securities will have a value of nan for 'score' on that date. In the above example, the pipeline is screened for 'sample.score.latest.notnull()'. Therefore, the only security in the pipeline will be MTSC with a 'score' value of .538719768.

Self serve data is forward filled. Pipeline will output the last known value of a users data. On 2015-12-03, MTSC is assigned a value of .538719768 for 'score'. This will be the value returned by the sample.score.latest method from that date onward. If you want a different value on a different date, your data file will need to update the value.

Also, not sure if it's an issue, but self serve data is never adjusted for splits or dividends. The data a user enters is presented unmodified to the pipeline.

Take a look at the self serve data docs for more on how self serve data is handled (https://www.quantopian.com/docs/user-guide/tools/self-serve#considerations)

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.