Back to Community
date is not correct in the prices i get

hi

can any one explain why the open and close prices i get when running the pipeline is not for the right date?

it looks like everything is shift one day ahead... for 05.11.2019 i am getting prices for the 04.11.2019

is is something to do with my timezone?

Loading notebook preview...
Notebook previews are currently unavailable.
3 responses

The data isn't really shifted one day ahead and it's not because of the timezone. The apparent 'shift' is is due to the interpretation of the dates when using pipeline data. Pipeline dates are the dates on which the pipeline fetched the data and not the data 'asof date'. By definition, pipelines fetch data every day before the market opens. Therefore, the open and close price data fetched today (before market open) would be from yesterday and be yesterday's open and close prices.

Why do it like this? It's primarily semantics. The majority of data fields which pipeline fetches are fundamental data. It's natural to refer to fundamental data as 'what fundamentals do I know as of today'. There are often many different 'asof_dates' for fundamentals such as the last quarter report filing date, the last earnings estimate release date, or the most recent analyst estimate update. However, the only one which typically matters is 'what would I have known as of today before trading starts'. That is the pipeline date. Ultimately, this all ties into preventing any lookahead bias in algorithms. Only data which would have been known on a given day is presented to an algorithm. As an example, at the beginning of today one doesn't know what today's close price will be.

Just think of the pipeline data as 'before trading on this day this is the data that could have been known'. For pricing data that will always be the previous trading day.

There are several posts in the forums here regarding this same question. Maybe take a look at this one https://www.quantopian.com/posts/pipeline-and-getpricing-price-shift

Hope that helps.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi

thank you Dan for the explanation!!!, i have a follow up question:

lets say i create a pipeline with some fundamental & technical calculation witch returns each day different stock's (from big universe)

now i want based on lets say first half hour of trading (gap/high/low etc...) on each given day decide to buy/sell stock from this pipeline.

is it possible?will the dates from get_pricing() and from pipeline match?

can the function handle_data() take assets from my pipeline? , can someone help me with "skeleton" of such algorithm?

One can easily add current data to the dataframe returned by pipeline. Since the pipeline output is a pandas dataframe and the results from the data.current method can be a pandas series, they play nice together. Something like this should work in most cases:

    # First get our data as of market open from pipeline  
    stock_data = pipeline_output('my_pipe')

    # Next, get current prices for all the securities in the pipeline and add as a new column  
    stock_data['current_price'] = data.current(stock_data.index, 'price')  

Once all the data is neatly in a single dataframe (both previous data fetched using pipeline and current data fetched using data.current), one can perform whatever logic one wishes. I like using the query method along with either nlargest or nsmallest to first select the pool of potential securities and then to take a set number of the 'best' to actually trade. There are a lot of ways to do the logic but here is one approach

    # Create whatever logic to choose longs and shorts  
    # Note that anything currently held but NOT in either of these will be closed  
    longs = (stock_data.  
             query('(current_price<yesterday_close) & (rsi<30) & (yesterday_close<sma_10)').  
             nsmallest(context.TARGET_LONGS, 'current_to_sma25_price').  
             index.tolist()  
             )  
    shorts = (stock_data.  
             query('(current_price>yesterday_close) & (rsi>70) & (yesterday_close>sma_10)').  
             nlargest(context.TARGET_SHORTS, 'current_to_sma25_price').  
             index.tolist()  
             )

A couple of things. First, if one uses current data then at least some of the trading logic needs to be outside of the pipeline definition. Pipeline definitions only apply to pipeline data and not current data. Moreover, one cannot really analyze such a 'factor' (or strategy) using Alphalens. Alphalens is premised on the assumption that all trades would be decided on before the market opens and then traded at market open. Using current days data doesn't fit this model. This approach is fine for backtesting just not for Alphalens.

So, "is it possible? will the dates from get_pricing() and from pipeline match?". Yes, in an algo, the pipeline returned will be all the data one knows just before market opens. One can add more recent data. This is also true in a notebook as long as one interprets that data as yesterdays data. It's often helpful to explicitly name pipeline prices as yesterday_close or previous_close. As a sidenote, get_pricing only works in notebooks. The similar method to use in an algo is data.history or data.current.

Also, "can the function handle_data() take assets from my pipeline?" . Yes, the pipeline output dataframe is indexed with the assets. One can get a list of those assets as follows

    #Get our data from pipeline  
    stock_data = pipeline_output('my_pipe')

    # Get the index and turn into a list. These will be the pipeline assets.  
    my_assets = stock_data.index.tolist()

Most methods will also work directly with the asset index so there may not be a need to always turn it into a list.

The attached backtest shows this approach in action. It could work as a "skeleton" for a working algo. Good luck!

Clone Algorithm
2
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
This is a sample algorithm on Quantopian using the order_optimal_portfolio.
It also shows how to combine current data with pipeline data.
One can also either use the plain order methods or the optimize methods.
"""

# Import necessary Pipeline modules
from quantopian.pipeline import Pipeline
from quantopian.algorithm import attach_pipeline, pipeline_output

# Import specific filters and factors which will be used
from quantopian.pipeline.filters import QTradableStocksUS, StaticAssets
from quantopian.pipeline.factors import RSI, SimpleMovingAverage

# Import datasets which will be used
from quantopian.pipeline.data.builtin import USEquityPricing

# import optimize
import quantopian.optimize as opt
 
# Import pandas
import pandas as pd

def initialize(context):
    """
    Initialize constants, create pipeline, and schedule functions
    This uses the default slippage and commission models
    """
    # Set the number of stocks we want to trade and our weights (evenly weighted)
    context.TARGET_LONGS = 50
    context.TARGET_SHORTS = 50
    context.TARGET_STOCKS = context.TARGET_LONGS + context.TARGET_SHORTS
    context.TARGET_WEIGHT = 1.0 / context.TARGET_STOCKS

    # Make our pipeline and attach to the algo
    attach_pipeline(make_pipeline(context), 'my_pipe')

    # Place orders usig optimize
    # Can change to 'place_orders_using_order_method' for comparison
    schedule_function(
        func=place_orders_using_optimize,
        date_rule=date_rules.week_start(),
        time_rule=time_rules.market_open()
    )
    

def make_pipeline(context):
    """
    Define a pipeline.
    This defines the data which our algo will use
    
    We break this code out into its own function to make it easier to
    test and modify in isolation. In particular, this function can be
    copy/pasted into research and run by itself.
    Parameters
    -------
    context : AlgorithmContext

    Returns
    -------
    pipe : Pipeline
        Represents computations we would like to perform on the assets
    """
    my_universe = QTradableStocksUS()
    
    # Create any needed factors.
    yesterday_close = USEquityPricing.close.latest
    rsi = RSI(window_length=10)
    sma_10 = SimpleMovingAverage([USEquityPricing.close], window_length=10)
    sma_25 = SimpleMovingAverage([USEquityPricing.close], window_length=25)
  
    # Create our pipeline
    pipe = Pipeline(
        columns={
            'yesterday_close': yesterday_close,
            'rsi': rsi,
            'sma_10': sma_10,
            'sma_25': sma_25,
        },
        screen=my_universe,
    )
    
    return pipe  
        
def get_longs_and_shorts(context, data):
    """
    Fetches pipeline data and adds current price data
    """
    # First get our data as of market open from pipeline
    stock_data = pipeline_output('my_pipe')
    # Next, get current prices for all the securities and
    # add as a column in our pipeline output along with a calculation
    # Remember that the dataframe index is all our securities
    stock_data['current_price'] = data.current(stock_data.index, 'price')
    stock_data['current_to_sma25_price'] = stock_data.current_price / stock_data.sma_25

    
    # Finally, create whatever logic to choose longs and shorts
    # Note that anything currently held but NOT in either of these will be closed
    longs = (stock_data.
             query('(current_price<yesterday_close) & (rsi<30) & (yesterday_close<sma_10)').
             nsmallest(context.TARGET_LONGS, 'current_to_sma25_price').
             index.tolist()
             ) 
    shorts = (stock_data.
             query('(current_price>yesterday_close) & (rsi>70) & (yesterday_close>sma_10)').
             nlargest(context.TARGET_SHORTS, 'current_to_sma25_price').
             index.tolist()
             )
  
    
    return longs, shorts

def place_orders_using_optimize(context, data):
    """
    Use Optimize to place orders all at once
    """
    # Get the list of securities we want to trade
    longs, shorts = get_longs_and_shorts(context, data)
    
    # Set the weights as a pandas series (in this case they are evenly weighted)
    long_weights = pd.Series(context.TARGET_WEIGHT, longs)
    short_weights = pd.Series(-context.TARGET_WEIGHT, shorts)

    # Combine the two
    weights = pd.concat([long_weights, short_weights])
    
    # Create our TargetWeights objective
    target_weights = opt.TargetWeights(weights) 

    # Execute the order_optimal_portfolio method with above objective and constraint
    # No need to loop through the stocks. 
    # The order_optimal_portfolio does all the ordering at one time
    # Also closes any positions not in 'target_weights'
    # As a bonus also checks for 'can_trade'
    # Could set constraints here if desired
    order_optimal_portfolio(
        objective = target_weights,
        constraints = []
    )
    
def place_orders_using_order_method(context, data):
    """
    Use order methods to place orders. Very similar to using optimize.
    """
    # Get the list of securities we want to trade
    longs, shorts = get_longs_and_shorts(context, data)
    
    # Set the weights as a pandas series (in this case they are evenly weighted)
    long_weights = pd.Series(context.WEIGHT, longs)
    short_weights = pd.Series(-context.WEIGHT, shorts)

    # Combine the two
    weights = pd.concat([long_weights, short_weights])

    # Order the stocks
    for stock, weight in weights.iteritems():
        if data.can_trade(stock):
            order_target_percent(stock, weight)
            
    # Close any positions not in our long or short list
    for stock, position in context.portfolio.positions.iteritems():
        if data.can_trade(stock) and stock not in weights.index:
            order_target_percent(stock, 0)
There was a runtime error.