Back to Community
Pipeline Now Available in Real Money Trading

The pipeline API is now available for use in real money trading. Algorithms using the pipeline API can now be connected to your Robinhood account or your Interactive Brokers account (both real money and paper). The pipeline API brings many key new capabilities to your real money trading. With it you can rank equities using a variety of data: pricing, fundamental and many of our partner data sources such as news sentiment, social media sentiment, earnings estimates, earnings calendars and more.

Any changes to real-money trading platform are carefully tested before we make them available. We have watched pipeline algorithms and monitored their performance in Quantopian paper trading over the past several months and have decided to open the gates and allow real money trading as well.

There is one limitation: There are 20 fundamental data fields that are not supported in live trading with pipeline because they are not properly adjusted. The pipeline API uses data that is both split and dividend adjusted (explained in detail here.) The fundamental data fields are not adjusted for splits and dividends. Our concern is that algorithms would be making decisions based on a combination of adjusted and unadjusted data, and those decisions could have unexpected results. For this reason we have restricted the fields listed below from being used in pipeline real money trading. Your algo will raise an exception and crash in real money trading if it uses these fields, however you can continue to use them in backtesting and zipline paper trading.

We're making pipeline real-money trading available today with this limitation because we didn't didn't want to hold up the entire API because of just these fields. In the future, we will adjust the fundamentals fields with splits and dividends so that they can be used in real money trading. We will post an update when they are adjusted and ready.

The pipeline API now is available in every piece of the Quantopian platform: research, backtesting, paper trading and real money trading.

Valuation Ratios:
book_value_per_share
cfo_per_share
fcf_per_share
sales_per_share
tangible_book_value_per_share
tangible_bv_per_share3_yr_avg
tangible_bv_per_share5_yr_avg
working_capital_per_share
working_capital_per_share3_yr_avg
working_capital_per_share5_yr_avg

Valuation:
share_class_level_shares_outstanding
shares_outstanding

Earnings Report:
basic_average_shares
basic_eps
continuing_and_discontinued_basic_eps
continuing_and_discontinued_diluted_eps
diluted_average_shares
diluted_continuous_operations
diluted_eps
dividend_per_share
normalized_basic_eps
normalized_diluted_eps

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

13 responses

This is very exciting!

I am still encountering timeout issues with some of the 3rd party datasets (namely stock twits). Is there a timeline on a fix for this as well? I know something was in the works, but I thought it might have coincided with this announcement.

Thanks

Hi Ben,

I saw a PR for performance improvements go in this weekend, so that tells me progress is being made but these types of performance problems can prove to be nasty and not straight forward so I can't commit to a timeline. Generally these pieces of work are separate and will be delivered on their own timelines.

Thanks for hanging in there!
Josh

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

There are 20 fundamental data fields that are not supported in live
trading with pipeline because they are not properly adjusted. The
pipeline API uses data that is both split and dividend adjusted

... does this mean we will soon be able to get split/div adjusted data natively in quantopian, without having to rely on pipeline ?

Hi Florent,
Yes, we are working on delivering fully adjusted data in the backtester. I can't give an exact date for delivery, but it's in progress.

This might be a dumb question, but if shares_outstanding is impacted is market_cap impacted?

Awesome, finally a reason to play with the pipelines

@Matt, not a dumb question.

Shares outstanding are impacted because the share count itself changes.

In the case of market cap, the market cap doesn't itself change because it's the product of the shares outstanding and the price per share. So when the shares outstanding increase, the price per share decreases by the same proportion. There is no discontinuity the market cap value.

Hope this helps,
Josh

Does this mean shares outstanding from traditional fundamentals queries has also been wrong all this time? Or is this somehow a specific fault in pipeline?

Also, does this mean that all the example pipeline factors which calculate MarketCap as shares_outstanding * close are all wrong?

@Josh Makes sense. Thanks for the explanation.

Simon,

A couple of elements to answering your question:

1) This problem isn't apparent in the get_fundamentals() queries used in algorithms because that function returns "today's best known value" for the requested metric. It doesn't provide a time series response. So the day before a split, it provides "today's shares outstanding" and it's accurate and then the day after a split it provides "today's shares outstanding" and it is split adjusted and indeed accurate.

Pipeline is the first instance of this data being exposed in an algorithm beyond this initial use case of a time series being exposed in algos for this data so it became a problem in it's use in pipeline.

2) The historical data for time period from 2002 - 2014 for fundamentals is generated based on the split adjusted data provided to us by Morningstar on June 1, 2014. So the per share metrics and the shares outstanding are split adjusted as of June 1, 2014. One of the pieces of work ahead of us is to un-adjust this data in our construction of that historical time period. So this part of the problem is independent of it's use by pipeline.

3) We discovered that we're not processing the updated values from Morningstar in our processing since June 1, 2014. The data comes, but we have a bug that improperly delays the reflection of this new data point (sent to us as a correction). This bug is in the process of being fixed but further diminishes the accuracy of the data.

I wouldn't say the data is "all wrong". But if a company has had a split, the data is not reliable for that market cap calculation.

Given the workaround for the lack of USEquityPricing.open prices (i.e. using the prior day's close), some values could be significantly affected by a split when using raw data instead of split adjusted data.

For example, this example in the 101 Alphas Project chat has the following formula for alpha factor no. 101 which is supposed to be ((close - open) / ((high - low) + .001)):

out[:] = (close[-1] - close[-2]) / ((high[-1] - low[-1]) + .001)  

I guess I'm confused. The help page clearly states:

Our data uses adjusted close prices. That means that the effect of all stock splits and merger activity are applied to price and volume data. As an example: A stock is trading at $100, and has a 2:1 split. The new price is $50, and all past price data is retroactively updated 2:1 (volume data is correspondingly updated 1:2). In effect, that means you can ignore stock splits unless you are storing prices in a live trading algorithm.

Further down, under the section discussing dividends, we have:

While other corporate events, such as splits and mergers, are handled by adjusting historical prices and volumes...Dividend events modify the security price and the portfolio's cash balance.

But then we have some exceptions described above, and on https://www.quantopian.com/posts/the-pipeline-api-dividends-and-splits-what-you-need-to-know, where there is discussion of other fine points, such as:

The history function doesn't account for dividends at all, and reports a trailing window of fully split-adjusted prices.
We have an in-process project to change the data object to align with pipeline. The data object will become split and dividend adjusted to the date for which the object is being called.
Fundamentals data is not split or dividend adjusted.
Data brought into research through the store is not split or dividend adjusted by Quantopian.

Given the risk of users ending up with "garbage in garbage out" or wasting time trying to compare "apples to oranges" you might consider creating a definitive section of your help docs that tabulates all of this information, and keeps it up to date, including flagging bugs (and their expected resolution dates, if known). Personally, I find it challenging to get it all straight.

Generally, I'd recommend some sort of public-facing bug tracker, as well (perhaps you could leverage your presence on github, https://github.com/quantopian). I think you need to move beyond the practice of posting on forums, "Oh, by the way such-and-such is broken." It would be better to say, "The problem has been logged as bug #57 on our bug tracker." Then, information germane to the bug could be posted there, including temporary work-arounds and details on how the bug was resolved and the testing performed to show that the fix was effective.

Another thought is that you might somehow poll users regarding the readiness of features for real-money trading. It seems there should be a comment/vetting period, no? And maybe Quantopian would run real money first, and publish the results (I don't see any mention that you've ever run real money with pipeline)?

This appears OK regarding splits. For example, Apple had a 7-1 split on 2014-06-09, so it is a good test candidate.

from quantopian.pipeline import Pipeline  
from quantopian.pipeline import CustomFactor  
from quantopian.algorithm import attach_pipeline, pipeline_output  
from quantopian.pipeline.data.builtin import USEquityPricing as bar  
from quantopian.pipeline.factors import SimpleMovingAverage, Returns

class ReturnFactor(CustomFactor):  
    inputs = [bar.close]  
    window_length = 2

    def compute(self, today, assets, out, close):  
        out[:] = close[-1] / close[-2] - 1

def initialize(context):  
    pipe = Pipeline()  
    attach_pipeline(pipe, 'pipe')  
    returns = Returns(window_length=2)  
    returns_pipeline = ReturnFactor()  
    sma_1 = SimpleMovingAverage(inputs=[bar.close], window_length=1)  
    sma_5 = SimpleMovingAverage(inputs=[bar.close], window_length=5)  
    pipe.add(returns, 'returns')  
    pipe.add(returns_pipeline, "ReturnFactor")  
    pipe.add(sma_1, "SMA_1")  
    pipe.add(sma_5, "SMA_5")  
    context.aapl = sid(24)

def handle_data(context, data):  
    output = pipeline_output('pipe')  
    log.info("pipe aapl SMA1: {0:.2f}".format(output.loc[context.aapl].SMA_1))  
    log.info("pipe aapl SMA5: {0:.2f}".format(output.loc[context.aapl].SMA_5))  
    log.info("data aapl close: {0:.2f}".format(data[context.aapl].price))  
    log.info("pipe aapl return: {0:.2%}".format(output.loc[context.aapl].returns))  
    log.info("pipe aapl return factor: {0:.2%}".format(output.loc[context.aapl].ReturnFactor))  
    log.info("\n")

Per the logs:

014-06-04handle_data:46INFOpipe aapl SMA1: 637.54  
2014-06-04handle_data:47INFOpipe aapl SMA5: 631.65  
2014-06-04handle_data:48INFOdata aapl close: 92.12  
2014-06-04handle_data:49INFOpipe aapl return: 1.44%  
2014-06-04handle_data:50INFOpipe aapl return factor: 1.44%  
2014-06-04handle_data:52INFO  
2014-06-05handle_data:46INFOpipe aapl SMA1: 644.82  
2014-06-05handle_data:47INFOpipe aapl SMA5: 635.81  
2014-06-05handle_data:48INFOdata aapl close: 92.48  
2014-06-05handle_data:49INFOpipe aapl return: 1.14%  
2014-06-05handle_data:50INFOpipe aapl return factor: 1.14%  
2014-06-05handle_data:52INFO  
2014-06-06handle_data:46INFOpipe aapl SMA1: 647.35  
2014-06-06handle_data:47INFOpipe aapl SMA5: 638.24  
2014-06-06handle_data:48INFOdata aapl close: 92.23  
2014-06-06handle_data:49INFOpipe aapl return: 0.39%  
2014-06-06handle_data:50INFOpipe aapl return factor: 0.39%  
2014-06-06handle_data:52INFO  
2014-06-09handle_data:46INFOpipe aapl SMA1: 92.23  
2014-06-09handle_data:47INFOpipe aapl SMA5: 91.54  
2014-06-09handle_data:48INFOdata aapl close: 93.70  
2014-06-09handle_data:49INFOpipe aapl return: -0.27%  
2014-06-09handle_data:50INFOpipe aapl return factor: -0.27%  
2014-06-09handle_data:52INFO  
2014-06-10handle_data:46INFOpipe aapl SMA1: 93.70  
2014-06-10handle_data:47INFOpipe aapl SMA5: 92.32  
2014-06-10handle_data:48INFOdata aapl close: 94.25  
2014-06-10handle_data:49INFOpipe aapl return: 1.60%  
2014-06-10handle_data:50INFOpipe aapl return factor: 1.60%  
2014-06-10handle_data:52INFO