Back to Community
The Pipeline API, Dividends and Splits - What You Need to Know

The pipeline API has some significant differences in how we are handling split and dividend price adjustments. Some questions have been asked about this, so I thought a detailed explanation in one place would be best.

In pipeline, the compute function uses pricing data that is split and dividend adjusted to the date for which the function is being called. (Go ahead and read that last sentence again. There's a lot of information to absorb there.)

Here is a simple dividend example. Company ABC puts out a $1 dividend quarterly with ex-dates of Jan 17, Apr 16, Jul 15, and Oct 15 in 2012. Company ABC is really boring and has the flattest stock price you've ever seen. It trades at 100 on the first day of the year and declines by a dollar on each ex-date as you'd expect, closing out the year at 96. In your algorithm, each morning before the market opens, your pipeline looks at a trailing window of data. You want to run a backtest over 2012. During the course of your backtest, the prices that the compute function uses look something like this:

Pipeline is using a different price adjustment method than we have used previously in Quantopian. The prices in the data object and in the history function continue to use the same adjustment method they always have. To refresh you on that behavior:

  • In the data object, all splits are applied before the backtest starts, and the fully split-adjusted prices are used throughout the backtest. The data object doesn't have the concept of dividend adjustments at all. https://www.quantopian.com/help#overview-datasources
  • The history function doesn't account for dividends at all, and reports a trailing window of fully split-adjusted prices.
  • In backtests, you get dividends as cash payments. https://www.quantopian.com/help#ide-dividends

The fact that pipeline and the data object use different pricing adjustments is, unfortunately, confusing. We have an in-process project to change the data object to align with pipeline. The data object will become split and dividend adjusted to the date for which the object is being called. Until that project ships, we have these two different adjustment schemes in play.

A few other key points:

  • Special dividends (where more than 25% of the value of the company is involved) are not handled by the backtester, not even as cash payments. However, special dividends are included in the pipeline calculation!
  • Fundamentals data is not split or dividend adjusted.
  • Data brought into research through the store is not split or dividend adjusted by Quantopian.

A final note about data accuracy: every time we extend our platform, it becomes a searchlight that exposes data errors that we hadn't seen before. Working with dividend-adjusted prices has been no different. We learned that a number of our dividends are in foreign currencies, and that results in some incorrect dividend adjustments. We are working on a solution. We expect that the community is going to find even more errors. The good news is that our data quality is always improving.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

7 responses

One of the frequent requests we have gotten from the community is to rank the universe based on the total returns of specific stocks over a given time period with dividend adjusted data. Now with the pipeline API, and the pricing adjustments described above, you can do this! Here is a quick example showing how.

Clone Algorithm
36
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 56181c9b8e21cb10c65933fa
There was a runtime error.

hmmm... it looks like it should be possible to exploit this to get a version of history() that accounts for dividends, which would be really useful for strategies that look at data over longer periods of time. But it doesn't look quite designed for that–is there an easy way to do it? (Of course, with the weekend coming up, I should have enough time to hack together the non-easy method.)

Christopher,
I'm not sure that I understand what you mean. You can set a window length for any factor you compute. The compute method will use split and dividend adjusted pricing data, as of the day it is being calculated, to calculate the factors. I believe this is essentially giving you dividend adjusted history. Can you explain what you are looking for that is different? I'd love to understand.

@Karen: I mean what if we want the dividend-adjusted closing prices for each of the last 30 trading days? Or 300? You could do it, but it doesn't really seem designed for that. Not that all 300 daily closing prices will be useful to know in itself, but you might have many different computations you want to do with them, say, if you want to calculate the covariance matrix for a dozen different stocks. But I guess if you wanted that, the Pipeline API way to do it is do it directly, not have an intermediate step where you first get the last 300 trading days.

As explained in here I see different values from pipeline and get_fundamentals.

The following algorithm fetchs the first 5 securities by market_cap every day from both get_fundamentals and pipeline and compare the results. They are not the same. Pipeline returns more entries than get_fundamentals.

I am pretty sure there is a good explanation that I'd like to know.

I'll copy and paste here the logs from the first mismatch:

2015-07-17before_trading_start:57WARNPossible mismatch (1)  
2015-07-17before_trading_start:73ERROREquity(8347 [XOM]) position in pipeline is 5 (market_cap=3.4602852208e+11) but in fundamentals the position is 4 (market_cap=347491909981)  
2015-07-17before_trading_start:76ERRORIt is definitely an error!  
2015-07-17before_trading_start:80DEBUG[entry 0]: Pipeline Equity(24 [AAPL]) market_cap=7.306138246e+11)  - fundamentals Equity(24 [AAPL]) market_cap=723642978300)  
2015-07-17before_trading_start:80DEBUG[entry 1]: Pipeline Equity(26578 [GOOG_L]) market_cap=3.99047495147e+11)  - fundamentals Equity(26578 [GOOG_L]) market_cap=399197831555)  
2015-07-17before_trading_start:80DEBUG[entry 2]: Pipeline Equity(46631 [GOOG]) market_cap=3.82824830008e+11)  - fundamentals Equity(46631 [GOOG]) market_cap=383426175641)  
2015-07-17before_trading_start:80DEBUG[entry 3]: Pipeline Equity(5061 [MSFT]) market_cap=3.70178964675e+11)  - fundamentals Equity(5061 [MSFT]) market_cap=369046424136)  
2015-07-17before_trading_start:80DEBUG[entry 4]: Pipeline Equity(1091 [BRK_A]) market_cap=3.495305024e+11)  - fundamentals Equity(8347 [XOM]) market_cap=347491909981)  
Clone Algorithm
12
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 56246b6495b8af10f6d24b8b
There was a runtime error.

Hi Luca,

There are a couple of reasons for the disparity that you see. The primary reason is that there instances in which multiple share classes (separate entities from Morningstar) that we map to a single SID. This happens for a couple of different reasons.

We discovered this issue previously because it manifested itself as the same SID being returned multiple times in the same get_fundamentals response. This itself caused problems for algorithms.

The root cause of this behavior is difficult to solve and didn't have a quick fix. I made a decision to first do work to simply filter out any of the duplicates from the get_fudamentals response in the absence of the long-term solution. The long term solution is to resolve the duplicate mapping at the database level. You're seeing the fact that we haven't solved this problem at the database level because get_fundamentals has extra logic that the Pipeline API does not have.

Attacking the long-term solution is close to the top of our data team's backlog at this point.

Hope that helps,
Josh

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thank you, knowing you are aware of this it's enough for me.