Back to Community
Fundamental Data from Morningstar Now Available for Backtesting

We have added a huge new data source to Quantopian today: Morningstar Corporate Fundamentals. With this data, you can now create a whole new class of algorithmic strategies.

"Fundamentals" refers to financial metrics reported by companies on a quarterly and annual basis. Working with Morningstar, Quantopian has integrated 12 years of historical data for over 670 financial metrics across more than 8,000 stocks. Measures provided include raw values such as revenue and earnings, as well as convenient pre-computed values and ratios such as market capitalization, earnings per share (EPS), price-to-earnings ratio (PE) and many, many more.

Fundamentals are available to you programmatically within the backtester. We think you’ll be excited to not only pursue new investment strategies, but also do so without the tedium of having to procure, cleanse, and align the data. We’ve done the heavy lifting for you, mapping Morningstar's company identifiers with Quantopian's existing market data set and providing convenient 'knowledge date' indexing to avoid look-ahead bias. We can’t imagine an easier way to work with fundamentals and we can't wait to see how the community utilizes this content!

Currently, we only make fundamental data available in the backtester. We'll add it to live trading in the near future. In the meantime, we want you to try out the data and tell us about your experience with it. We expect there are some data issues that need to be resolved, and it's prudent to find those problems while backtesting. We're hoping this community can help us improve the usage and quality of fundamental data in Quantopian.

For the time being we can provide this data for free. We are not certain that we will be able to economically support that forever. We have two offers that you can use to lock in free access for a while:

  1. Every registered user of Quantopian, as of January 1, will be guaranteed 6 months complimentary access to this data in Quantopian. So if you've been lurking here for a while, now is the time to register. And if you have friends who are fundamental investors, let them know!
  2. Share your coolest fundamentals based algorithm to the forums before January 1 for a chance to win an additional 12 months of free access. Justin Lent, Quantopian's new director of fund development will review submissions and selecting the best sample algos to be highlighted on Quantopian's blog in January. Tag your entry with #Fundamentals in the post title so we'll be sure not to miss it!

Ready to get started with fundamentals? Clone the algo below or read our detailed documentation in the help section.

Happy Trading!

P.S. With the inclusion of this new data, we have changed the Terms of Use for Quantopian. We added a new section (number 11) to the terms that reflects the new relationship with Morningstar, and we renumbered subsequent sections. No other changes were made.

Clone Algorithm
745
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5487af878d374d2939a15d57
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

51 responses

This is great, really excited to try it out. Opens up a whole range of new possibilities. A few questions:

1) RE: before_trading_starts, can price history data be used in that function? Meaning - I'd like to create a universe only out of low beta stocks, for example (beta is not a fundamental field but it can be easily calculated from price history). Seems like the custom universe functionality here could apply to several use cases, not just fundamental filtering.

2) Any chance you can update the documentation to be more specific about update frequencies for the various fields? Meaning - which are quarterly, which are daily, weekly, etc? And for fields like P/E, which earnings are used to calculate it (I'm assuming last 4 quarters but different services use different values, e.g. Valueline's default is past 2 quarters plus next 2 quarters estimated). Seems nitpicky but in my experience with fundamental-based algorithms, it matters.

Awesome news! I can't wait to try out a backtest of a basic EBIT/EV, then we can get to work on replicating the Quantitative Value screen!

Matt, thanks for the questions and comments.

Both are things we can look to do in the future. Generally as folks begin to use these features more, we'll prioritize our efforts around how to flesh out these methods. So accessing pricing data would definitely fall in that category.

With respect to the documentation, we can definitely look into doing that. For now, you can also try to look at the "as_of" field for each metric. By as_of, I mean that there is an companion date field for each metric. So basic_eps also has basic_eps_as_of which indicates the end date for which the metric applies. So if it is a quarterly metrics, you'll see tell-tale as_of dates like 6-30 and 9-30 (the last day of each quarter). For more frequently updated metrics, the as_of dates will have more variety.

By the way, I should introduce myself -- I'm a new product manager here at Quantopian. I started last month. My general sphere of responsibility is improving and expanding Quantopian's access to data for use in backtesting and beyond. If you have feedback for me on this front, I'd love to hear it. Feel free to message me or email me.

Hi Josh,

Overall, this is very exciting.... I have one quick question. Since get_fundamentals is called before the start of each trading day, wouldn't that limit flexibility or the scope of trading strategy. For example, what if someone wants to incorporate current EPS value on the day it is announced. I guess that problem is irrelevant if all the companies announced their EPS before 9:30 am. Am I missing something here?

Hi Josh,

This is really great news ! Ive been waiting for Quantopian to come up with this. With this addition, I think Quantopian is one of the best platforms around for backtesting survivourship bias free trading strategies. Keep up the great work !

@Vyomakesh, Thanks!

@Nyan, we process updates from Morningstar on a daily frequency. This is typically done when the market is closed, in the evening EST.

Is this data point in time?

Is the Morningstar data available in the research platform (which I still have not yet tried :/ ) ?

@Michasel S: yes it is point in time.

@Simon: It is not yet available in Research or Live Trading. Those are high on our backlog.

Josh, Out of curiosity, how do you reconcile the filing date issue on morningstar? I was on the impression that filing dates are relatively recent with them.

Are you matching to edgar?

Hi Michael,

I was too brief with my answer on your point in time question.

We generate our date logic in the point-in-time database through a variety of approaches, depending on the data and it's age. In scenarios in the past, where a file date is not available to us, we approximate it based on the as_of date by adding 45 days to the as_of date. We describe this in the point in time section in the help docs:

Quantopian takes care of this logic for you in the backtester. For
data updates since Quantopian began subscribing to Morningstar's data,
Quantopian tracks the file date based on when the information changes
in Morningstar. For historic changes, Morningstar also provides a file
date to reconstruct how the data looked at specific points in time. In
circumstances where the file date is not known to Quantopian, the file
date is defaulted to be 45 days after the as_of date.

Ok, that's probably close enough. That was my question, I didn't realize it was in the docs.

The 45 days approximation is definitely just that. We are definitely open to ways to improve that bit of course logic.

Thanks!

I mean the only way to add it with morningstar is to reconcile it with sec data.

I mean the only way to add it with morningstar is to reconcile it with sec data.

It looks like there's a small bug in the custom data graph for that algorithm:

    # track how many positions we're holding  
    record(num_positions = len(context.fundamental_df))  

I don't see how that would track the number of positions. Shouldn't it be:
record(num_positions = len(context.portfolio.positions))

Note securities can often be in .positions with 0 amount (shares).

Any update on plans to include fundamental data for companies listed on non-US exchanges, TSX, FTSE, DAX etc.?

Regards,
Mark

Generally Quantopian supports the US equities markets (which includes ADR from other countries). Next up in the queue for us in terms of markets is futures.

Can anyone explain what format the Morningstar file_date is in? It appears to be some number of units from some date but is just an integer.

For example:
Equity(48091 [VA])
pe_ratio 3.7998
file_date 14557536000000000

When will Morningstar data be allowed in active real time trading algorithms? I see messages in my back tests that states certain Morningstar data is not available for real time trading.

Most fields are available for use in live trading. There are a certain subset of fields (like shares_outstanding) for which we need to improve our data processing. As such we don't allow them in live trading until we've solved these problems. We're actively working to solve this problem currently.

Hi Josh, does the data set contain a pre computed Cash per Share?

If so, could you please give us an example of, for example, going long all companies which have Cash per Share greater than Share Price by X%?

Thanks in advance,
BT

EDIT: Please disregard this question. It was answered previously in this thread.

I would just love a better understanding of the morningstar data. Primarily if changes to data (such as cash balance) correspond with the exact date in which the company filed their 10-Q or 10-K. I am hoping this is the case, but I have not located any evidence to confirm, and it would be a big effort to analyze this in the backtester or in Research. Is there some documentation for the Morningstar data that I have not seen?

For my own reference:

Quantopian takes care of this logic for you in the backtester. For
data updates since Quantopian began subscribing to Morningstar's data,
Quantopian tracks the file date based on when the information changes
in Morningstar. For historic changes, Morningstar also provides a file
date to reconstruct how the data looked at specific points in time. In
circumstances where the file date is not known to Quantopian, the file
date is defaulted to be 45 days after the as_of date.

@ BT

I am running an algo that goes long stocks with Cash Per Share greater than Share Price. Finding about 300-600 exist in the top 4000 stocks per market cap. I will post it on your original thread after the back test is finished.

@Josh - I am particularly interested in the following fields: shares_outstanding & tangible_book_value_per_share to utilize in live trading, when would you anticipate availability? I am looking to launch my algorithm soon.

Thanks!

@Josh - I am particularly interested in the following fields: shares_outstanding & tangible_book_value_per_share to utilize in live trading, when would you anticipate availability? I am looking to launch my algorithm soon.

Thanks!

Hi all,
I'm loath to provide a specific estimate because this project (as most software projects do) has been very difficult to complete. I'd put it at weeks or a month or so.

All fields provided by Morningstar in the database are listed out here: quantopian.com/help/fundamentals if you're looking for information on the availability of specific metrics.

Thanks
Josh

Hi Josh,

Thank you very much for the update!

Best
Adam

Hi Josh,

Can you provide a status update on the morningstar data availability for broker-backed trading?

We are in the process of rebuilding the database. It takes . . . a long time as it replicates our nightly processing of the data. So we're code complete. We just need to run through the processing and test the results before swapping over. This will make available shares_outstanding.

Just to be clear: Morningstar data is generally available in broker backed trading. It is just a subset of fields that measure companies on a per share basis that are ineligible.

Hi Josh,

Thank you for the update.

So, per share fields are an issue? So, would tangible_book_value_per_share be ineligible? When would it become eligible?

Thank you,
Adam

Yes, that field would be impacted. I would recommend using tangible_book_value instead. The work to make all of these per share fields usable in live trading is large and we're at the outset of the project. I cannot commit to a timeframe.

Josh, Can you be more specific about point-in-time dating? Specifically, is there at date before which you approximate all restated as_of dates by adding 45 days to the as_of date? Also, can you point to the point in time section in the help docs? Can't quite find that section.

June 2014 is when we began processing files from Morningstar.

Here is the relevant section of the documentation but it doesn't expand much further than what you've likely read here: https://www.quantopian.com/help#ide-fundamentals

Thank you, Josh. Just to make sure I understand, let's say you have GE earnings:

  • 6/30/10 Earnings released 7/15/10 as $10 / share
  • 6/30/10 Earnings now REVISED 8/15/10 to $8 / share

In Quantopian today we would see the following?:

  • 6/30/10 Earnings released 7/15/10 as $8 / share

@Josh - can you provide a status update on morningstar's shares_outstanding availability in live trading?

Thanks!

The reprocessing continues. It's had a few bumps in the road slowing us down but we are continuing to make process.

Thanks Josh. Do you have a sense of when it will be completed? I have an algo I would like to launch, and this is the only dependency that is stopping me from launch. Thank you.

Also @Josh, in addition to my question above, can you confirm if you do include RESTATED data in addition to originally reported?

Thanks so much.

@Adam, my guess is within a month or so (conservatively). Of course now that I've said that, I've probably jinxed it. We're processing and iterating through bug fixes as they arise. Fingers crossed.

@Kim, the answer to the revisions question is "sometimes". If the most recent data point for a metric/company combination is restated by Morningstar, we process that change and surface it. That applies to the timeframe after June 2014.

This is one of the reasons we're anxious to improve by re-architecting the underlying data source.

Regards,
Josh

Thanks Josh!

@Josh - can you provide a status update on morningstar's shares_outstanding availability in live trading?

Thanks!

Hi Adam,

We recently fixed the shares_outstanding field in backtesting. We haven't yet unblocked it from live trading but we should be doing so shortly.

Jamie

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

@Jamie

Could you provide an update on when the shares_outandinting field will be unblocked in live trading? I have an algo that I would like to launch that leverages that field.

Thanks
Adam

Hi Adam,

Unfortunately, I don't have a timeline yet. It's on the list, we just haven't to it yet.

Sorry for the inconvenience.

Hi Jamie,

To be candid, I am frustrated. I have been asking about adding this data since April 2016, and have not had a response that indicates a timeline for actual release. (I keep getting shortly, or a month from now) I understand it is being worked on, but an understanding of how the process works and where it is on the list, and why it is taking so long would be helpful for the community to understand. I expect there are others with similar frustrations.

Thanks
Adam

Hi Adam,

I'm sorry that you've had such a frustrating experience. The big fundamentals project to fix shares_outstanding took longer than expected. The reason it hasn't yet been unblocked from live trading is because there are still some limitations to the dataset. For example, the data is only updated monthly by Morningstar, and it is not split-adjusted. As a result, we were hesitant to unblock it from live trading. Since then, we have reached out to some community members who have expressed an interest in live trading with shares_outstanding and we have come to the decision that we will unblock the field from live trading. The next step is to actually unblock it. I can't promise an exact timeline, but it's on the list.

Thank you Jamie. I look forward to it being unblocked.

Best
Adam

line 123 in code has been depreciated update_universe(context.fundamental_df.columns.values) should data.history(context.fundamental_df.columns.values') be inserted instead to make this Quant 2?

thank you