Back to Community
Fundamentals.total_liabilities.latest always NaN before 2010

How could it be that a such importat data point is missing before 2010?

from quantopian.pipeline import Pipeline  
from quantopian.research import run_pipeline  
from quantopian.pipeline.data import Fundamentals  
import pandas as pd  
from quantopian.pipeline.filters import  StaticAssets

def make_pipeline():  
    return Pipeline(  
    columns={  
            'name': Fundamentals.standard_name.latest,  
            'total_liabilities': Fundamentals.total_liabilities.latest  
        },  
        screen= StaticAssets(symbols(['MSFT', 'ORCL', 'F', 'AAPL', 'IBM']))  
    )

my_pipe = make_pipeline()  
date = '2009-04-01'  
result = run_pipeline(my_pipe, date, date)  
result  

and the result is:

2009-04-01 00:00:00+00:00  
name    total_liabilities  
Equity(24 [AAPL])   Apple Inc   NaN  
Equity(2673 [F])    Ford Motor Co   NaN  
Equity(3766 [IBM])  International Business Machines Corp    NaN  
Equity(5061 [MSFT])     Microsoft Corp  NaN  
Equity(5692 [ORCL])     Oracle Corporation  NaN  

total_liabilities was even removed from the help page: https://www.quantopian.com/help/fundamentals

It is a data error, or have I missed something?
Please let me know what happened, thank!

3 responses

Hi Constantino, thanks for the feedback. Morningstar recently made changes to their fundamental dataset, you can read the details here.

The long story short:
Morningstar stopped providing the total_liabilities field, because it was an exact duplicate of the total_liabilities_net_minority_interest field. We made a change on our backend so that algorithms using total_liabilities will not break, but removed it from the help page to encourage the use of total_liabilities_net_minority_interest. You can still use total_liabilities, but we prefer that you use the new field name.

As for data being missing before 2010, you are also correct. Morningstar occasionally adds new fields, and it seems they started providing that particular data around April of 2009. This leaves you with two choices:

1: Testing your algorithms which use that data field on dates starting after, say, May 2009. This is my suggestion.
2: Continue testing your algorithms on dates prior to 2010, but use a notnan() filter on the data field. We encourage the use of notnan() and notnull() filters anyways. However, this method will change an algorithm's behavior once the liability data starts coming in, which will give you an inconsistent idea of it's performance.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Cal,

thank you very much for the info. I knew that post, but missed that total_liabilities is not mapped to total_liabilities_net_minority_interest.

I'm quite sure that with the old data, there where values for total_liabilities even before 2010.
Anyway I've found a simple alternative to get also older data, just compute it ;-)

Total Liabilities is defines as Total Assets - Stockholder's Equity.
Total Assets is directly available in Q, while Stockholder's Equity can be computed as total_equity - minority_interest_balance_sheet.

That is, accounting for the fact that minority_interest_balance_sheet can be NaN:

total_liabilities = Fundamentals.total_assets.latest - Fundamentals.total_equity.latest - LatestNanToNum(inputs=[Fundamentals.minority_interest_balance_sheet])

Where the custom factor LatestNanToNum is defined as follows:

class LatestNanToNum(CustomFactor):  
    window_length = 1

    def compute(self, today, assets, out, data):  
        out[:] = np.nan_to_num(data[-1])  

and this is the updated notebook:

Loading notebook preview...

All the above consideration embedded in a custom factor:

class TotalLiabilities(CustomFactor):  
    inputs = [Fundamentals.total_assets,  
              Fundamentals.total_equity,  
              Fundamentals.minority_interest_balance_sheet,  
              Fundamentals.total_liabilities_net_minority_interest]  


    window_length = 1

    def compute(self, today, sids, out, assets, equity, minority, liabilities):  
        isnan = np.isnan(liabilities[-1])  
        liabilities[-1, isnan] = assets[-1, isnan] - equity[-1, isnan] - np.nan_to_num(minority[-1, isnan])  
        out[:] = liabilities[-1]  
Loading notebook preview...