Back to Community
How to filter equity types or classes in Pipeline?

I am trying to replicate a filter from get_fundamentals in Pipeline, but given the current implementation limitations I have not been successful.
Can anyone help to implement the below filter (or equivalent) in Pipeline ?

.filter(fundamentals.share_class_reference.security_type == 'ST00000001') # common stock only .filter(~fundamentals.share_class_reference.symbol.contains('_WI')) # drop when-issued .filter(fundamentals.share_class_reference.is_primary_share == True) # remove ancillary classes

many thanks

13 responses

Your query may be overbroad, ie. return too many securities. You could try adding .limit(500) at the end.

There's some of this that we don't actually support yet in pipeline - booleans and strings. The support for that should be coming soon.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Dan - But string and Boolean data are present in the fundamentals database, and can be used in filters as above, as long as they are not returned in query results, correct?

Thanks Dan,

I think that would be very important as currently the full functionality of the .filter in get_fundamentals can't be duplicated in Pipeline.

Its actually quite important to filter on Boolean and equity string class types as certain classes of equities such as preferred or _WI are not liquid nor have real pricing signal that can be reliably used in a wide selection universe -- specially for the long/short strategies where one is looking for divergence in attributes.

Looking forward to the new feature!

I've the same problem, I'd like to port to the Pipeline API the following filter

        # No Financials (103), Real Estate (104), Basic Materials (101) and ADR  
        .filter(~fundamentals.asset_classification.morningstar_sector_code.in_([103, 104, 101]))  
        .filter(fundamentals.share_class_reference.is_depositary_receipt == False)  
        .filter(fundamentals.share_class_reference.is_primary_share == True)

        # Only pick active common stocks  
        .filter(fundamentals.share_class_reference.share_class_status == "A")  
        .filter(fundamentals.share_class_reference.security_type == "ST00000001")

        # Exclude When Distributed(WD), When Issued(WI) and VJ - usuallly companies in bankruptcy  
        .filter(~fundamentals.share_class_reference.symbol.like('%\_WI'))  
        .filter(~fundamentals.share_class_reference.symbol.like('%\_WD'))  
        .filter(~fundamentals.share_class_reference.symbol.like('%\_VJ'))

        # Exclude Halted stocks  
        .filter(~fundamentals.share_class_reference.symbol.like('%\_V'))  
        .filter(~fundamentals.share_class_reference.symbol.like('%\_H'))

         # Only NYSE, AMEX and Nasdaq  
        .filter(fundamentals.company_reference.primary_exchange_id.in_(['NYSE', 'NAS', 'AMEX']))  

Please update this thread, when this feature is implemented for the Pipeline API.

+1 Would also be helpful to have some canned, vetted/clean point-in-time lists of securities, in addition to the more complex, multi-line filtering.

We shipped a fix for the boolean factors yesterday. String type fields are still not available in pipeline but booleans, ints, and floats all are.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thanks a lot, Jamie! This is a great news for me.
I've implemented the following filter... I've found also a trick to get the symbol and the exchange... I hope you will support soon also string type fields!

class UniverseFilter(CustomFactor):  
    """  
    Return 1.0 for the following class of assets, otherwise 0.0:  
      * No Financials (103), Real Estate (104), Basic Materials (101) and ADR  
        (Basic Materials are too much sensitive to exogenous macroeconomical shocks.)  
      * Only primary common stocks  
      * Exclude When Distributed(WD), When Issued(WI) and VJ - usuallly companies in bankruptcy  
      * Exclude Halted stocks (_V, _H)  
      * Only NYSE, AMEX and Nasdaq  
    """  
    window_length = 1  
    inputs = [morningstar.share_class_reference.is_primary_share,  
              morningstar.share_class_reference.is_depositary_receipt,  
              morningstar.asset_classification.morningstar_sector_code  
              ]  
    def compute(self, today, assets, out, is_primary_share, is_depositary_receipt, sector_code):  
        criteria = is_primary_share[-1] # Only primary Common Stock  
        criteria = criteria & (~is_depositary_receipt[-1]) # No ADR  
        criteria = criteria & (sector_code[-1] != 101) # No Basic Materials  
        criteria = criteria & (sector_code[-1] != 103) # No Financials  
        criteria = criteria & (sector_code[-1] != 104) # No Real Estate  
        def accept_symbol(equity):  
            symbol = equity.symbol  
            if symbol.endswith("_PR") or symbol.endswith("_WI") or symbol.endswith("_WD") or \  
               symbol.endswith("_VJ") or symbol.endswith("_V") or symbol.endswith("_H"):  
                return False  
            else:  
                return True  
        def accept_exchange(equity):  
            exchange = equity.exchange  
            if exchange == "NEW YORK STOCK EXCHANGE":  
                return True  
            elif exchange == "AMERICAN STOCK EXCHANGE":  
                return True  
            elif exchange.startswith("NASDAQ"):  
                return True  
            else:  
                return False  
        vsid = np.vectorize(sid)  
        equities = vsid(assets)  
        # Exclude When Distributed(WD), When Issued(WI) and VJ (bankruptcy) and Halted stocks (V, H)  
        vaccept_symbol = np.vectorize(accept_symbol)  
        accept_symbol = vaccept_symbol(equities)  
        criteria = criteria & (accept_symbol)  
        # Only NYSE, AMEX and Nasdaq  
        vaccept_exchange = np.vectorize(accept_exchange)  
        accept_exchange = vaccept_exchange(equities)  
        criteria = criteria & (accept_exchange)  
        out[:] = criteria.astype(float)  

does this work in research. I cant get a filter to work with this or even the simple

share_class = Latest([morningstar.share_class_reference.is_primary_share])
share_class_filter = (share_class == 1.0)
or
share_class_filter = (share_class == True)

it just returns False as a single value ! and the pipe.set_screen just gives a type error

TypeError: zipline.pipeline.pipeline.set_screen() expected a value of type zipline.pipeline.filters.filter.Filter for argument 'screen', but got bool instead.

is_primary_share = morningstar.share_class_reference.is_primary_share.latest  
pipe.set_screen (is_primary_share)  

seems to work

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

humm thanks !! I wonder why the other way does not work ...

Jamie mentioned above we recently shipped a fix to make booleans work better. Part of that fix includes booleans returning a filter instead of a factor, which is why James' suggestion works.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thanks. It would be great to document these API changes in an update blog or by some other method. It would be very helpful for us all.