Exclude stocks from pipeline by sid.

Hi there,

Previously I used to be able to exclude stocks by sid by creating a condition that would check if the stock was in an array and if so then set order target to 0. Now I have moved to the Optimize API and want to keep this sort of exclusion as part of the pipeline generation. What would be the best way to implement a simple exclusion array of sid's into the pipeline?

Ross D

15 responses

Hi Ross,

You can use the StaticAssets filter to construct a filter that excludes a static list of securities. I've attached an example algorithm that demonstrates how to to do.

60
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline import Pipeline
from quantopian.pipeline.filters.morningstar import Q1500US
from quantopian.pipeline.filters import StaticAssets

def initialize(context):

context.exclusion_list = [sid(24), sid(5061), sid(8554)]

attach_pipeline(make_pipeline(context), 'my_pipeline')

def make_pipeline(context):

# Base universe set to the Q500US
base_universe = Q1500US()

exclusion_filter = ~StaticAssets(context.exclusion_list)

pipe = Pipeline(
screen = (base_universe & exclusion_filter)
)
return pipe

context.output = pipeline_output('my_pipeline')

print 'Is AAPL in our pipeline output? %s' % (sid(24) in context.output)

There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Works a charm! Thank you!

I'm curious if I could use StaticAssets to make sure I include my current holdings in my pipeline. My guess is that StaticAssets is evaluated once at the beginning when the pipeline is created and it would not be possible to update the set of assets to include on a daily basis. Am I wrong? Is there a better way to do this?

Hi Rudiger -

I'm confused by your question. Are you wanting to dynamically change the limited set of stocks that one of your pipeline factors uses as inputs? Or are you just wanting to check that your pipeline universe contains a specific set of of stocks at any given point in time? I guess my confusion is that I thought unless restricted in some fashion, pipeline has access to all stocks available to trade point-in-time?

Hi Rudiger,

That's correct, you cannot incorporate your current positions in your pipeline. Since a pipeline is defined in initialize, you won't have access to the current positions each day. Currently, there isn't a good way to include your current positions in your pipeline definition. You will have to get data for these stocks using data.history.

I've passed along your request for the ability to access current positions in pipeline internally.

OK, let me try to clarify.

Let's say I have a strategy where I rank the Q1500US by PE ratio and buy the lowest 10 stocks. Then I will exit a position if it's rank goes above 20 and I find the lowest ranked stock that I don't hold to replace it.

The problem is: what if one of the stocks leaves the Q1500US universe? I would prefer if it stays in my universe so that the pipeline can continue to rank it compared to all the other Q1500US stocks.

How do I force my held positions to remain in my pipeline universe? The StaticAssets factor appears to be the wrong tool because it seems like its constituents are probably only set once.

i.e. I can't go: screen = Q1500US | StaticAssets(context.list_of_currently_held_stocks)

...Because it wouldn't be evaluated each time, only when the pipeline was created.

Hi Rudiger,

If I understand correctly, you'd need to run your PE ratio (or whatever) factor across all stocks, and then use the point-in-time Q1500US list combined with your context.list_of_currently_held_stocks to do the ranking and selection. This way, you wouldn't be forced to drop certain stocks just because they don't persist in the Q1500US universe.

Grant

Yeah, that appears to be the solution for now, although it seems like a waste given that the point of the pipeline is to be able to set screens, compute ratios, rank, etc very efficiently.

I guess in before_trading_start, I could get my pipeline output, see if any holdings are not included in the pipeline output, if a stock is missing I could query for the PE ratio of that stock and compare it to the 20th ranked stock. This would avoid having to rerank.

My hunch is that for a something like PE ratio, it'll be just as efficient to get the PE ratios for all stocks in the database, and eliminate the ones that are not within the point-in-time Q1500US and your current portfolio, do your ranking, etc. Ranking should be a reasonably efficient process. And if you aren't trading every day, you could limit the ranking to weekly/monthly/quarterly.

The current implementation of the pipeline fetches data and calculates all factors, filters, and classifiers asynchronously from the algorithm. This greatly improves performance but has the side-effect of not being able to interact 'real time' with the algorithm. In this case, one cannot modify a filter based upon current algorithm holdings.

However, there may be certain factors which are used to determine closing existing positions which are separate from those used in 'open' decisions. As Grant pointed out, one cannot filter any of the factors used in closing positions with a filter which may change (eg Q1500US). Such a filter potentially would exclude data for existing positions and therefore wouldn't generate a proper close signal. One can however, still filter or mask any factors used in opening positions only.

The Q1500US doesn't change that quickly - it was designed with stability in mind. This may not be a problem you need to engineer for.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Is there a write up somewhere that describes exactly how the Q1500US is determined and how often it is updated?

For the Q1500US, presumably the default update is monthly, as for the Q500US.

Note that you can construct your own universe and control the update period, based on the example given in the help.

Would like to dynamically both include and exclude any particular stocks in pipeline.

I've passed along your request for the ability to access current positions in pipeline internally.

Any ETA?

from quantopian.pipeline  import Pipeline
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline.filters import Q500US, StaticAssets

def initialize(context):
context.includes = [ symbol('GBR') ]    # Random penny stock, definitely not in Q500US
context.excludes = [ symbol('XOM'), symbol('JNJ') ]    # In top 10 market cap

base_Q_US = Q500US()
in_filter =  StaticAssets(context.includes)
ex_filter = ~StaticAssets(context.excludes)    # ~ means not.
pipe      = attach_pipeline(Pipeline(), 'zoo')
pipe.set_screen( (base_Q_US & ex_filter) | in_filter )    # & <- and     | <- or

context.z = pipeline_output('zoo')

log.info('Number of stocks ' + str(len(context.z)))

for s in context.excludes + context.includes:
log.info('{} in pipe: {}'.format(s.symbol, s in context.z.index))

# Redefine includes
context.includes = [ symbol('DAC') ]   # Another random low dollar stock, replacing the first.

log.info('Static log line _ {} in pipe: {}'.format('GBR', symbol('GBR') in context.z.index))

'''
2017-03-17 05:45 before_trading_start:18 INFO Number of stocks 499
2017-03-17 05:45 before_trading_start:21 INFO XOM in pipe: False
2017-03-17 05:45 before_trading_start:21 INFO JNJ in pipe: False
2017-03-17 05:45 before_trading_start:21 INFO GBR in pipe: True
2017-03-17 05:45 before_trading_start:26 INFO Static log line _ GBR in pipe: True
2017-03-20 05:45 before_trading_start:18 INFO Number of stocks 499
2017-03-20 05:45 before_trading_start:21 INFO XOM in pipe: False
2017-03-20 05:45 before_trading_start:21 INFO JNJ in pipe: False
2017-03-20 05:45 before_trading_start:21 INFO DAC in pipe: False                   <== Want to be True
2017-03-20 05:45 before_trading_start:26 INFO Static log line _ GBR in pipe: True  <== Want False here, replaced
'''


Something went haywire with the original from 2016, says KeyError: AssetExists()
Here's a replacement. You can toggle the exclusion list on-off.

3
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline  import Pipeline
from quantopian.pipeline.filters import Q1500US, QTradableStocksUS, StaticAssets

def initialize(context):
context.do_screen = 1      # 0 - No exclusions    1 - Do screen for static_list

context.static_list = [sid(24), sid(5061), sid(8554)]

attach_pipeline(make_pipeline(context), 'someone_elses_pipeline_not_my')

def make_pipeline(context):

m &=  StaticAssets(context.static_list)

else:  # The tilde (~) means 'not' so this will exclude the static list
m &= ~StaticAssets(context.static_list)

return Pipeline(
screen = m
)

context.output = pipeline_output('someone_elses_pipeline_not_my')

log.info('AAPL in pipe ? {}   pipe len: {}'.format(
sid(24) in context.output.index,
len(context.output)
))

if len(context.output) < 10:
log.info([s.symbol for s in context.output.index])

for s in context.static_list:
if s not in context.output.index:
log.info('%s not in pipe' % s.symbol)

log.info('.')  # separator between days, easier reading


There was a runtime error.