Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Mask causes research pipeline to fail?

Hi all,

any help on the following is much appreciated!

I have a somewhat computational heavy pipeline, that (hopefully) helps to find long-term and short term support trends, and resistance in trade patterns. It looks like so:

def make_pipeline():  
#    test_securities = symbols(['NVDA', 'CNP'])  
#    base_universe = StaticAssets(test_securities)  
    base_universe = Q500US()

    universe      = base_universe  
    longTermTrend = confirmTrendWeeklyData(window_length = 400, mask = base_universe)  
    dailyTrend    = confirmTrendDailyData(window_length = 400, mask = longTermTrend.trendStartOffset.notnull())  
#    dailyTrend    = confirmTrendDailyData(window_length = 400, mask = base_univers  
    Resistance    = findResistance(window_length = 400, mask = longTermTrend.trendStartOffset.notnull())  
#    Resistance    = findResistance(window_length = 400, mask = base_universe)

    return Pipeline(columns={  
                             '1. LT Trend start offset'       : longTermTrend.trendStartOffset,  
                             '1. LT nr of confirmations'      : longTermTrend.confirmationCount,  
                             '1. LT relative OBV'             : longTermTrend.relOBV,  
                             '1. LT Pos. relative to Supp'    : longTermTrend.positionRelToSupp,  
                             '2. Trend start offset'          : dailyTrend.trendStartOffset,  
#                             '2. Trend nr of confirmations'   : dailyTrend.confirmationCount,  
#                             '2. Trend relative OBV'          : dailyTrend.relOBV,  
#                             '2. Trend pos. relative to Supp' : dailyTrend.positionRelToSupp,  
                             '3. Best R curve start offset'   : Resistance.trendStartOffsetBestR,  
#                             '3. Nr of conf. best R'          : Resistance.noBestRConfirmed,  
#                             '3. Pos. rel. to best R'         : Resistance.relPriceBestR,  
                             '4. Highest R start offset'      : Resistance.trendStartOffsetHighestR,  
#                             '4. Nr. of conf. highest R'      : Resistance.noHighestRConfirmed,  
#                             '4. Pos. rel. to highest R'      : Resistance.relPriceHighestR,  
                             '5. Latest resistance start'     : Resistance.trendStartOffsetLatestR,  
#                             '5. Nr. of conf. latest R'       : Resistance.noLatestRConfirmed,  
#                             '5. Pos. rel. to latest R'       : Resistance.relPriceLatestR  
                            }, screen = base_universe)

result = run_pipeline(make_pipeline(), '2015-01-08', '2015-01-08')

#result = run_pipeline(make_pipeline(), '2020-07-13', '2020-07-13')

Now, to save on computation time, I decided to only compute short-term support, and resistance for those assets for which a long term trend has been identified, like so:

mask = longTermTrend.trendStartOffset.notnull()

The longTermTrend.TrendStartOffset factor output produces 500 rows (per trading day) as expected with 500 assets. As such, i would expect that the .notnull() would qualify it for use as a mask.
The weird thing is that if I use it to mask, it seems to work at first, but fails somewhere amid-computations???? Can't remember the exact error message) but it says the confirmTrendDailyData factor isn't returning the results as expexted.
However, If i run all factors without any masks (thus processing all 500 assets) everything runs fine?!!

The obvious solution would be to simply run it all on my entire universe, but it simply takes to much time with no additional added value.

Does anyone have a clue as to what could be causing this?