Back to Community
Having Trouble with Filters specifically NumExprFilter

In my make_pipeline() func, I have

HIGH_ROA = (roa > 0) if HIGH_ROA == 'True': SCORE +=1
where SCORE is the variable I want to screen for. I want stocks below 3 on the Score and above 8. However, my screen which looks like this:
SHORT = False if SCORE <= 2: SHORT = True LONG = False if SCORE >= 9: LONG = True # Our screen to remove certain stocks is_tradeable = ( close_price_below_SMA & high_dollar_volume & (SHORT | LONG) ) does not filter out any stocks because of SHORT or LONG. When I try to add SCORE to my Filter, I get a Value Error; int detected error. The docs imply it has something to do with filters.

10 responses

How is SCORE defined? Since Pipelines are cross-sectional, the return from the if statements should be an array of numbers - not just a single integer.

I define it in the pipeline as SCORE = 0. I see what you are saying. Can you explain how to do that on this platform? I have worked with pandas, not this abstract pipeline so I am a little unsure how to.

In your other post, you had this:

def make_pipeline():  
    # Return on Asset rules  
    roa = morningstar.operation_ratios.roa.latest  
    ROA = (roa > 0)                         # BOOL  
    CONDITION_TWO = .....         # BOOL  
    CONDITION_THREE = ....      # BOOL  
    SCORE = 0  

Pipeline computations (before you run them) are defined as a set of expressions which are performed on references to BoundColumn data. It may be better to define a CustomFactor like this:

from quantopian.factors import CustomFactor 

class Screen(CustomFactor):  
    inputs = [morningstar.operation_ratios.roa.latest , *args]  
    def compute(self, today, assets, out, roa, *args):  
        score = 0

        # Conditions  
        if (roa >0):  
            score += 1

        out[:] = score

def make_pipeline():  
    score_screen = Screen()  
    return Pipeline( columns = {'Screen':score_screen}  

This will return a Pandas DataFrame of SCORE for each asset for each date. You can then use that to define your trading logic, for example if Screen > 4 you can call order() with longs and Screen == 0 you can short.

You are a hero. Next question.
I have a class:

class Previous(CustomFactor):  
    # Returns value of input x trading days ago where x is the window_length  
    # Both the inputs and window_length must be specified as there are no defaults

    def compute(self, today, assets, out, inputs):  
        out[:] = inputs[0]  

that I use within that new Custom Factor class.


class SCORE(CustomFactor):  
    inputs = [morningstar.operation_ratios.roa.latest,  
              *args]  
    # how is roa defined  
    def compute(self, today, assets, roa, roa_1_year_ago, out, inputs):  
        score = 0  
        #----------------------------------------------------------------------------------------------------  
        # Return on Asset rules  
        roa = morningstar.operation_ratios.roa.latest  
        if (roa > 0): score += 1  
        #----------------------------------------------------------------------------------------------------  
       #----------------------------------------------------------------------------------------------------  
        roa_1_year_ago = Previous(inputs = [morningstar.operation_ratios.roa], window_length = 252)  
        if (roa > roa_1_year_ago): score +=1  
        #----------------------------------------------------------------------------------------------------

My question is two-fold, how would i list the roa_1_year_ago in the inputs to the function? Also, how does the inputs variable map to the compute parameters. In your answer I did not see roa being defined at all. Was I correct in defining roa in the compute method?

The inputs in compute are mapped by position in the inputs list.

So for example:

class SCORE(CustomFactor):  
    inputs = [#Field_1, #Field_2, ...]

    def compute(self, today, assets, out, f1, f2):  
        out[:] = # Do Something  

Here f1, f2 are arbitrary and are mapped to the first and second entries in inputs, respectively.

To use a CustomFactor within a CustomFactor, you can simply specify it in inputs. However, make sure to manually toggle window_safe = True.

class SCORE(CustomFactor):  
    prev = Previous()  
    inputs = [prev]  
    window_length = #  
    prev.window_safe = True  

This is what I have plus some:

class Previous(CustomFactor):  
    # Returns value of input x trading days ago where x is the window_length  
    # Both the inputs and window_length must be specified as there are no defaults

    def compute(self, today, assets, out, inputs):  
        out[:] = inputs[0]

class SCORE(CustomFactor):  
    window_length = 365  
    roa_1_year_ago = Previous(inputs = [morningstar.operation_ratios.roa], window_length = 252)  
    roa_1_year_ago.window_safe = True

    inputs = [  
                morningstar.operation_ratios.roa.latest,  
                roa_1_year_ago,  
    ]  
    def compute(self, today, assets, out, roa, roa_1_year_ago):  
        score = 0  
        #----------------------------------------------------------------------------------------------------  
        if (roa > 0): score += 1  
        if (roa > roa_1_year_ago): score +=1  
        #----------------------------------------------------------------------------------------------------  
        out[:] = score

when I run it, I get this error which involves what you said about window_safe:

NonWindowSafeInput: Can't compute windowed expression FS_SCORE([Latest(...), Latest(...), Latest(...), Latest(...), Previous(...), Previous(...), Latest(...), Previous(...), Latest(...), Latest(...), Previous(...), Previous(...), Latest(...), Latest(...), Previous(...), Previous(...), Latest(...), Previous(...)], 365) with windowed input Latest([operation_ratios<US>.roa], 1).

This error seems odd to me because I have never had to set the Latest to window safe before:
should i declare a variable to hold it before the input block and see it to window_safe there and load it into input like the Previous() variables?

Full Error:
``` NonWindowSafeInputTraceback (most recent call last)
in ()
241
242
--> 243 my_pipe = make_pipeline()
244 result = run_pipeline(my_pipe, '2005-01-01', '2006-01-01').dropna()
245 print 'Head: ', result.head(20)

in make_pipeline()
220 close_price_below_SMA = (latest_close < mean_close_200)
221
--> 222 score = SCORE()
223 score.window_safe = True
224

/build/src/qexec_repo/zipline_repo/zipline/pipeline/mixins.pyc in new(cls, inputs, outputs, window_length, mask, dtype, missing_value, ndim, **kwargs) 138 missing_value=missing_value,
139 ndim=ndim,
--> 140 **kwargs
141 )
142

/build/src/qexec_repo/zipline_repo/zipline/pipeline/term.pyc in new(cls, inputs, outputs, window_length, mask, domain, *args, **kwargs) 508 window_length=window_length,
509 domain=domain,
--> 510 *args, **kwargs
511 )
512

/build/src/qexec_repo/zipline_repo/zipline/pipeline/term.pyc in new(cls, domain, dtype, missing_value, window_safe, ndim, *args, **kwargs) 133 ndim=ndim,
134 params=params,
--> 135 *args, **kwargs
136 )
137 return new_instance

/build/src/qexec_repo/zipline_repo/zipline/pipeline/term.pyc in _init(self, inputs, outputs, window_length, mask, *args, **kwargs) 516 self.window_length = window_length
517 self.mask = mask
--> 518 return super(ComputableTerm, self)._init(*args, **kwargs)
519
520 @classmethod

/build/src/qexec_repo/zipline_repo/zipline/pipeline/term.pyc in _init(self, domain, dtype, missing_value, window_safe, ndim, params) 275 # should set this flag to True.
276 self._subclass_called_super_validate = False
--> 277 self._validate()
278 assert self._subclass_called_super_validate, (
279 "Term._validate() was not called.\n"

/build/src/qexec_repo/zipline_repo/zipline/pipeline/factors/factor.pyc in _validate(self) 1567 def _validate(self):
1568 try:
-> 1569 super(CustomFactor, self)._validate()
1570 except UnsupportedDataType:
1571 if self.dtype in CLASSIFIER_DTYPES:

/build/src/qexec_repo/zipline_repo/zipline/pipeline/mixins.pyc in _validate(self) 40 """
41 def _validate(self):
---> 42 super(PositiveWindowLengthMixin, self)._validate()
43 if not self.windowed:
44 raise WindowLengthNotPositive(window_length=self.window_length)

/build/src/qexec_repo/zipline_repo/zipline/pipeline/mixins.pyc in _validate(self) 85
86 def _validate(self):
---> 87 super(RestrictedDTypeMixin, self)._validate()
88 assert self.ALLOWED_DTYPES is not NotSpecified, (
89 "ALLOWED_DTYPES not supplied on subclass "

/build/src/qexec_repo/zipline_repo/zipline/pipeline/term.pyc in _validate(self) 582 for child in self.inputs:
583 if not child.window_safe:
--> 584 raise NonWindowSafeInput(parent=self, child=child)
585
586 def _compute(self, inputs, dates, assets, mask):

NonWindowSafeInput: Can't compute windowed expression FS_SCORE([Latest(...), Latest(...), Latest(...), Latest(...), Previous(...), Previous(...), Latest(...), Previous(...), Latest(...), Latest(...), Previous(...), Previous(...), Latest(...), Latest(...), Previous(...), Previous(...), Latest(...), Previous(...)], 365) with windowed input Latest([operation_ratios.roa], 1).

I guessing, I should make the variables outside of the Score class....you had Previous() with no arguments. Geez this is a lot less intuitive than I thought :(

Have you used morningstar.operation_ratios.roa.latest as an input to a CustomFactor before or simply by calling it? The WindowSafe errors generally only come from CustomFactors. I had Previous() with no arguments previously as I thought you would fill the arguments in later - what you've done is correct.

Try simply manually toggling all of the inputs to window_safe = True.

class Previous(CustomFactor):  
    # Returns value of input x trading days ago where x is the window_length  
    # Both the inputs and window_length must be specified as there are no defaults  
    # Sets all inputs to window_safe  
    for factor in inputs:  
        factor.window_safe = True

    def compute(self, today, assets, out, inputs):  
        out[:] = inputs[0]

class SCORE(CustomFactor):  
    window_length = 1 # Change this to 1  
    roa_1_year_ago = Previous(inputs = [morningstar.operation_ratios.roa], window_length = 252)  
    roa_1_year_ago.window_safe = True

    inputs = [  
                morningstar.operation_ratios.roa.latest,  
                roa_1_year_ago,  
    ]  
    morningstar.operation_ratios.roa.latest.window_safe = True # Add this

    def compute(self, today, assets, out, roa, roa_1_year_ago):  
        score = 0  
        #----------------------------------------------------------------------------------------------------  
        if (roa > 0): score += 1  
        if (roa > roa_1_year_ago): score +=1  
        #----------------------------------------------------------------------------------------------------  
        out[:] = score  

If I understand what you are trying to do here, you could actually do this with one CustomFactor.


class SCORE(CustomFactor):  
    window_length = 252 # Returns Operation Ratios for the last 252 days  
    inputs = [morningstar.operation_ratios.roa]  
    morningstar.operation_ratios.roa.window_safe = True

    def compute(self, today, assets, out, roa):  
        score = 0  
        #----------------------------------------------------------------------------------------------------  
        if (roa[-1] > 0):  
            score += 1  
        if (roa[-1] > roa[0]):  
            score +=1  
        #----------------------------------------------------------------------------------------------------  
        out[:] = score  

I changed my format to be your final listed layout with OneCustom Factor: with

        if (roa[-1] > 0):  
            score += 1  

I get the error "ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()" which is weird because nd arrays should be usable that way.

Full Error:
``` ValueErrorTraceback (most recent call last)
in ()
184
185 my_pipe = make_pipeline()
--> 186 result = run_pipeline(my_pipe, '2005-01-01', '2006-01-01').dropna()
187 print 'Head: ', result.head(20)
188 print 'Number of securities that passed the filter: %d' % len(result)

/build/src/qexec_repo/qexec/research/api.py in run_pipeline(pipeline, start_date, end_date, chunksize) 488 chunksize,
489 pipeline_engine,
--> 490 holdout_manager,
491 )
492

/build/src/qexec_repo/qexec/research/_api.pyc in inner_run_pipeline(pipeline, start_date, end_date, chunksize, engine, holdout_manager) 732 adjusted_start_date,
733 adjusted_end_date,
--> 734 chunksize=chunksize,
735 )
736

/build/src/qexec_repo/zipline_repo/zipline/pipeline/engine.pyc in run_chunked_pipeline(self, pipeline, start_date, end_date, chunksize) 345 chunksize,
346 )
--> 347 chunks = [self.run_pipeline(pipeline, s, e) for s, e in ranges]
348
349 if len(chunks) == 1:

/build/src/qexec_repo/zipline_repo/zipline/pipeline/engine.pyc in run_pipeline(self, pipeline, start_date, end_date) 326 )
327
--> 328 results = self.compute_chunk(graph, dates, assets, initial_workspace)
329
330 return self._to_narrow(

/build/src/qexec_repo/zipline_repo/zipline/pipeline/engine.pyc in compute_chunk(self, graph, dates, sids, initial_workspace) 586 mask_dates,
587 sids,
--> 588 mask,
589 )
590 if term.ndim == 2:

/build/src/qexec_repo/zipline_repo/zipline/pipeline/mixins.pyc in _compute(self, windows, dates, assets, mask) 216 inputs = format_inputs(windows, inputs_mask)
217
--> 218 compute(date, masked_assets, out_row, *inputs, **params)
219 out[idx][out_mask] = out_row
220 return out

in compute(self, today, assets, out, roa, fcf, total_assets, long_term_debt, current_ratio, equity_repurchases, equity_issuance, gross_margin, total_revenue, asset_turnover_ratio)
90 #----------------------------------------------------------------------------------------------------
91 # First condition
---> 92 if (roa[-1] > 0):
93 score += 1
94 #----------------------------------------------------------------------------------------------------

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```

is [-1] the current period and [0] before it?