Back to Community
ValueError: Cannot convert NA to integer

Hi Community,

I've been really struggling to finish this backtest but apparently it seems there's an error on my code. I have checked Nans (even though i have explicitly rejected them on my make_pipeline function) and i have also check the data type introduced into my portfolio weights (context.weights variable) but i can't find out the solution. I would really appreciate you feedback.

Thanks a lot!!

Axel

Clone Algorithm
4
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
This is a template algorithm on Quantopian for you to adapt and fill in.
"""
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline import Pipeline
from quantopian.pipeline.data import Fundamentals
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.filters import QTradableStocksUS
import quantopian.optimize as opt
import quantopian.algorithm as algo
import pandas as pd

bottom_percentile = 20
up_percentile = 80


def initialize(context):
    """
    Called once at the start of the algorithm.
    """
    # Check cointegrated pairs every month
    schedule_function(rebalance, date_rules.month_start(), time_rules.market_open())
    'schedule_function(rebalance)'
    
    # Record tracking variables at the end of each day.
    schedule_function(record_vars,date_rules.month_start(),time_rules.market_open())
        
    # Create our dynamic stock selector.
    attach_pipeline(make_pipeline(), 'my_pipeline')

def make_pipeline():
    
    # Define a tradeable universe.
    Universe =  QTradableStocksUS()
    
    # Create basic indicators for each stock, such as its sector 
    # classification, country of domicile, last price and market_cap
    sector = Fundamentals.morningstar_sector_code.latest
    
    #Let's modify all of our valuation ratios to be LVRB
    
    # Price to earnings ratio
    PE = Fundamentals.pe_ratio.latest
    PE = PE.zscore(mask=PE.percentile_between(1,99))
    
    #EVCFO
    EVCFO =Fundamentals.enterprise_value/(Fundamentals.cfo_per_share.latest*Fundamentals.shares_outstanding.latest)
    EVCFO = EVCFO.zscore(mask=EVCFO.percentile_between(1,99))
    
    # Free cash flow
    FCF = 1.0/(Fundamentals.free_cash_flow.latest)
    FCF = FCF.zscore(mask=FCF.percentile_between(1,99))
    
    # Price to sales
    PS = Fundamentals.ps_ratio.latest
    PS = PS.zscore(mask=PS.percentile_between(1,99))
    
    
    factor = (PE+EVCFO+PS+FCF)/4.0
    
    Universe = (PE.notnull() & EVCFO.notnull() & PS.notnull() & FCF.notnull()& QTradableStocksUS())
    
    longs = factor.percentile_between(0,bottom_percentile,mask=Universe)
    shorts = factor.percentile_between(up_percentile,100,mask=Universe)
    

    return Pipeline(
            columns = { 'sector': sector,
                        'factor': factor,
                        'longs':longs,
                        'shorts':shorts
                      },
            screen=(longs|shorts)
            )


def rebalance(context, data):
    """
    Execute orders according to our schedule_function() timing.
    """
    context.dataset = pipeline_output('my_pipeline')
    'remove possible NANS'
    context.dataset.dropna()
    
    'Store our securities in a context variable and create and integer to get the total amount'
    'TAKE INTO ACCOUNT THAT THE NUMBER OF LONG AND SHORT'
    context.longs = context.dataset.loc[context.dataset['longs'] == True].index.tolist()
    context.shorts = context.dataset.loc[context.dataset['shorts'] == True].index.tolist()
    context.securities = context.longs + context.shorts
    context.N = len(context.securities)
    
    'Create weights according the pb ratio value, +50% longs -50% shorts'
    
    'longs'
    context.ll = context.dataset.loc[context.longs,'factor']
    'context.ll = context.ll.multiply(0.5/sum(context.ll))'
    context.ll = pd.Series(0.5/len(context.longs), index= context.ll.index)
    
    'shorts'
    context.ss = context.dataset.loc[context.shorts,'factor']
    'context.ss = context.ss.multiply(-0.5/sum(context.ss))'
    context.ss = pd.Series(-0.5/len(context.shorts), index = context.ss.index)
    
    'Create an unique list with all securities that have to traded.'
    context.weights = context.ll.append(context.ss)
    
    'before doing the portfolio allocation, retrieve the current portfolio.positions'
    context.current_positions = context.portfolio.positions
    print('Number of active positions: '+ str(len(context.current_positions)))
    
    'retrieve the equities that are not longer on the trading list.'
    context.closing_positions = set(context.current_positions)-set(context.weights.index.tolist())
    context.closing_positions = pd.Series(0.0, index = context.closing_positions)
    allocate(context,data,context.closing_positions)
    print('Number of positions to close'+str(len(context.closing_positions.index.tolist())))
    
    'Allocate the portfolio with the current weights'
    allocate(context,data,context.weights)
    pass


def record_vars(context, data):
    """
    Plot variables at the end of each day.
    """
    num_longs = len(context.longs)
    num_shorts = len(context.shorts)
    
    record(longs = num_longs,
          shorts = num_shorts,
          number_of_positions = len(context.portfolio.positions))
    
    record(portfolio_cash = context.portfolio.cash,
           leverage = context.account.leverage)
    pass

def allocate(context, data, weights):    
    # Set objective to match target weights as closely as possible, given constraints.
    # opt.TargetWeights takes as input a dictionary/series with assets as keys/index and weights as values.
    # Let's try how opt.MaximezeAlpha works
    # selected MaximizeAlpha because we believe our combined factor
    # ranking to be proportional to expected returns. This routine
    # will optimize the expected return of our algorithm, going
    # long on the highest expected return and short on the lowest.
        if len(weights.index) !=0:
            objective = opt.TargetWeights(weights)

            # Define constraints
            constraints = []
            #constraints.append(opt.MaxGrossExposure(1.0))

            algo.order_optimal_portfolio(
            objective=objective,
            constraints=constraints,
            )
There was a runtime error.
9 responses

That's a bug where the values to opt are all 0.0. Python complains about NA and integer and Q would like to know I think, catch that and make it user-friendly.

The easiest solution is to use try ... except, allows the backtest to continue and provides the opportunity to log info and do something, anything else.
The positive results here are simply from the minus sign flipping long short.

Clone Algorithm
5
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
This is a template algorithm on Quantopian for you to adapt and fill in.
"""
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline import Pipeline
from quantopian.pipeline.data import Fundamentals
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.filters import QTradableStocksUS
import quantopian.optimize as opt
import quantopian.algorithm as algo
import pandas as pd

bottom_percentile = 20
up_percentile = 80


def initialize(context):
    """
    Called once at the start of the algorithm.
    """
    # Check cointegrated pairs every month
    schedule_function(rebalance, date_rules.month_start(), time_rules.market_open())
    'schedule_function(rebalance)'
    
    # Record tracking variables at the end of each day.
    schedule_function(record_vars,date_rules.month_start(),time_rules.market_open())
        
    # Create our dynamic stock selector.
    attach_pipeline(make_pipeline(), 'my_pipeline')

def make_pipeline():
    
    # Define a tradeable universe.
    Universe =  QTradableStocksUS()
    
    # Create basic indicators for each stock, such as its sector 
    # classification, country of domicile, last price and market_cap
    sector = Fundamentals.morningstar_sector_code.latest
    
    #Let's modify all of our valuation ratios to be LVRB
    
    # Price to earnings ratio
    PE = Fundamentals.pe_ratio.latest
    PE = PE.zscore(mask=PE.percentile_between(1,99))
    
    #EVCFO
    EVCFO =Fundamentals.enterprise_value/(Fundamentals.cfo_per_share.latest*Fundamentals.shares_outstanding.latest)
    EVCFO = EVCFO.zscore(mask=EVCFO.percentile_between(1,99))
    
    # Free cash flow
    FCF = 1.0/(Fundamentals.free_cash_flow.latest)
    FCF = FCF.zscore(mask=FCF.percentile_between(1,99))
    
    # Price to sales
    PS = Fundamentals.ps_ratio.latest
    PS = PS.zscore(mask=PS.percentile_between(1,99))
    
    
    factor = (PE + EVCFO + PS + FCF) / 4.0
    
    Universe = (PE.notnull() & EVCFO.notnull() & PS.notnull() & FCF.notnull()& QTradableStocksUS())
    
    longs  = factor.percentile_between(0,bottom_percentile,mask=Universe)
    shorts = factor.percentile_between(up_percentile,100,mask=Universe)

    return Pipeline(
        columns = { 
            'fcf'   : FCF,
            'ps'    : PS,
            'pe'    : PE,
            'evcfo' : EVCFO,
            'sector': sector,
            'factor': factor,
            'longs' : longs,
            'shorts': shorts
        },
            screen=(longs|shorts)
    )


def rebalance(context, data):
    """
    Execute orders according to our schedule_function() timing.
    """
    context.dataset = pipeline_output('my_pipeline')
    'remove possible NANS'
    context.dataset.dropna()
    
    
    
    if 'log_pipe_done' not in context:       # show pipe info once
        df = context.dataset ; num = 4
        log_pipe(context, data, num, df) #, details=['bbr', 'alpha'])
        
        
    
    'Store our securities in a context variable and create and integer to get the total amount'
    'TAKE INTO ACCOUNT THAT THE NUMBER OF LONG AND SHORT'
    context.longs  = context.dataset.loc[context.dataset['longs'] == True].index.tolist()
    context.shorts = context.dataset.loc[context.dataset['shorts'] == True].index.tolist()
    context.securities = context.longs + context.shorts
    context.N = len(context.securities)
    
    'Create weights according the pb ratio value, +50% longs -50% shorts'
    
    'longs'
    context.ll = context.dataset.loc[context.longs,'factor']
    'context.ll = context.ll.multiply(0.5/sum(context.ll))'
    context.ll = pd.Series(0.5/len(context.longs), index= context.ll.index)
    
    'shorts'
    context.ss = context.dataset.loc[context.shorts,'factor']
    'context.ss = context.ss.multiply(-0.5/sum(context.ss))'
    context.ss = pd.Series(-0.5/len(context.shorts), index = context.ss.index)
    
    'Create an unique list with all securities that have to traded.'
    context.weights = context.ll.append(context.ss)
    
    'before doing the portfolio allocation, retrieve the current portfolio.positions'
    context.current_positions = context.portfolio.positions
    print('Number of active positions: '+ str(len(context.current_positions)))
    
    
    'retrieve the equities that are not longer on the trading list.'
    context.closing_positions = set(context.current_positions)-set(context.weights.index.tolist()) 
    
    if len(context.closing_positions):
        context.closing_positions = pd.Series(0.0, index = context.closing_positions)        
        allocate(context,data,context.closing_positions)
        
    print('Number of positions to close ' + str(len(context.closing_positions)))
    
    
    'Allocate the portfolio with the current weights'
    allocate(context,data,context.weights)

def record_vars(context, data):
    """
    Plot variables at the end of each day.
    """
    num_longs = len(context.longs)
    num_shorts = len(context.shorts)
    
    record(longs = num_longs,
          shorts = num_shorts,
          number_of_positions = len(context.portfolio.positions))
    
    record(portfolio_cash = context.portfolio.cash,
           leverage = context.account.leverage)

def allocate(context, data, weights):    
    # Set objective to match target weights as closely as possible, given constraints.
    # opt.TargetWeights takes as input a dictionary/series with assets as keys/index and weights as values.
    # Let's try how opt.MaximezeAlpha works
    # selected MaximizeAlpha because we believe our combined factor
    # ranking to be proportional to expected returns. This routine
    # will optimize the expected return of our algorithm, going
    # long on the highest expected return and short on the lowest.
    #log.info(len(weights))
    
    # The '0 and' has this turned off at the moment
    if   0 and    (weights.sort_values()[0] == 0 and weights.sort_values()[-1] == 0 ):
        log.info('       Weights are all zero')
        return
    
    try:
        if len(weights):
            constraints = []
            #constraints.append(opt.MaxGrossExposure(1.0))

            algo.order_optimal_portfolio(    # minus, flipping long short
                objective   = opt.TargetWeights(    -weights    ),
                constraints = constraints,
            )
    except Exception as e:
        log.info(e)
        
        df = context.dataset ; num = 4
        log_pipe(context, data, num, df) #, details=['bbr', 'alpha'])
        
def log_pipe(context, data, num, df, details=None):
    c = context
    c.log_pipe_done = 1
    log.info('len {}'.format(len(df)))
    if not len(df): return
    
    if isinstance(df, pd.Series):
        log.info('       min      mean       max   Series {}'.format(df.name))
        nan_count = len(df[df != df])
        nan_count = 'NaNs {}/{}'.format(nan_count, len(df)) if nan_count else ''
        log.info('    {}    {}    {}   {}'.format(
           ('%.2f' % df.min()) .rjust(6),
           ('%.2f' % df.mean()).rjust(6),
           ('%.2f' % df.max()) .rjust(6),
           nan_count
        ))
        return
        
    log.info('            min      mean       max')
    for col in df.columns:
        if col == 'sector': continue
        nan_count = len(df[col][df[col] != df[col]])
        nan_count = 'NaNs {}/{}'.format(nan_count, len(df)) if nan_count else ''
        log.info('{}    {}    {}    {}   {}'.format(col.rjust(18),
           ('%.2f' % df[col] .min()).rjust(6),
           ('%.2f' % df[col].mean()).rjust(6),
           ('%.2f' % df[col] .max()).rjust(6),
           nan_count
        ))

    if details == None: details = df.columns
    for detail in details:
        if detail == 'sector': continue
        log.info('_ _ _   {}   _ _ _\n'.format(detail))
        log.info('{} highs \n{}'.format(detail,
            df.sort_values(by=detail, ascending=False).head(num)))
        log.info('{} lows \n{}'.format(detail,
            df.sort_values(by=detail, ascending=False).tail(num)))

            
There was a runtime error.

Hi Blue,

First of all, thanks for your response. Despite not crashing, it seems that my closing orders are not being proceeded, so i understand that on that day no closing orders will be placed. I don't quite get the nature of the problem, i mean, what's happening on that specifid date to crash? I have used the same logic for a bigger timespan and it only crashes on that date apparently. Another solution that has come to me is to append my closing _positions_list to my trading_positions list, in order to don't have a unique list with values of 0.0, but it seems that has got the same problem. I am desperate to understand the nature of the problem. However, thanks for your contribution, hope that both can find a solution.

Axel

Thanks for reporting that, it's a pandas bug: https://github.com/pandas-dev/pandas/issues/14687

Try emailing [email protected] and with good fortune it might make its way to a developer. There could be nothing they can do or they might be able to capture the error and at least avoid the crash. Do want them to know in any case.

Expected any of the keys (securities) that are current positions would be closed since the values are all 0.0 when that happens.

Thanks @Blue, i've posted the ticket so let's cross fingers!

Any response?

Hi @Axel, @Blue,
For me, as someone who struggles constantly with python in general and with pandas in particular, i am in awe of people such as yourselves who can (not only) use pandas effectively and/ but can also go so far as to find bugs in it!!!! I'm sure i will thank you even more profusely when my python skills improve! Cheers, all the best, TonyM.

I'm getting this error message as well. Would upgrading pandas in the IDE fix it? try-except workaround works, but not ideal. Looks like the pandas version in the IDE is 0.18.1, which is about 3 years old?

@Q Support team,

Could you advice please?

@Blue Seahawk,

I'm using the below in 'rebalance' to remove any NaNs or Null values in my alpha-factor pandas series, but I still get this error message. Do you know why, or if there's a better way to do this? Why is it trying to convert to an integer anyway, when all other values are floats?? You don't reckon it's a bug in the Optimize API, rather than the pandas bug?

pipeline_data = pipeline_data[pd.notnull(pipeline_data)]  

Looks like that was from values not 0.0 yet very close like 4.351e-7.
Joakim thoughtfully used Quantopian's collaboration feature to share affected code with me.
Is np.isclose() involved somehow? I was seeing np errors showing up in debug where the object was just a pd Series.

Thanks for your help on this @Blue, really appreciate it!