Back to Community
Bug: order_optimal_portfolio tosses small weights

This has been driving me nuts. I've been trying to order portfolios consisting of the entire QTradableStocksUS universe (2000+ stocks) and ending up under-leveraged and with only 600 positions. It's not an issue with partial fills. I've boiled it down to a bug in order_optimal_portfolio.

To illustrate, here's an algorithm that uses rank & demean to create a nice even gradient of portfolio weights from a nonsense alpha factor.

pipLen is the number of values returned by pipeline. posLen is the number of positions held at the end of the day. As you can see, the number of positions held is already lagging the number of position weights fed into order_optimal_portfolio.

Clone Algorithm
6
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Rubbish Example
"""
import quantopian.algorithm as algo
import quantopian.optimize as opt
from quantopian.pipeline import Pipeline
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.filters import QTradableStocksUS, StaticAssets


def initialize(context):
    algo.schedule_function(rebalance, algo.date_rules.every_day(), algo.time_rules.market_open())
    algo.schedule_function(rec_vars, algo.date_rules.every_day(), algo.time_rules.market_close())
    algo.attach_pipeline(make_pipeline(), 'pipeline')


def make_pipeline():
    base_universe = QTradableStocksUS() & (~StaticAssets(symbols('FCE_A')))
    
    yesterday_close = USEquityPricing.close.latest
    return Pipeline(
        columns={
            'score': yesterday_close.rank(mask=base_universe).demean(),
        },
        screen=base_universe
    )


def rebalance(context, data):
    output = algo.pipeline_output('pipeline')
    record(pipLen = len(output.index))
    
    weight = output.score / output.score.abs().sum()
    algo.order_optimal_portfolio(
        objective=opt.TargetWeights( weight ),
        constraints=[  ],
    )
    
    
def rec_vars(context, data):
    record(posLen = len(context.portfolio.positions))
There was a runtime error.
6 responses

What happens if we decrease all the weights by 1/10th and increase the starting capital from $10mm to $100mm? You'd think the effect on portfolio construction would be negligible. However, this is not the case. It doesn't order anything, because it clips away any weights below some 0.01%.

Clone Algorithm
6
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Rubbish Example
"""
import quantopian.algorithm as algo
import quantopian.optimize as opt
from quantopian.pipeline import Pipeline
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.filters import QTradableStocksUS, StaticAssets


def initialize(context):
    algo.schedule_function(rebalance, algo.date_rules.every_day(), algo.time_rules.market_open())
    algo.schedule_function(rec_vars, algo.date_rules.every_day(), algo.time_rules.market_close())
    algo.attach_pipeline(make_pipeline(), 'pipeline')


def make_pipeline():
    base_universe = QTradableStocksUS() & (~StaticAssets(symbols('FCE_A')))
    
    yesterday_close = USEquityPricing.close.latest
    return Pipeline(
        columns={
            'score': yesterday_close.rank(mask=base_universe).demean(),
        },
        screen=base_universe
    )


def rebalance(context, data):
    output = algo.pipeline_output('pipeline')
    record(pipLen = len(output.index))
    
    weight = output.score / output.score.abs().sum() * 0.10
    
    algo.order_optimal_portfolio(
        objective=opt.TargetWeights( weight ),
        constraints=[  ],
    )
    
    
def rec_vars(context, data):
    record(posLen = len(context.portfolio.positions))
There was a runtime error.

Using the old order_target... functions instead of order_optimal_portfolioand it works exactly as one would expect.

There's something wrong with order_optimal_portfolio. How is anybody constructing diversified portfolios with a smooth gradient of portfolio weights if the function simply tosses all the positions weighted lower than 0.01%? 0.01% times ~2000 QTU stocks means that up to 20% of an intended portfolio ($2mm out of a $10mm book) could be unaccounted for. That's not an insignificant contribution!

Clone Algorithm
6
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Rubbish Example
"""
import quantopian.algorithm as algo
import quantopian.optimize as opt
from quantopian.pipeline import Pipeline
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.filters import QTradableStocksUS, StaticAssets


def initialize(context):
    algo.schedule_function(rebalance, algo.date_rules.every_day(), algo.time_rules.market_open())
    algo.schedule_function(rec_vars, algo.date_rules.every_day(), algo.time_rules.market_close())
    algo.attach_pipeline(make_pipeline(), 'pipeline')


def make_pipeline():
    base_universe = QTradableStocksUS() & (~StaticAssets(symbols('FCE_A')))
    
    yesterday_close = USEquityPricing.close.latest
    return Pipeline(
        columns={
            'score': yesterday_close.rank(mask=base_universe).demean(),
        },
        screen=base_universe
    )


def rebalance(context, data):
    output = algo.pipeline_output('pipeline')
    record(pipLen = len(output.index))

    weight = output.score / output.score.abs().sum()
    weight *= 0.1 # divide portfolio weights by 1/10th
    
    # Order using order function instead of optimal_portfolio function
    for s, w in weight.items():
        order_target_percent(s, w)
    for s in context.portfolio.positions:
        if s not in weight.keys():
            order_target(s, 0)
    
    #algo.order_optimal_portfolio(
    #    objective=opt.TargetWeights( weight ),
    #    constraints=[  ],
    #)
    
    
def rec_vars(context, data):
    record(posLen = len(context.portfolio.positions))
There was a runtime error.

In case run_optimization() might shed some light on it.

Clone Algorithm
1
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Rubbish Example

Code added from https://www.quantopian.com/posts/optimization-weights-logging
"""
import quantopian.algorithm as algo
import quantopian.optimize as opt
from quantopian.pipeline import Pipeline
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.filters import QTradableStocksUS, StaticAssets
import pandas as pd

def initialize(context):
    algo.schedule_function(rebalance, algo.date_rules.every_day(), algo.time_rules.market_open())
    algo.schedule_function(rec_vars, algo.date_rules.every_day(), algo.time_rules.market_close())
    algo.attach_pipeline(make_pipeline(), 'pipeline')

    context.show_weights   = 1   # log weights before and after optimize
    context.show_positions = 0   # log symbols of those held

def make_pipeline():
    base_universe = QTradableStocksUS() & (~StaticAssets(symbols('FCE_A')))
    
    yesterday_close = USEquityPricing.close.latest
    return Pipeline(
        columns={
            'score': yesterday_close.rank(mask=base_universe).demean(),
        },
        screen=base_universe
    )


def rebalance(context, data):
    output = algo.pipeline_output('pipeline')
    record(pipLen = len(output.index))
    
    weight = output.score / output.score.abs().sum()
    
    c = context

    c.alpha = weight
    
    if not len(c.alpha):
        print('no alpha')
        return

    conc  = 1.0 / len(c.alpha)   # Values for PositionConcentration
    #conc *= 3.8                  # for trying multipliers like 3.0 or .8

    # Switch between MaximizeAlpha & TargetWeights
    #objective = opt.MaximizeAlpha(c.alpha)
    objective = opt.TargetWeights(c.alpha)

    constraints = []
    do_constraints = 0      # 0 for an easy way to test empty constraints
    if do_constraints:
        constraints = [
            #opt.DollarNeutral(),
            opt.MaxGrossExposure(1.0),
            opt.PositionConcentration.with_equal_bounds(-conc, conc),
            #opt.experimental.RiskModelExposure(c.risk_loading_pipeline, version=opt.Newest),
            #opt.FactorExposure(pipeline_data[['beta']], min_exposures={'beta': -0.05}, max_exposures={'beta':  0.05},
        ]

    if 'init_done' not in c: show_settings(c, objective, constraints, conc)

    ''' The heart of this code. Return from run_optimization() has old vs new weights. '''
    if c.show_weights:
        # https://www.quantopian.com/help#quantopian_optimize_run_optimization
        result = opt.run_optimization(
            objective   = objective,
            constraints = constraints,
        )
        show_opt_weights(c, result.old_weights, result.new_weights)    # Show some changes

    algo.order_optimal_portfolio(
        objective   = objective,
        constraints = constraints,
    )

def show_opt_weights(c, old, new):    # Show some optimization changes
    sids      = []   # list of security id's to track exclusively, like [24, 32790]
    max_lines = 9    # for Open & Close. Change is allowed to run wild till limit.
    p = 8            # padding, for vertical alignment
 
    ''' Closing positions '''
    closes = []
    for s in old.sort_values().index:
        if sids and s.sid not in sids: continue
        if old.min() == 0.0 and old.max() == 0.0: break
        if old[s] == 0.0: continue
        if new[s] != 0.0: continue  # maybe better if not *near* zero
        closes.append('{}  {} => {}  {}'.format(
                ('%.4f' % c.alpha[s] if s in c.alpha else '0').rjust(p),
                ('%.4f' % old[s]).rjust(p), '0'.rjust(p), s.symbol))
    if closes:
        count = 0 ; exceeded = 0
        out  = '\nClose'
        out += '\n{}  {}   {}\n'.format('alpha'.rjust(p), 'old'.rjust(p), 'new'.rjust(p))
        for cl in closes:
            count += 1
            if count > max_lines:
                exceeded = 1
                break

            out += "{}\n".format( cl )
            if len(out) > 950:
                log.info(out)  # this batch
                out = '\n{}  {}   {}\n'.format('alpha'.rjust(p), 'old'.rjust(p), 'new'.rjust(p))

        if exceeded:
            out += ('    {} more'.format(len(closes) - max_lines))

        log.info(out)  # anything remaining

    ''' Opening positions '''
    opens = []
    for s in new.sort_values().index:
        if sids and s.sid not in sids: continue
        if old[s] != 0.0: continue
        if new[s] == 0.0: continue
        opens.append('{}  {} => {}  {}'.format(
                ('%.4f' % c.alpha[s] if s in c.alpha else '0').rjust(p),
                '0'.rjust(p), ('%.4f' % new[s]).rjust(p), s.symbol))
    if opens:
        count = 0 ; exceeded = 0
        out  = '\nOpen'
        out += '\n{}  {}   {}\n'.format('alpha'.rjust(p), 'old'.rjust(p), 'new'.rjust(p))
        for o in opens:
            count += 1
            if count > max_lines:
                exceeded = 1
                break

            out += "{}\n".format( o )
            if len(out) > 950:
                log.info(out)  # this batch, then restart
                out = '\n{}  {}   {}\n'.format('alpha'.rjust(p), 'old'.rjust(p), 'new'.rjust(p))

        if exceeded:
            out += ('    {} more'.format(len(opens) - max_lines))
        log.info(out)  # anything remaining

    ''' Changing positions, weights that are changing '''
    changes = pd.DataFrame([], columns=['chng', 'info'])
    for s in new.index:
        if sids and s.sid not in sids: continue
        if old[s] == 0.0: continue
        if new[s] == 0.0: continue
        if old[s] == new[s]: continue

        changes.loc[s, 'chng'] = ('%.1f' % (100 * new[s] / old[s])).rjust(p)
        changes.loc[s, 'info'] = ('{}  {} => {}  {}%    {}'.format(
            ('%.4f' % c.alpha[s] if s in c.alpha else '_').rjust(p),
            ('%.4f' % old[s]).rjust(p), ('%.4f' % new[s]).rjust(p),
            ('%.1f' % (100 * new[s] / old[s])).rjust(p), s.symbol ))

    if not len(changes): return

    changes.sort_values(by='chng', ascending=False, inplace=True)

    out  = '\nChanges'
    out += '\n{} {}    {}  {}\n'.format('alpha'.rjust(p),
                    'old'.rjust(p), 'new'.rjust(p), 'pct'.rjust(p))

    for s in changes.index:
        out += "{}\n".format( changes.loc[s]['info'] )
        if len(out) > 950:
            log.info(out)  # this batch
            out = '\n{} {}    {}  {}\n'.format('alpha'.rjust(p),
                            'old'.rjust(p), 'new'.rjust(p), 'pct'.rjust(p))
    log.info(out)  # anything remaining

def show_settings(c, objective, constraints, conc):
    '''
    INFO <quantopian.optimize.objectives.TargetWeights object at 0x7f84b4742a10>
    INFO [MaxGrossExposure(1.0), PositionConcentration]
    INFO conc      : 0.03333
    INFO alpha min : -0.03118
    INFO alpha mean: -0.00000
    INFO alpha max : 0.03118
    INFO alpha len : 30
    '''
    log.info(objective)
    log.info(constraints)
    log.info('conc      : {}'.format('%.5f' % conc))
    log.info('alpha min : {}'.format('%.5f' % c.alpha.min()))
    log.info('alpha mean: {}'.format('%.5f' % c.alpha.mean()))
    log.info('alpha max : {}'.format('%.5f' % c.alpha.max()))
    log.info('alpha len : {}'.format(len(c.alpha)))
    c.init_done = 1

def show_positions(context):
    # Like INFO Held: APRN ASNA ATRS AUY AVP BLDP BTE BTG CHK DNR DRYS EGO GSAT GTE JCP LC MUX NGD NVAX OPK PDLI PDS PLUG RIGL SGYP STNG TRQ TTI WFT ZNGA
    out = 'Held: '
    for s in sorted([str(s.symbol) for s in context.portfolio.positions]):
        out += "{} ".format( s )
        if len(out) > 1011:
            log.info(out)  # this batch
            out = ''       # start over
    log.info(out)  # anything remaining

    
def rec_vars(context, data):
    record(posLen = len(context.portfolio.positions))
    
    
There was a runtime error.

The magic number is .0001. If the absolute value of a weight is less than .0001 it will effectively be 'rounded' to zero. That's what's causing the behavior you are seeing.

Before getting into the 'why' let's look at the 'what'. For a $10M portfolio, the order_optimal_portfolio method will effectively not open any position with an absolute weight less than .0001 or $1000 (.0001 x $10,000,000). It will also close any existing position with a value less than $1000. For a stock that trades at $20 that's 50 shares. That of course is a percentage. For a $1M portfolio that minimum is only $100.

Why is this? This was introduced to 'fix' the issue of the optimizer leaving only a few shares of a stock in one's portfolio. This issue had the undesirable consequence of potentially leaving a stock, which may no longer be in the QTradableUniverse (QTU), in one's portfolio. Having too many stocks for too long not in the QTU can disqualify an algo from the contest. So, now the order_optimal_portfolio method will close any positions with a portfolio weight less than .0001 to avoid this. The behavior now also acts as a behind the scenes 'housekeeper' closing (or never opening) very small positions.

This minimum weight is actually implemented as adding an extra penalty on these target-zero positions. It isn't a "hard constraint" that a position can never be under .0001. If the optimizer cannot find any other solution to satisfy all the constraints there is the potential for some positions to be weighted less than .0001. It effectively acts as a "soft constraint".

I made a change to the algo above which "clips" all the weights to a value a little more than .0001. This ensures all the target weights are above that magic number and do not get 'rounded' to zero. The algo now runs as expected. The number of securities ordered by order_optimal_portfolio is generally equal to the number of securities returned by the pipe. There are times where the portfolio positions are greater than the pipeline however, these are due to close orders not completely filling by the end of the day.

Hope that helps explain the sometimes baffling behavior of the optimizer.

Clone Algorithm
1
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Rubbish Example
"""
import quantopian.algorithm as algo
import quantopian.optimize as opt
from quantopian.pipeline import Pipeline
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.filters import QTradableStocksUS, StaticAssets


def initialize(context):
    algo.schedule_function(rebalance, algo.date_rules.every_day(), algo.time_rules.market_open())
    algo.schedule_function(rec_vars, algo.date_rules.every_day(), algo.time_rules.market_close())
    algo.attach_pipeline(make_pipeline(), 'pipeline')


def make_pipeline():
    base_universe = QTradableStocksUS() & (~StaticAssets(symbols('FCE_A')))
    
    yesterday_close = USEquityPricing.close.latest
    return Pipeline(
        columns={
            'score': yesterday_close.rank(mask=base_universe).demean(),
        },
        screen=base_universe
    )


def rebalance(context, data):
    output = algo.pipeline_output('pipeline')
    record(pipLen = len(output.index))
    
    weight = output.score / output.score.abs().sum()
    
    # Let's ensure no abs values of weight are smaller than the min of .0001
    # That is 1/100 of 1%. For a $10M portfolio that's $1000.
    min_target_weight = .0001
    some_tiny_amount = .000001
    adjusted_target = min_target_weight + some_tiny_amount
    
    weight[weight>0] = weight[weight>0].clip(lower=adjusted_target)
    weight[weight<0] = weight[weight<0].clip(upper=-adjusted_target)

    algo.order_optimal_portfolio(
        objective=opt.TargetWeights( weight ),
        constraints=[  ],
    )
    
    
def rec_vars(context, data):
    record(posLen = len(context.portfolio.positions))
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Awesome explanation Dan, as always. Thank you!

Thanks, Dan. I agree that a $1000 position in a $10mm portfolio is insignificant. However, once you have 600 $1000 positions disappearing, it starts causing unintended consequences with the gross leverage and also some effect on reducing the amount of diversification present.

I tend to use the opt.CannotHold(set(context.portfolio.positions) - set(qtu)) constraint to make sure I'm not holding any non-QTU positions. I also make sure their weights are set to 0.

Thanks for the clip example code! That'll probably help. I'll just round up to the cutoff for weights that are close enough.