Back to Community
Optimize API Required For Allocations

As mentioned here, we released the stable version of the Optimize API today. One thing mentioned in the post is that the Optimize API is required for an allocation - this post explains how that works.

Why this change?

The Optimize API allows Quantopian to intelligently adjust an algorithm's orders when external circumstances prevent us from achieving exactly the algorithm's desired portfolio. For example:

  • There may not be enough short inventory for a stock
  • The prime broker may restrict Quantopian from trading in certain stocks on a given day

Situations like these mean that an algorithm can't always get exactly what it wants. Optimize handles these kinds of situations by getting you to an achievable portfolio that's as close as possible to the originally desired portfolio, but also respects both the external constraints placed upon the algorithm and the constraints provided by the author.

Because these kind of situations can, and do, happen often in trading, we require algorithms to use Optimize in order to get an allocation, and every algorithm that has received an allocation to date uses Optimize.

How do I switch an existing algo over to Optimize?

If you'd like to easily port over an existing order_target_percent algorithm to use Optimize, your best bet is the TargetWeights objective. For example, if you wanted to switch over:

order_target_percent(sid(24), 0.5)  
order_target_percent(sid(5861), 0.5)  

to Optimize, you'd do:

order_optimal_portfolio(  
  objective=TargetWeights({sid(24): 0.5, sid(5861): 0.5}),  
  constraints=[],  
)

This is a fairly basic example, but there's a lot more you can do with Optimize - head over to the docs to see what you can do with the API.

What happens to my algos that don't use Optimize?

Algorithms that don't use Optimize will still be considered for an allocation. As a part of the diligence process, we'll work with you to update your algorithm to use Optimize.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

15 responses

Hi Abhijeet -

I'm not entirely clear what the Optimize API is doing. As you say, it "respects...the external constraints placed upon the algorithm" but what does this mean, precisely? For example, say this code is executed:

order_optimal_portfolio(  
  objective=TargetWeights({sid(24): 0.5, sid(5861): 0.5}),  
  constraints=[],  
)

Would you send the new weights to the broker, get some data back, complete the optimization, and then re-submit a new set of weights? Or does the broker get the new weights, apply another layer of optimization, and then send the orders on their merry way into the market? Or maybe since you know the universe of stocks the algo is using, the status of each stock is known prior to running the Optimize API (i.e. the algo is getting a real-time minutely (or more frequent?) feed of data from the broker, upon market open, for its universe)?

Also, what happens if order_optimal_portfolio is called multiple times within one trading minute (for both backtesting, paper trading, and real-money trading)? On the other post, you say "Optimize accounts for open market orders when determining target positions, unlike the order_target family of methods which only account for filled orders" which implies that if my algo calls order_optimal_portfolio more than once in a trading minute, for each subsequent call, any orders generated by prior calls will be considered completely unfilled (i.e. effectively not yet submitted)? Or are open orders only accounted for properly if order_optimal_portfolio is called at most once per trading minute?

Hi Grant,

This all happens before the order goes out to the broker. In a multi-strategy portfolio you have to consider the other algorithms as well as per-algo limits. In your example, your 50% AAPL portfolio might breach some risk constraints (e.g. maximum position concentration), or if you were to want to short the stock, it might not be available to short. The optimize call knows these constraints behind the scenes and will provide a portfolio that is actually feasible.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Thomas -

So what happens if opt.calculate_optimal_portfolio is used during backtesting? Will it incorporate the additional constraints? And if so, where does it get the data (e.g. availability of shorts)? Why not expose the data?

Still a lot of details missing here on the mechanics of the mysterious external constraints. No big deal, I suppose, but one can waste a lot of time by being confused (and contacting Q support) when complete documentation will clear things up at the outset.

@Grant: No additional constraints are applied during backtesting, only the ones you provide. This only applies for a fund-algorithm. We try to make as much of that data available as possible. Usually we do a first cut to where we can use it internally and then productize it properly to make it available to the larger Q community, which takes some time.

The global constraints are not really anything a user needs to be concerned with, especially not during algorithm development, as they don't apply there. The optimization tries to get as close to the target portfolio provided by the algorithm as possible.

Thanks Thomas -

Hmm? Wouldn't that result in backtests that could be significantly different from real trading? Or once you identify candidate algos for the fund, you would apply the external constraints and run a backtest?

Also, what about my question regarding multiple calls to order_optimal_portfolio within a trading minute? Does the algo now track pending orders within a trading minute and account for them?

This all happens before the order goes out to the broker.

Just curious how this works. If you are getting a feed of short availability, for example, and it is minutely, then might it be kinda stale? In theory, the order could be transmitted to the broker up to ~50 seconds later.

1) Yes, applying global portfolio level constraints to an algorithm's portfolio can/will cause that algorithm's performance to deviate from a run without global portfolio constraints. Portfolio construction is an interesting technical problem.

2) If you make multiple calls to order_optimal_portfolio with the same parameters, it is a no-op. If you change the parameters, it will assume the previous orders aren't filled. I can't think of a good reason to do that in the current environment, but we want to support it potentially for the future.

3) Short availability is another interesting technical problem. Nobody can ever guarantee they will fill orders. Orders sometimes fail to be filled. We can work to make it less frequent and be more resilient when it does happen.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Dan,

If I'm following correctly, when I run a backtest (and paper trading), only the constraints defined in my code are applied to the Optimize API (the "constraints provided by the author"). There could also be "external constraints placed upon the algorithm" which would not apply for ordinary algos. However, is it correct that external constraints would be imposed on a fund algo? And as a user, I would then see the effect on backtests? Or perhaps the external constraints would be applied, but only Quantopian could see the effect on the backtest? Or would external constraints only be applied at the point when the algo trades real money with the prime broker (i.e. only out-of-sample real money trading)?

Also, I'm still not clear on "Optimize accounts for open market orders when determining target positions, unlike the order_target family of methods which only account for filled orders" and when orders are considered pending versus submitted in backtesting and real-money trading. Under backtesting, presumably all orders are considered unfilled until the next whole trading minute, and the same would apply with your prime broker, assuming that you'll still be getting minutely feedback. Presumably, any open orders (partial fills left over from the prior trading minute and orders already submitted within the trading minute) will be taken into account by the Optimize API, and that order status is only available minutely, for both backtesting/paper trading and real money trading with your prime broker.

By the way, why not just use the name of your prime broker. Eventually, I'd think that as part of engaging institutional investors, you'd have to reveal it, per regulatory requirements, no?

Q - Thanks! This has saved me hundreds of lines of code.

Attached is a most simple algorithm as a starting point.

Clone Algorithm
47
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline import Pipeline, CustomFilter, CustomFactor
from quantopian.pipeline.data.builtin import USEquityPricing

import quantopian.optimize as opt
import quantopian.algorithm as algo

import numpy as np
 
def initialize(context):
    """
    Called once at the start of the algorithm.
    """ 
    schedule_function(trade, date_rules.every_day(), time_rules.market_open(minutes=30))
    schedule_function(record_leverage, date_rules.every_day(), time_rules.market_close())

    attach_pipeline(make_pipeline(context), 'my_pipeline')
    
    return
 
def before_trading_start(context, data):
    """
    Called every day before market open.
    New pipeline stocks are retrieved
    """

    context.output = pipeline_output('my_pipeline')
  
    return     


def trade(context, data):
    stock_dict = {}
    for stock in context.output.index:
        stock_dict[stock] = 1.0 / len( context.output.index )
        
    try:
        objective = opt.TargetWeights(stock_dict)
    	constraints=[opt.MaxGrossExposure(0.98)]
    	algo.order_optimal_portfolio(objective, constraints) 
    except Exception as e:
        log.info('@@@@@@@@@@@@@@@@@@@ EXCEPTION: Trade @@@@@@@@@@@@@@@@@@@@@')
        log.info('Exception is %s' %e)
	return

def record_leverage(context, data):
    record(leverage=context.account.leverage)


def make_pipeline(context):
    
    # Custom filter to return only sids in the list
    my_sid_filter = SidInList(
        sid_list = (   
            symbol('AAPL').sid,    # Apple
            symbol('AMZN').sid,    # Amazon
            symbol('MA').sid,    # Mastercard
         )
    )
    

      
    pipe = Pipeline(
            columns = {
                    'open' : USEquityPricing.open.latest,
                    'high' : USEquityPricing.high.latest,
                    'low' : USEquityPricing.low.latest,
                    'close' : USEquityPricing.close.latest,
                    'volume' : USEquityPricing.volume.latest,
                    },
            screen = my_sid_filter,
            )
    
    return pipe

class SidInList(CustomFilter):
    """
    Filter returns True for any SID included in parameter tuple passed at creation.
    Usage: my_filter = SidInList(sid_list=(23911, 46631))
    https://www.quantopian.com/posts/using-a-specific-list-of-securities-in-pipeline
    """    
    inputs = []
    window_length = 1
    params = ('sid_list',)

    def compute(self, today, assets, out, sid_list):
        out[:] = np.in1d(assets, sid_list)       
There was a runtime error.

It would be refreshingly transparent and interesting if Q could "open source" how the "global optimization" problem is being approached, both in selecting algos for the fund that will be accretive and applying the "external constraints" when executing the fund trading. I suppose it is understandable that the details would be kept confidential, but it is another opportunity to teach quantitative finance to the masses.

Having updated a bunch of code to use order_optimal_portfolio() I have found that it does not appear to exit positions before entering new ones.

Say I have 1 stock (MU) that has weight 1.0 with MaxGrossExposure(1.0). It fills.
Some ticks later I want to exit MU and replace it with INTC with weight = 1.0

With MaxGrossExposure=1.0 I would expect that MU would be exited and then INTC would be filled.

Unfortunately that doesn't seem to be happening. See for example April 27th in the attached backtest.

If I really have to manually sequence my calls to order_optimal_portfolio() to manage leverage then most of the code I just got rid of will have to come back.

Other than manually checking, is there a way to ensure the sequence of orders?

Clone Algorithm
25
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
'''
Day Trade Within Whole Numbers
by Ken Calhoun

Algorithm:
  1) Find stock getting ready to take out their 2-day high
     * just above a whole number
     * priced $15 to $50 (configurable) 
     * >15k shares per minute traded (configurable)
  2) Set buy-stop order 0.20 above whole number (configurable)
  3) On entry set stop at whole number
  4) Exit trade on 0.80 gain (configurable)
  
Note:
1) Could run a volume check every minute for 15k shares but
   to use it in pipeline we approximate it with daily
   15k*390 minutes = 5.85M shares per day
   
Extensions:
	1) You can set global_tolerance_to_whole_number to allow stocks at 13.98, 14.02, etc.
    2) You can cease entering new positions at end of day (configurable)
    3) You can set the time at end of day to close all positions (configurable)

John Glossner
[email protected]
http://Linkedin.com/in/glossner

MIT License
https://opensource.org/licenses/MIT 

2017 09 10 Initial version
'''
from quantopian.algorithm import attach_pipeline, pipeline_output, order_optimal_portfolio
from quantopian.pipeline import Pipeline
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.filters.morningstar import Q500US, Q1500US
from quantopian.pipeline import CustomFactor, CustomFilter
from collections import OrderedDict
import quantopian.optimize as opt

import numpy as np
import math

global_tolerance_to_whole_number = 0.05		#How close to whole number does close need to be
											#Must be less than open_position_delta
 
def initialize(context):
    """
    Called once at the start of the algorithm.
    """         
    #Stock Selection User Defined Variables
    context.open_position_delta  = 0.20      #0.2 would be price plus 20 cents to enter position
    context.close_position_delta = 0.80      #0.8 take profit at 80 cents
    context.desired_leverage     = 0.98      #Change up or down as desired

    context.min_stock_price      = 15.00     #min stock trading price. 
    context.max_stock_price      = 50.0      #max stock trading price
    context.min_volume_per_day	 = 5850000

    
    #Stop new positions near end of day
    schedule_function(cease_new_positions, date_rules.every_day(), time_rules.market_close(minutes=120))
    #Close at end of day no matter what
    schedule_function(close_all_positions, date_rules.every_day(), time_rules.market_close(minutes=30))
        
    ################################################################################

    context.max_leverage = [0]    	#for plotting
    context.portfolio_weights = {}	#Equity():weight
    context.stop_list = {}			#Equity():Stop class. All the securities and their stops
        
    attach_pipeline(make_pipeline(context), 'my_pipeline') 
    
    return


 
def before_trading_start(context, data):
    """
    Called every day before market open.
    Any positions not exited the previous day are added to exit list
    New pipeline stocks are retrieved
    """    
    context.end_of_day = False
    context.allow_opening_positions = True
    
    #If any positions didn't exit the previous day set them up to exit
    context.portfolio_weights = {}
    for pos in context.portfolio.positions:
        context.portfolio_weights[pos] = 0.0
        log.info("Did NOT EXIT at close. Stock= " + str(pos))
    
    context.output = pipeline_output('my_pipeline').dropna()

    #Print pipeline output 
    log.info('Number of stocks from pipeline= ' + str(len(context.output.index)))
    if len(context.output.index):
    	log.info(' \n'.join( sorted([stock_.symbol + ' prev close=  ' \
            + str(context.output['close'][stock_]) \
            +'  2day high= ' + str(context.output['2d_high'][stock_]) \
            for stock_ in context.output.index] )))    
    
    #Set stops for stocks identified in pipeline
    context.stop_list = {}    #Reset each day
    for stock in context.output.index:
        close_loss_stop = math.floor( context.output['2d_high'][stock] + global_tolerance_to_whole_number )
        close_profit_stop = close_loss_stop + context.close_position_delta
        open_stop = close_loss_stop + context.open_position_delta
        context.stop_list[stock] = Stop(context, data, stock, open_stop, close_profit_stop, close_loss_stop)
        
    return     

def handle_data(context,data):
    """
    Called first before any function scheduled at Open
    Called every minute in order of importance of actions
    """
    
    #Check stops and optimize portfolio
    # trade_fixed_stops() returns true if portfolio weights are updated
    if context.allow_opening_positions and trade_fixed_stops(context, data):
        optimize_portfolio(context, data)
        
    my_record_vars(context, data)
    return

    

def optimize_portfolio(context, data):
    '''
    Update positions based on weights
    '''
    if len(context.portfolio_weights) == 0: return
    objective = opt.TargetWeights(context.portfolio_weights)
    constraints=[opt.MaxGrossExposure(context.desired_leverage)]
    order_optimal_portfolio(objective, constraints) 
    print_orders(context, data)
    return

def cease_new_positions(context, data):
    context.allow_opening_positions = False
    return
    

def close_all_positions(context, data):
    '''
    Set target weights and call optimizer
    '''
    cancel_all_orders(context, data)
    if len(context.portfolio.positions) == 0: return
    
    context.portfolio_weights = {}
    for pos in context.portfolio.positions:
        context.portfolio_weights[pos] = 0.0
        if pos in context.stop_list: del context.stop_list[pos]
            
    objective = opt.TargetWeights(context.portfolio_weights)
    constraints=[opt.MaxGrossExposure(0.0)]
    order_optimal_portfolio(objective, constraints)
    
    print_orders(context, data)
        
    return


def trade_fixed_stops(context, data):
    '''
    Check if any stops have triggered and add them to a list
    Update weights and return True if portfolio needs rebalancing
    '''
    if len(context.stop_list) == 0: return
    
    openList = []
    closeList = []
    
    #keep any positions with 0 weights in the closeList
    for pos in context.portfolio.positions:
        if pos in context.portfolio_weights and context.portfolio_weights[pos] == 0.0:
            closeList.append(pos)
            
    for stock in context.stop_list:
        #New positions
        if allow_trade(context, data, stock) and not in_portfolio(context, data, stock) and context.stop_list[stock].open_stop_hit(): 
            openList.append(stock)
            
        #Closing positions
        if in_portfolio(context, data, stock) and context.stop_list[stock].close_stop_hit(): 
            closeList.append(stock)
        
    #Determine number of stocks to set portfolio weights
    if len(openList) or len(closeList) :	#have adjustments to make
        number_of_stocks = len(openList) + len(context.portfolio.positions) - len(closeList)

        #It is possible that the net change in stocks is 0
        if number_of_stocks == 0: number_of_stocks = len(context.portfolio.positions)            
        if number_of_stocks == 0: return False
                
        log.info('TOTAL NUMBER OF STOCKS FOR WEIGHTS= ' + str(number_of_stocks))
        
        #Make changes
        for stock in context.portfolio.positions:
            context.portfolio_weights[stock] = 1.0 / number_of_stocks
        
        for stock in openList:
            context.portfolio_weights[stock] = 1.0 / number_of_stocks
            
        for stock in closeList:
            context.portfolio_weights[stock] = 0.0
            if stock in context.stop_list: del context.stop_list[stock]
            
        return True	#adjustments were made
    return False
        

def cancel_all_orders(context, data, stock=None):
    '''
    If stock given cancels open orders for that Equity Object
    Otherwise closes all orders for every equity object
    '''
    if stock == None:
        orders = get_open_orders()
    else:
        orders = get_open_orders(stock)

    if len( orders ) > 0:
        for order in orders:
            cancel_order(order)
    return

def allow_trade(context, data, security):
    '''
    Logic for placing trades
    - can trade, no open orders, not in exit list, not a failed order
    '''
    if data.can_trade(security) \
    	and not get_open_orders(security):
        return True
    else:
        return False

def print_orders(context, data):
    open_orders = get_open_orders()
    if len(open_orders) == 0: 
        log.info('check_orders(): no active orders')
    
    else:
        for stock, orders in open_orders.iteritems():
            for ord in orders:
                if context.portfolio.positions[stock].amount == 0:
                	log.info('OPENING order for ' + str( ord.amount) + ' for ' + str(stock))
            	else:
            		log.info('CLOSING order for ' + str( ord.amount) + ' for ' + str(stock))
    return    
    

def in_portfolio(context, data, security):
    '''
    Determine if a specific security is in the portfolio
    '''
    for pos in context.portfolio.positions:
        if pos.sid == security.sid : return True
    return False


def my_record_vars(context, data):
    """
    Plot variables at the end of each day.
    """
    record(leverage=context.account.leverage)
    leverage = context.account.leverage
    for num in context.max_leverage:
        if leverage > num:
            context.max_leverage.remove(num)
            context.max_leverage.append(leverage)

    record(Max_Leverage = context.max_leverage[-1], leverage=leverage)

    return

class Stop(object):
    '''
    Class to handle fixed stops
    
    open_stop is a stop to buy
    close_profit_stop is a stop for when profit target is met
    close_loss_stop is a stop for max loss on a position
    '''
    def __init__(self, context, data, asset, open_stop=None, close_profit_stop=None, close_loss_stop=None):
        self.context = context
        self.data = data
        self.asset = asset
        self.open_stop = open_stop
        self.close_profit_stop = close_profit_stop
        self.close_loss_stop = close_loss_stop

    def close_stop_hit(self):
        price = self.data.current(self.asset, 'price')
    
        #Long
        if self.context.portfolio.positions[self.asset].amount > 0:
            #Loss
            if price <= self.close_loss_stop:
                log.info(str(self.asset) + '  close_loss_stop LONG hit at stop= ' \
                         + str(self.close_loss_stop) + '  price= ' + str(price))
                return True
            #Profit
            if price >= self.close_profit_stop:
                log.info(str(self.asset) +'  close_profit_stop LONG hit at stop=' \
                         +  str(self.close_profit_stop) + '  price= ' + str(price)) 
                return True
        #Short
        elif self.context.portfolio.positions[self.asset].amount < 0:
            #Loss
            if price >= self.close_loss_stop:
                log.info(str(self.asset) + '  close_loss_stop SHORT hit at stop= ' \
                         + str(self.close_loss_stop) + '  price= ' + str(price))
                return True
            #Profit
            if price <= self.close_profit_stop:
                log.info(str(self.asset) +'  close_profit_stop SHORT hit at stop=' \
                         +  str(self.close_profit_stop) + '  price= ' + str(price)) 
                return True        
        return False

    def open_stop_hit(self, short=False):
        price = self.data.current(self.asset, 'price')
        if short and price >= self.open_stop: return False
        if not short and price <= self.open_stop: return False
        log.info(str(self.asset) + ' open_stop hit at stop= ' + str(self.open_stop) \
                 +  '  price= ' + str(price))
        return True


class under_two_day_high(CustomFilter):
    '''
    Check if a stock is nearing its 2-day high
    close[-2] > close[-1]
    '''
    inputs = [USEquityPricing.close, USEquityPricing.high]
    window_length = 2

    def compute(self, today, assets, out, close, high):
        out[:] =  high[-2] > close[-1]
        
class two_day_high(CustomFactor):
    '''
    2-day high = high[-2]
    '''
    inputs = [USEquityPricing.high]
    window_length = 2

    def compute(self, today, assets, out, high):
        out[:] =  high[-2]
        
class whole_number(CustomFilter):
    '''
    Previous 2 day high should be a whole number
    global: tolerance within variable cents of whole number
    '''
    inputs = [USEquityPricing.high]
    window_length = 2
    
    def compute(self, today, assets, out, high ):
        frac, whole = np.modf(high)
        out[:] = np.logical_or(frac[-2] <= global_tolerance_to_whole_number, (1.0 - frac[-2]) <= global_tolerance_to_whole_number)

def make_pipeline(context):
    """
    A function to create our dynamic stock selector (pipeline). Documentation on
    pipeline can be found here: https://www.quantopian.com/help#pipeline-title
    """

    # Factors and/or Filters
    base_universe = Q1500US()

    yesterday_close = USEquityPricing.close.latest
    
    pipe_screen =  base_universe \
                  &( yesterday_close > context.min_stock_price ) \
                  &( yesterday_close < context.max_stock_price ) \
            	  &( USEquityPricing.volume.latest > context.min_volume_per_day) \
                  &( under_two_day_high(mask=base_universe ) ) \
                  &( whole_number(mask=base_universe) )
        
    pipe = Pipeline(
        screen = pipe_screen,
        columns = {
            'close'            : yesterday_close,
            '2d_high'		   : two_day_high(mask=base_universe)
        }
    )
    return pipe

There was a runtime error.

I used the new Optimize API in two of my contest entries, however the 2 year contest backtest results are significantly different from the research backtest results (7% vs. 58%). The set up is to place all orders at 9:32am and close all at 3:56pm. In research mode, this implementation is done correctly. I understand that this could change in contest/live trading environment depending on many factors such as whether orders are filled accordingly as intended by the code, short order inventory, etc. As mentioned... "Optimize handles these kinds of situations by getting you to an achievable portfolio that's as close as possible to the originally desired portfolio, but also respects both the external constraints placed upon the algorithm and the constraints provided by the author." In my example, close as possible, is not achieved by optimize api. Short order inventory is probably why my short orders at filled partially at one minute intervals, this I understand. However, buy market orders are also partially filled at minute intervals, this baffles me because these buy orders are not large and these are very liquid stocks. What would cause these buy market orders to not be filled instantaneously as there are in the research environment? These are what's causing the huge discrepancies in my results and very misleading in my choices of algorithms to field in the contest. I need more guidance on how to avoid these pitfalls.

@ Dan Dunn,

You said and I quote above:
1) Yes, applying global portfolio level constraints to an algorithm's portfolio can/will cause that algorithm's performance to deviate from a run without global portfolio constraints. Portfolio construction is an interesting technical problem.
3) Short availability is another interesting technical problem. Nobody can ever guarantee they will fill orders. Orders sometimes fail to be filled. We can work to make it less frequent and be more resilient when it does happen.

My personal experience as stated in prior reply regarding huge discrepancies in contest and research backresults makes me want to to scrutinize the details of how optimize api works. For example, does Q know apriori short inventory of stocks from broker and if so, is this info incorporated in the optimize algo. If it doesn't, then the stock selection process is hit or miss even if you pipe in Dollar Volume to get highly liquid stocks. Also, if you factor in other constraints like leverage/exposure, market neutral, sector neutral...is partial filling of shorts due to lack of short inventory adjusting the overall optimization process, i.e, which takes precendence?

Is it possible to 'catch' the positions the Optimize API is suggesting along with their weights inside your algorithm so you can then use that data to help your performance?

See calculate_optimal_portfolio on the help page:

Returns:

optimal_portfolio (pd.Series) – A Series containing portfolio weights
that maximize (or minimize) objective without violating any
constraints. Weights should be interpreted in the same way as
current_portfolio.