Back to Community
Optimize API Generally Available

Today, we're announcing the stable release of the Optimize API. A number of classes and API methods have changed since its first experimental release, and this post details the changes.

Optimize is the recommended method for placing orders on Quantopian. It uses advanced optimization techniques to move your portfolio from its current state to your desired state, managing constraints and objectives that you can pass in. There are several reasons why it's better to use Optimize than order_target_percent or other manual ordering methods:

  • Optimize allows you to easily specify constraints when ordering, to manage your algorithm's leverage, position concentration, and exposures to risk factors.
  • Optimize accounts for open market orders when determining target positions, unlike the order_target family of methods which only account for filled orders.
  • Because Optimize is a declarative API, there's less manual order management to take care of, and the focus can instead be on alpha discovery and portfolio construction. You simply specify what you'd like your desired portfolio to be, either in the form of portfolio weights or alpha vectors, and Optimize handles placing the orders.
  • With the MaximizeAlpha objective, you now have the option of specifying alpha vectors as your ordering objectives, unlike the previous ordering methods which all needed a quantity of a specifically named stock.

Algorithms need to use the Optimize API in order to receive an allocation. We require this because having an objective and constraints allows us to more effectively respect an algorithm's desires if we're unable to hold the algorithm's exact optimal portfolio (e.g., because an asset is halted intraday or restricted by our prime broker). This post contains a lot more detail.

If you're new to Optimize and want to get started, the best place is the help docs.

If you'd like to see fully worked examples, go here to see Optimize in a notebook, or clone the attached backtest to see Optimize in an algorithm.

If you've already been using Optimize during the experimental release, here's what's new:

New Features

  • The error messages resulting from InfeasibleConstraints errors are now much more detailed. You'll see details about the constraints violated by the current portfolio, the empty portfolio - and when using the TargetWeights objective, the target portfolio as well.
  • A new method called run_optimization has been added. run_optimization is similar to calculate_optimal_portfolio, but returns an OptimizationResult - an object containing diagnostic information about the attempted portfolio optimization. run_optimization is primarily useful as a debugging tool.
  • Docs! Full documentation is now available for Optimize.
  • Objectives can now be rendered as LaTeX in research notebooks via their .math attribute. For example:

Parameter Changes

  • The universe parameter to order_optimal_portfolio is deprecated. Calls to order_optimal_portfolio will ignore the universe parameter, if it's passed in. In the original release of the API, order_optimal_portfolio required the user to pass provide a list of assets to act as the "optimization universe". The universe was used by the optimizer to define the assets that could be considered as potential candidates for orders and/or positions. After working with the Optimize API, however, we found that in practice the only reasonable choice for universe was the union of three sets of assets:

    • Assets referenced by the algorithm's objective.
    • Assets held in the algorithm's portfolio.
    • Assets with unfilled open orders.

    We also found that Optimize API users often passed a much larger list of assets than was desirable. A particularly common mistakes was to pass the entire index of a pipeline result, which might contain thousands of assets, even though the user only had intentions of trading a small percentage of those assets. Passing very large universes slowed down optimizations and caused problems where the optimizer would sometimes take a large number of tiny (e.g. 1-share) hedging positions to avoid violating users' constraints.
    order_optimal_portfolio now always uses the union of the three sets of assets listed above as the optimization universe.

  • The current_portfolio parameter to calculate_optimal_portfolio is now optional. If it is not provided, calculate_optimal_portfolio will default to the current portfolio in an algorithm, and the empty portfolio in a notebook.

  • Several constraints that previously took only a single asset, now accept multiple. Frozen, ReduceOnly, LongOnly, ShortOnly, and CannotHold will now either take an Asset or an iterable (for example, a list) of Assets.

  • The Basket constraint (formerly NeutralBasket) now takes in a single list of assets instead of two lists for longs and shorts respectively.

Name Changes

A number of modules and classes have been renamed. While the old names will continue to work, using them will bring up a deprecation warning.

  • The quantopian.experimental.optimize module has been renamed to quantopian.optimize. Algorithms that used to do import quantopian.experimental.optimize as opt should now do import quantopian.optimize as opt.
  • The TargetPortfolioWeights objective has been renamed to TargetWeights.
  • The NetLeverage constraint has been renamed to NetExposure.
  • Likewise, MaxGrossLeverage has been renamed to MaxGrossExposure.
  • The NetPartitionExposure constraint (often used to constrain sector and industry exposure) has been renamed to NetGroupExposure.
  • The WeightedExposure constraint has been renamed to FactorExposure.
  • The NeutralBasket constraint has been reworked substantially, as detailed in the 'Parameter Changes' section. It has also been renamed to Basket.
Clone Algorithm
58
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 59a9ceeee30e41510c2d9e15
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

9 responses

Here's a cloneable copy of the Optimize API tutorial notebook.

Loading notebook preview...
Notebook previews are currently unavailable.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

@Karl: I think it might have been a transient issue as our servers rolled over. Are you still seeing the errors this morning? Existing algos using the deprecated names should continue to run fine, aside from the deprecation warnings.

@Karl just to make sure it's clear, the expected argument to TargetWeights is a dictionary mapping individual asset objects to their target weights. So, for examples, a usage might look something like:

# Allocate 25% long to AAPL/MSFT and 25% short to TSLA/IBM  
my_objective = TargetWeights({AAPL: 0.25, MSFT: 0.25, TSLA: -0.25, IBM: -0.25})  

There's also an alternative constructor, TargetWeights.equal_weight for the common case where you have a list of assets that you want to allocate to with equal weight:

# Allocate 25% long to all four assets.  
my_objective = TargetWeights.equal_weight([AAPL, MSFT, TLSA, IBM])  

In writing up this example, I noticed that equal_weight is missing from the docs for TargetWeights. I'll make sure that gets updated.

One thing we don't currently have a convenience method for is the common case of wanting an equal-weight long-short portfolio. That would probably be a nice addition to the API.

Well noted @Scott, with thanks.

It may be that I was unable to convey my query clearly.. In the examples above, the arguments {AAPL: 0.25, MSFT: 0.25} are constants. I am wondering if I could simply pass in:

{context.longs: long_weight, context.shorts: short_weight}

Thanks again,

Karl

@Karl your example wouldn't currently work as written, but you could write a small utility function do something like it easily enough:

def equal_weight_long_short(longs, long_weight, shorts, short_weight):  
    weights = {asset: long_weight for asset in longs}  
    weights.update({asset: short_weight for asset in shorts})  
    return opt.TargetWeights(weights)  

Hi Scott,

Are you still planning to put the optimize API on a public github repository? Will it be part of zipline?

Thanks,

Grant

Optimize accounts for open market orders when determining target positions, unlike the order_target family of methods which only account for filled orders.

How is this implemented in the Optimize API?

What happens to equities that are in current portfolio but absent in the TargetWeights specification.

Say: My current portfolio positions are {GOOG: 0.5, X: -0.5}

If I specify my targetWeigths as follows:
Case A)
my_objective = TargetWeights({AAPL: 0.25, MSFT: 0.25, TSLA: -0.25, IBM: -0.25})

Will GOOG, X be automatically closed since they are part of the universe(based on Abhijeet's definition). Or do I need to add GOOG, X to the TargetWeights explicitly to close them as in Case B below.

Case B)
my_objective = TargetWeights({AAPL: 0.25, MSFT: 0.25, TSLA: -0.25, IBM: -0.25, GOOG: 0, X: 0})

Will Case A and Case B result in the same resultant portfolio (i.e addition of GOOG:0, X:0 has no effect and is implicitly assumed)?

Any feedback on my question above regarding the details of how the Optimize API "accounts for open market orders when determining target positions"? I'm interested in all use cases, including backtesting, Q paper trading, and real money (& paper?) prime broker trading.