Back to Community
Optimize API Generally Available

Today, we're announcing the stable release of the Optimize API. A number of classes and API methods have changed since its first experimental release, and this post details the changes.

Optimize is the recommended method for placing orders on Quantopian. It uses advanced optimization techniques to move your portfolio from its current state to your desired state, managing constraints and objectives that you can pass in. There are several reasons why it's better to use Optimize than order_target_percent or other manual ordering methods:

  • Optimize allows you to easily specify constraints when ordering, to manage your algorithm's leverage, position concentration, and exposures to risk factors.
  • Optimize accounts for open market orders when determining target positions, unlike the order_target family of methods which only account for filled orders.
  • Because Optimize is a declarative API, there's less manual order management to take care of, and the focus can instead be on alpha discovery and portfolio construction. You simply specify what you'd like your desired portfolio to be, either in the form of portfolio weights or alpha vectors, and Optimize handles placing the orders.
  • With the MaximizeAlpha objective, you now have the option of specifying alpha vectors as your ordering objectives, unlike the previous ordering methods which all needed a quantity of a specifically named stock.

Algorithms need to use the Optimize API in order to receive an allocation. We require this because having an objective and constraints allows us to more effectively respect an algorithm's desires if we're unable to hold the algorithm's exact optimal portfolio (e.g., because an asset is halted intraday or restricted by our prime broker). This post contains a lot more detail.

If you're new to Optimize and want to get started, the best place is the help docs.

If you'd like to see fully worked examples, go here to see Optimize in a notebook, or clone the attached backtest to see Optimize in an algorithm.

If you've already been using Optimize during the experimental release, here's what's new:

New Features

  • The error messages resulting from InfeasibleConstraints errors are now much more detailed. You'll see details about the constraints violated by the current portfolio, the empty portfolio - and when using the TargetWeights objective, the target portfolio as well.
  • A new method called run_optimization has been added. run_optimization is similar to calculate_optimal_portfolio, but returns an OptimizationResult - an object containing diagnostic information about the attempted portfolio optimization. run_optimization is primarily useful as a debugging tool.
  • Docs! Full documentation is now available for Optimize.
  • Objectives can now be rendered as LaTeX in research notebooks via their .math attribute. For example:

Parameter Changes

  • The universe parameter to order_optimal_portfolio is deprecated. Calls to order_optimal_portfolio will ignore the universe parameter, if it's passed in. In the original release of the API, order_optimal_portfolio required the user to pass provide a list of assets to act as the "optimization universe". The universe was used by the optimizer to define the assets that could be considered as potential candidates for orders and/or positions. After working with the Optimize API, however, we found that in practice the only reasonable choice for universe was the union of three sets of assets:

    • Assets referenced by the algorithm's objective.
    • Assets held in the algorithm's portfolio.
    • Assets with unfilled open orders.

    We also found that Optimize API users often passed a much larger list of assets than was desirable. A particularly common mistakes was to pass the entire index of a pipeline result, which might contain thousands of assets, even though the user only had intentions of trading a small percentage of those assets. Passing very large universes slowed down optimizations and caused problems where the optimizer would sometimes take a large number of tiny (e.g. 1-share) hedging positions to avoid violating users' constraints.
    order_optimal_portfolio now always uses the union of the three sets of assets listed above as the optimization universe.

  • The current_portfolio parameter to calculate_optimal_portfolio is now optional. If it is not provided, calculate_optimal_portfolio will default to the current portfolio in an algorithm, and the empty portfolio in a notebook.

  • Several constraints that previously took only a single asset, now accept multiple. Frozen, ReduceOnly, LongOnly, ShortOnly, and CannotHold will now either take an Asset or an iterable (for example, a list) of Assets.

  • The Basket constraint (formerly NeutralBasket) now takes in a single list of assets instead of two lists for longs and shorts respectively.

Name Changes

A number of modules and classes have been renamed. While the old names will continue to work, using them will bring up a deprecation warning.

  • The quantopian.experimental.optimize module has been renamed to quantopian.optimize. Algorithms that used to do import quantopian.experimental.optimize as opt should now do import quantopian.optimize as opt.
  • The TargetPortfolioWeights objective has been renamed to TargetWeights.
  • The NetLeverage constraint has been renamed to NetExposure.
  • Likewise, MaxGrossLeverage has been renamed to MaxGrossExposure.
  • The NetPartitionExposure constraint (often used to constrain sector and industry exposure) has been renamed to NetGroupExposure.
  • The WeightedExposure constraint has been renamed to FactorExposure.
  • The NeutralBasket constraint has been reworked substantially, as detailed in the 'Parameter Changes' section. It has also been renamed to Basket.
Clone Algorithm
116
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 59a9ceeee30e41510c2d9e15
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

19 responses

Here's a cloneable copy of the Optimize API tutorial notebook.

Loading notebook preview...
Notebook previews are currently unavailable.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

@Karl: I think it might have been a transient issue as our servers rolled over. Are you still seeing the errors this morning? Existing algos using the deprecated names should continue to run fine, aside from the deprecation warnings.

@Karl just to make sure it's clear, the expected argument to TargetWeights is a dictionary mapping individual asset objects to their target weights. So, for examples, a usage might look something like:

# Allocate 25% long to AAPL/MSFT and 25% short to TSLA/IBM  
my_objective = TargetWeights({AAPL: 0.25, MSFT: 0.25, TSLA: -0.25, IBM: -0.25})  

There's also an alternative constructor, TargetWeights.equal_weight for the common case where you have a list of assets that you want to allocate to with equal weight:

# Allocate 25% long to all four assets.  
my_objective = TargetWeights.equal_weight([AAPL, MSFT, TLSA, IBM])  

In writing up this example, I noticed that equal_weight is missing from the docs for TargetWeights. I'll make sure that gets updated.

One thing we don't currently have a convenience method for is the common case of wanting an equal-weight long-short portfolio. That would probably be a nice addition to the API.

@Karl your example wouldn't currently work as written, but you could write a small utility function do something like it easily enough:

def equal_weight_long_short(longs, long_weight, shorts, short_weight):  
    weights = {asset: long_weight for asset in longs}  
    weights.update({asset: short_weight for asset in shorts})  
    return opt.TargetWeights(weights)  

Hi Scott,

Are you still planning to put the optimize API on a public github repository? Will it be part of zipline?

Thanks,

Grant

Optimize accounts for open market orders when determining target positions, unlike the order_target family of methods which only account for filled orders.

How is this implemented in the Optimize API?

What happens to equities that are in current portfolio but absent in the TargetWeights specification.

Say: My current portfolio positions are {GOOG: 0.5, X: -0.5}

If I specify my targetWeigths as follows:
Case A)
my_objective = TargetWeights({AAPL: 0.25, MSFT: 0.25, TSLA: -0.25, IBM: -0.25})

Will GOOG, X be automatically closed since they are part of the universe(based on Abhijeet's definition). Or do I need to add GOOG, X to the TargetWeights explicitly to close them as in Case B below.

Case B)
my_objective = TargetWeights({AAPL: 0.25, MSFT: 0.25, TSLA: -0.25, IBM: -0.25, GOOG: 0, X: 0})

Will Case A and Case B result in the same resultant portfolio (i.e addition of GOOG:0, X:0 has no effect and is implicitly assumed)?

Any feedback on my question above regarding the details of how the Optimize API "accounts for open market orders when determining target positions"? I'm interested in all use cases, including backtesting, Q paper trading, and real money (& paper?) prime broker trading.

@Abhijeet Regarding the name changes "quantopian.experimental.optimize as opt should now do import quantopian.optimize as opt.
The TargetPortfolioWeights objective has been renamed to TargetWeights..."

Not sure what is the purpose of throwing deprecation warnings and make the community make a new revision of their existing algorithms when the corresponding changes are purely cosmetic. There should be an easier workaround on the server side by redirecting the old namespace and functions to the new nomenclature when we expect identical results.

@Leo - the old names continue to work (for example, you could still import from quantopian.experimental.optimize instead of quantopian.optimize at the moment, and they will both do the same thing). However, we recommend using the new namespaces, since Optimize is no longer in its experimental release. The main purpose of the deprecation warnings is to help authors upgrade to the new names, and also to indicate that the old names may not be available in the future.

@Abhijeet, I will definitely upgrade to the new namespace as I wont want my algos to stop working in the future.

My concern is that we lose some months of out of sample performance if the old namespace are done away with at some point in future. To clarify, if I wrote an algorithm 3 months ago with order_optimal_portfolio and now I no longer have the guarantee that that algo will run in the future, then I essentially lose 3 additional months of out of sample and get reset to T=0 (with my transformed algorithm with the new namespace) to start accumulating out of sample performance.

Seems a waste of a few months if releases are not backward compatible and we sunset the old namespace at some point.

It would be nice if namespace transitions are handled seemlessly by Quantopian, automatically transitioning to the new namespace and continuing to function as before. I have seen that in some old threads with a new tab suggesting "Migrated to new ...". is that not possible to setup for order optimal portfolio?

Leo, the good news is that I don't think you're really going to lose any out-of-sample performance. If you update only that code, and the performance of the updated algo matches the previous version, then the effective out-of-sample date is when you finished the first version. If you're getting good performance 6 months after the first finish date - let's talk!

The other thing to keep in mind here is that you are using code that we had categorized as "experimental." Experimental APIs are exactly that, and don't come with the same level of commitment as our regular API. Now that optimize is generally available, you can expect a higher level of stability from the API.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

@Dan, I didn't realize revisions start date can be that of original algorithm if performance of the algorithm doesn't change. Since you maintain a copy of the code when each backtest is run, I assumed that the backtest essentially crystallized the start date (and code). Thanks for clarifying. I will go ahead and make the code change to the existing algorithm, instead of creating a new algorithm which I was initially planning to do (and it appears that mistake would have made me lose the out of sample period).

You are correct that the backtest date is the date that crystallizes the start date of the out-of-sample period. But let's imagine how this could unfold:

You ran a backtest with the experimental API in June. In December, six months having elapsed, it will get automatically evaluated. If it meets our criteria, we'll contact you. You'll mention that it used the experimental API, but you have an updated version in September that is otherwise identical. At that point we can all discuss the merits of using the updated code, and whether we should use it or wait another 3 months for it to season. If the performance of the two algorithms is identical we would take the new code and use it instead.

When you think about it, we're always making judgement calls like that. Every time we update anything, Quantopian APIs, pandas, numpy, sklearn, etc., we're potentially disrupting the effectiveness of the out-of-sample period. There are some parts of this process that require good judgement.

@Dan, my intent was to state that it would be nice if the experimental api is retained till at least end of Mar 2018, so that atleast one of the algorithms either the existing experimental version or the version transformed to the new namespace (and a backtest kicked off now) is always in the running for an allocation at any point in the future (after they both have reached their 6 month out of sample periods respectively). A few months (of delayed algorithm consideration) is not a big deal but giving the existing experimental versions some grace period before the newer namespace versions can be 6 months out of sample will mean no lost out of sample time for the algorithm developer with regard to consideration by your automated discovery process. I am relatively new to Quantopian so not sure how much time there is between deprecation warnings and actual sunset (i.e apis stop working), so it is possible I am more concerned here than the situation warrants.

Oh, yeah, that will be fine. We don't have any plan to remove the old namespace, and it will be around for many months.

@Dan / @Quantopian,

Is the Optimize API available for use in Zipline / or available on Github? Thanks,

How do I change the data fed to the optimise API so that instead of using a z score (which I fully don't understand), it uses something tangible like pe_ratio as the input to the algorithm?
Also, the optimise API seems to be erratic in performance -meaning it does seem to give good results, but can also crash big time and correlation with market swings is pretty high despite opting for a market neutral optimiseer (1 of the options to the algorithm).

thanks
-kamal

Hi, just another friendly enquiry about whether the Optimize API will be open sourced. I understood from this comment on the CVXPY project's github that you wanted to do this. Now that the license issue has been resolved for some time, is open sourcing this library still on the roadmap?
Regards, Mark