Back to Community
Optimize API Required For Allocations

As mentioned here, we released the stable version of the Optimize API today. One thing mentioned in the post is that the Optimize API is required for an allocation - this post explains how that works.

Why this change?

The Optimize API allows Quantopian to intelligently adjust an algorithm's orders when external circumstances prevent us from achieving exactly the algorithm's desired portfolio. For example:

  • There may not be enough short inventory for a stock
  • The prime broker may restrict Quantopian from trading in certain stocks on a given day

Situations like these mean that an algorithm can't always get exactly what it wants. Optimize handles these kinds of situations by getting you to an achievable portfolio that's as close as possible to the originally desired portfolio, but also respects both the external constraints placed upon the algorithm and the constraints provided by the author.

Because these kind of situations can, and do, happen often in trading, we require algorithms to use Optimize in order to get an allocation, and every algorithm that has received an allocation to date uses Optimize.

How do I switch an existing algo over to Optimize?

If you'd like to easily port over an existing order_target_percent algorithm to use Optimize, your best bet is the TargetWeights objective. For example, if you wanted to switch over:

order_target_percent(sid(24), 0.5)  
order_target_percent(sid(5861), 0.5)  

to Optimize, you'd do:

order_optimal_portfolio(  
  objective=TargetWeights({sid(24): 0.5, sid(5861): 0.5}),  
  constraints=[],  
)

This is a fairly basic example, but there's a lot more you can do with Optimize - head over to the docs to see what you can do with the API.

What happens to my algos that don't use Optimize?

Algorithms that don't use Optimize will still be considered for an allocation. As a part of the diligence process, we'll work with you to update your algorithm to use Optimize.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

13 responses

Hi Abhijeet -

I'm not entirely clear what the Optimize API is doing. As you say, it "respects...the external constraints placed upon the algorithm" but what does this mean, precisely? For example, say this code is executed:

order_optimal_portfolio(  
  objective=TargetWeights({sid(24): 0.5, sid(5861): 0.5}),  
  constraints=[],  
)

Would you send the new weights to the broker, get some data back, complete the optimization, and then re-submit a new set of weights? Or does the broker get the new weights, apply another layer of optimization, and then send the orders on their merry way into the market? Or maybe since you know the universe of stocks the algo is using, the status of each stock is known prior to running the Optimize API (i.e. the algo is getting a real-time minutely (or more frequent?) feed of data from the broker, upon market open, for its universe)?

Also, what happens if order_optimal_portfolio is called multiple times within one trading minute (for both backtesting, paper trading, and real-money trading)? On the other post, you say "Optimize accounts for open market orders when determining target positions, unlike the order_target family of methods which only account for filled orders" which implies that if my algo calls order_optimal_portfolio more than once in a trading minute, for each subsequent call, any orders generated by prior calls will be considered completely unfilled (i.e. effectively not yet submitted)? Or are open orders only accounted for properly if order_optimal_portfolio is called at most once per trading minute?

Hi Grant,

This all happens before the order goes out to the broker. In a multi-strategy portfolio you have to consider the other algorithms as well as per-algo limits. In your example, your 50% AAPL portfolio might breach some risk constraints (e.g. maximum position concentration), or if you were to want to short the stock, it might not be available to short. The optimize call knows these constraints behind the scenes and will provide a portfolio that is actually feasible.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Thomas -

So what happens if opt.calculate_optimal_portfolio is used during backtesting? Will it incorporate the additional constraints? And if so, where does it get the data (e.g. availability of shorts)? Why not expose the data?

Still a lot of details missing here on the mechanics of the mysterious external constraints. No big deal, I suppose, but one can waste a lot of time by being confused (and contacting Q support) when complete documentation will clear things up at the outset.

@Grant: No additional constraints are applied during backtesting, only the ones you provide. This only applies for a fund-algorithm. We try to make as much of that data available as possible. Usually we do a first cut to where we can use it internally and then productize it properly to make it available to the larger Q community, which takes some time.

The global constraints are not really anything a user needs to be concerned with, especially not during algorithm development, as they don't apply there. The optimization tries to get as close to the target portfolio provided by the algorithm as possible.

Thanks Thomas -

Hmm? Wouldn't that result in backtests that could be significantly different from real trading? Or once you identify candidate algos for the fund, you would apply the external constraints and run a backtest?

Also, what about my question regarding multiple calls to order_optimal_portfolio within a trading minute? Does the algo now track pending orders within a trading minute and account for them?

This all happens before the order goes out to the broker.

Just curious how this works. If you are getting a feed of short availability, for example, and it is minutely, then might it be kinda stale? In theory, the order could be transmitted to the broker up to ~50 seconds later.

1) Yes, applying global portfolio level constraints to an algorithm's portfolio can/will cause that algorithm's performance to deviate from a run without global portfolio constraints. Portfolio construction is an interesting technical problem.

2) If you make multiple calls to order_optimal_portfolio with the same parameters, it is a no-op. If you change the parameters, it will assume the previous orders aren't filled. I can't think of a good reason to do that in the current environment, but we want to support it potentially for the future.

3) Short availability is another interesting technical problem. Nobody can ever guarantee they will fill orders. Orders sometimes fail to be filled. We can work to make it less frequent and be more resilient when it does happen.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Dan,

If I'm following correctly, when I run a backtest (and paper trading), only the constraints defined in my code are applied to the Optimize API (the "constraints provided by the author"). There could also be "external constraints placed upon the algorithm" which would not apply for ordinary algos. However, is it correct that external constraints would be imposed on a fund algo? And as a user, I would then see the effect on backtests? Or perhaps the external constraints would be applied, but only Quantopian could see the effect on the backtest? Or would external constraints only be applied at the point when the algo trades real money with the prime broker (i.e. only out-of-sample real money trading)?

Also, I'm still not clear on "Optimize accounts for open market orders when determining target positions, unlike the order_target family of methods which only account for filled orders" and when orders are considered pending versus submitted in backtesting and real-money trading. Under backtesting, presumably all orders are considered unfilled until the next whole trading minute, and the same would apply with your prime broker, assuming that you'll still be getting minutely feedback. Presumably, any open orders (partial fills left over from the prior trading minute and orders already submitted within the trading minute) will be taken into account by the Optimize API, and that order status is only available minutely, for both backtesting/paper trading and real money trading with your prime broker.

By the way, why not just use the name of your prime broker. Eventually, I'd think that as part of engaging institutional investors, you'd have to reveal it, per regulatory requirements, no?

Q - Thanks! This has saved me hundreds of lines of code.

Attached is a most simple algorithm as a starting point.

Clone Algorithm
13
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 59b29888fe371b5532c0bb1c
There was a runtime error.

It would be refreshingly transparent and interesting if Q could "open source" how the "global optimization" problem is being approached, both in selecting algos for the fund that will be accretive and applying the "external constraints" when executing the fund trading. I suppose it is understandable that the details would be kept confidential, but it is another opportunity to teach quantitative finance to the masses.

Having updated a bunch of code to use order_optimal_portfolio() I have found that it does not appear to exit positions before entering new ones.

Say I have 1 stock (MU) that has weight 1.0 with MaxGrossExposure(1.0). It fills.
Some ticks later I want to exit MU and replace it with INTC with weight = 1.0

With MaxGrossExposure=1.0 I would expect that MU would be exited and then INTC would be filled.

Unfortunately that doesn't seem to be happening. See for example April 27th in the attached backtest.

If I really have to manually sequence my calls to order_optimal_portfolio() to manage leverage then most of the code I just got rid of will have to come back.

Other than manually checking, is there a way to ensure the sequence of orders?

Clone Algorithm
6
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 59b5a83c11c46b50eb7ebd5d
There was a runtime error.

I used the new Optimize API in two of my contest entries, however the 2 year contest backtest results are significantly different from the research backtest results (7% vs. 58%). The set up is to place all orders at 9:32am and close all at 3:56pm. In research mode, this implementation is done correctly. I understand that this could change in contest/live trading environment depending on many factors such as whether orders are filled accordingly as intended by the code, short order inventory, etc. As mentioned... "Optimize handles these kinds of situations by getting you to an achievable portfolio that's as close as possible to the originally desired portfolio, but also respects both the external constraints placed upon the algorithm and the constraints provided by the author." In my example, close as possible, is not achieved by optimize api. Short order inventory is probably why my short orders at filled partially at one minute intervals, this I understand. However, buy market orders are also partially filled at minute intervals, this baffles me because these buy orders are not large and these are very liquid stocks. What would cause these buy market orders to not be filled instantaneously as there are in the research environment? These are what's causing the huge discrepancies in my results and very misleading in my choices of algorithms to field in the contest. I need more guidance on how to avoid these pitfalls.

@ Dan Dunn,

You said and I quote above:
1) Yes, applying global portfolio level constraints to an algorithm's portfolio can/will cause that algorithm's performance to deviate from a run without global portfolio constraints. Portfolio construction is an interesting technical problem.
3) Short availability is another interesting technical problem. Nobody can ever guarantee they will fill orders. Orders sometimes fail to be filled. We can work to make it less frequent and be more resilient when it does happen.

My personal experience as stated in prior reply regarding huge discrepancies in contest and research backresults makes me want to to scrutinize the details of how optimize api works. For example, does Q know apriori short inventory of stocks from broker and if so, is this info incorporated in the optimize algo. If it doesn't, then the stock selection process is hit or miss even if you pipe in Dollar Volume to get highly liquid stocks. Also, if you factor in other constraints like leverage/exposure, market neutral, sector neutral...is partial filling of shorts due to lack of short inventory adjusting the overall optimization process, i.e, which takes precendence?