Improved Weight Handling in the Optimize API

Today, we are releasing a small change to the Optimize API that improves the reliability of exiting positions when using the TargetWeights objective. When you use TargetWeights to exit a position (either by not passing an asset you currently hold or by passing the asset and giving it a target weight of 0), it is now much more likely that your resulting portfolio will not include the undesired position.

What problem is this fixing?

Previously, when you used TargetWeights to place orders with the Optimize API, it was possible for your portfolio to include small positions in assets that you did not intend to hold. Specifically, if you held a position in an asset at the start of an optimization and you attempted to exit that position — either by not passing the asset to TargetWeights or by passing the asset with a target weight of 0 — it was possible for your resulting portfolio to still include a small holding in the undesired asset. To understand why this happened, we need to review some basic Optimize API concepts.

When you construct a portfolio using the Optimize API, you provide two pieces of information: an objective, and a list of constraints. Your objective tells the Optimize API what you want from your new portfolio and your constraints give a set of conditions that your new portfolio must satisfy. When you ask the Optimize API to find a portfolio, it finds the portfolio that minimizes (or maximizes) the function defined by your objective, but the optimizer ensures that the resulting portfolio satisfies all the conditions defined by your constraints.

In the case of the TargetWeights objective, the Optimize API tries to find a portfolio that’s as close as possible to your target, ensuring that the result satisfies your constraints. This is easy when your target doesn’t conflict with any constraints: the optimal portfolio is simply your target. Things get more interesting if your target does violate one or more constraints. When that happens, the optimizer has to figure out how to “adjust” your target portfolio as little as possible while ensuring that all your constraints are satisfied.

When the optimizer searches for target positions to adjust, it considers assets referenced by your objective as well as assets in which your algorithm currently holds a position. This means that if you try to exit a position using TargetWeights, the optimizer may determine that the portfolio that minimizes the TargetWeights objective function is a portfolio that still contains a position you were trying to exit. In particular, because of the distance metric used by TargetWeights, a common outcome for certain constraint violations was that the optimizer would choose a portfolio with many small (e.g., 1-5 shares) holdings for positions that were supposed to be exited.

What is the new behavior?

In this release, we’ve tweaked the optimization function used by TargetWeights to penalize holding positions that your algorithm is attempting to exit. This affects positions for which you passed a target weight of 0, as well as currently-held positions for which you didn’t provide a target at all. The result of this change is that TargetWeights will prioritize zeroing positions that your algorithm is trying to exit when it considers which target positions to adjust.

This change should help amend issues like the ones seen in the following forum posts:

In both of these cases, the optimizer previously had difficulty maintaining a portfolio within the QTU. With this change in effect, non-QTU positions will be more effectively zeroed-out because the optimizer now has a strong preference against holding these positions where a weight is undesired. This will allow more of these strategies’ holdings to be in the QTU.

In short, this means that the optimizer behaves more intuitively when using TargetWeights — positions for which no weight is desired are much more likely to be dropped from the portfolio.

Note: This change improves the behavior of TargetWeights when adjusting portfolios after constraint violations, but TargetWeights still generally works best when your target portfolio satisfies or nearly satisfies your constraints. It’s usually not a good idea to use TargetWeights with a target portfolio that you expect the optimizer to significantly transform. For example, if you apply a DollarNeutral constraint in your optimization, you should ensure that your target portfolio contains an equal mix of long and short positions.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

6 responses

Hi Gus,
My contest algorithm had been disqualified due to the change. My algorithm was designed so that the leverage target could be met under the old Target_Weight behavior. While I sincerely believe that the change is necessary, I don't think it is fair to apply it on the contest algorithms.
I am especially disheartened when I ranked third in the contest Yesterday, but learned it was disqualified.

best wishes,
Golden Hen

I had the same problem. A really good algorithm suddenly became useless. Returns and leverage dropped too.

I don't mind my algorithm becoming useless due to say alpha decay, but in my case disqualification happened due to leverage breach of a FEW MONTHS AGO.
I was suddenly told, that my leverage constrains weren't met in several occasions for this past 4 months when the algorithm was running fine in the contest

best,

Sorry to hear this fix is causing your algorithms to not meet contest criteria anymore, I can see how that frustrating as you didn't do anything wrong. Unfortunately, after having thought about this before we shipped this, there is no other way. Bugs happen and the fixes sometimes affect contest algorithms. To keep the contest as fair as possible, we will always want to make sure backtests are as accurate as they can be. So I would encourage you to fix the underlying issue that was uncovered by this fix which will hopefully make your algorithm even stronger, and then resubmit it to the contest.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi, Gus and Thomas, great news! However, I'm still having trouble passing the Tradable Universe criteria despite going through great lengths to make sure I'm not including any non-QTSUS stocks in my target weights.

As you can see in my algo's log, it is trying to drop expired assets, but not able to:

2016-07-14 14:00 WARN Dropping expired assets from optimization universe:
['Equity(27704 [TAL])']
2016-07-14 14:00 WARN Dropping expired assets from optimization universe:
['Equity(27704 [TAL])']
2016-07-14 19:59 record_vars:229 WARN TAL does not belong in portfolio.
2016-07-15 14:00 WARN Dropping expired assets from optimization universe:
['Equity(27704 [TAL])']
2016-07-15 14:00 WARN Dropping expired assets from optimization universe:
['Equity(27704 [TAL])']
2016-07-15 19:59 record_vars:229 WARN TAL does not belong in portfolio.
2016-07-18 14:00 WARN Dropping expired assets from optimization universe:
['Equity(27704 [TAL])']
2016-07-18 14:00 WARN Dropping expired assets from optimization universe:
['Equity(27704 [TAL])']
2016-07-18 19:59 record_vars:229 WARN TAL does not belong in portfolio.
2016-09-06 14:00 WARN Dropping expired assets from optimization universe:
['Equity(4117 [JCI])']
2016-09-06 14:00 WARN Dropping expired assets from optimization universe:
['Equity(4117 [JCI])']
2016-09-06 19:59 record_vars:229 WARN JCI does not belong in portfolio.
2016-09-07 14:00 WARN Dropping expired assets from optimization universe:
['Equity(4117 [JCI])']
2016-09-07 14:00 WARN Dropping expired assets from optimization universe:
['Equity(4117 [JCI])']
2016-09-07 19:59 record_vars:229 WARN JCI does not belong in portfolio.
2016-09-08 14:00 WARN Dropping expired assets from optimization universe:
['Equity(4117 [JCI])']
2017-01-12 15:00 WARN Dropping expired assets from optimization universe:
['Equity(38760 [CLNY])']
2017-01-12 15:00 WARN Dropping expired assets from optimization universe:
['Equity(38760 [CLNY])']
2017-01-12 20:59 record_vars:229 WARN CLNY does not belong in portfolio.
2017-01-13 15:00 WARN Dropping expired assets from optimization universe:
['Equity(38760 [CLNY])']
2017-01-13 15:00 WARN Dropping expired assets from optimization universe:
['Equity(38760 [CLNY])']
2017-01-13 20:59 record_vars:229 WARN CLNY does not belong in portfolio.
2017-01-17 15:00 WARN Dropping expired assets from optimization universe:
['Equity(38760 [CLNY])']
2017-01-17 15:00 WARN Dropping expired assets from optimization universe:
['Equity(38760 [CLNY])']
2017-01-17 20:59 record_vars:229 WARN CLNY does not belong in portfolio.
2017-09-08 14:00 WARN Dropping expired assets from optimization universe:
['Equity(12067 [KNX])']
2017-09-08 14:00 WARN Dropping expired assets from optimization universe:
['Equity(12067 [KNX])']
2017-09-08 19:59 record_vars:229 WARN KNX does not belong in portfolio.
2017-09-11 14:00 WARN Dropping expired assets from optimization universe:
['Equity(12067 [KNX])']
2017-09-11 14:00 WARN Dropping expired assets from optimization universe:
['Equity(12067 [KNX])']
2017-09-11 19:59 record_vars:229 WARN KNX does not belong in portfolio.
2017-09-12 14:00 WARN Dropping expired assets from optimization universe:
['Equity(12067 [KNX])']
2017-09-12 14:00 WARN Dropping expired assets from optimization universe:
['Equity(12067 [KNX])']
2018-07-13 14:00 WARN Dropping expired assets from optimization universe:
['Equity(38708 [SPB])']
2018-07-13 14:00 WARN Dropping expired assets from optimization universe:
['Equity(38708 [SPB])']
2018-07-13 19:59 record_vars:229 WARN SPB does not belong in portfolio.
2018-07-16 14:00 WARN Dropping expired assets from optimization universe:
['Equity(38708 [SPB])']
2018-07-16 14:00 WARN Dropping expired assets from optimization universe:
['Equity(38708 [SPB])']
2018-07-16 19:59 record_vars:229 WARN SPB does not belong in portfolio.
2018-07-17 14:00 WARN Dropping expired assets from optimization universe:
['Equity(38708 [SPB])']
2018-07-17 14:00 WARN Dropping expired assets from optimization universe:
['Equity(38708 [SPB])']
2018-07-17 19:59 record_vars:229 WARN SPB does not belong in portfolio.
2018-08-24 14:00 WARN Dropping expired assets from optimization universe:
['Equity(33609 [SMCI])']
2018-08-24 14:00 WARN Dropping expired assets from optimization universe:
['Equity(33609 [SMCI])']
2018-08-24 19:59 record_vars:229 WARN SMCI does not belong in portfolio.
2018-08-27 14:00 WARN Dropping expired assets from optimization universe:
['Equity(33609 [SMCI])']
2018-08-27 14:00 WARN Dropping expired assets from optimization universe:
['Equity(33609 [SMCI])']
2018-08-27 19:59 record_vars:229 WARN SMCI does not belong in portfolio.
2018-08-28 14:00 WARN Dropping expired assets from optimization universe:
['Equity(33609 [SMCI])']
2018-08-28 14:00 WARN Dropping expired assets from optimization universe:
['Equity(33609 [SMCI])']
2018-08-28 19:59 record_vars:229 WARN SMCI does not belong in portfolio.


I'm guessing this means the stock is "halted" or otherwise not tradable any more. My hunch is that certain corporate events (merges/acquisitions) are the culprit. As these events are more-or-less unforeseeable by an algorithm, it's a bit harsh to be prevented from participating in the contest.

Could the issue also be data errors? When I run a tearsheet I get a peculiar message:

/usr/local/lib/python2.7/dist-packages/pyfolio/round_trips.py:213: UserWarning: Negative price detected, ignoring forround-trip.
warnings.warn('Negative price detected, ignoring for'


Could that have something to do with split/dividend adjustments?

Why isn't the QTradableStocksUS criteria placed on orders (as opposed to overnight positions)? It's a bit of a headache to control for what happens to a stock after we've ordered it. Or is there some other solution?

Nevermind, I've somehow managed to resolve the issue. 🤷