Singapore Conference / Questions Regarding Optimize API

Dear All,

A big thank you to the entire Quantopian team for a very well organized conference. I gained valuable insights every day, and I look forward to being able to read the presentation decks of the many speakers that I missed (with 4 presentation streams to choose from every hour on Friday, I had to make some very difficult choices).

While I had only half the time for the hackathon (I had to leave at 1:30pm to catch my flight back to Hong Kong), I really appreciated being "forced" to use some of the key zipline APIs in an environment where I had all these experts ready to answer my questions. I definitely feel that I significantly improved my understanding of pipeline custom factors and the optimize API thanks to the Hackathon.

As I knew I had a very short amount of time for the hackathon (I only had 2.5hours), I decided that it made a lot more sense to start from something that was already satisfying competition criteria and was already working. I therefore just cloned the long/short example from Delaney's lectures as a starting point. I just replaced the value factor with a custom factor for earnings growth. I then made minor adjustments to the momentum factor and I transformed the momentum factor ranking using a quadratic manipulation (to lower the ranking score of the top ranked scores - Alpha lens analysis showed me that 1yr momentum is a reliable and persistent factor, but it's not linear as the top quintile of momentum ranked stocks are all over the place depending on different time periods. Momentum leadership downwards persists a lot more than leadership upwards...).

While I'm sure that my very minor changes caused a weird bug, I was stunned that backtested returns were so volatile with more than 300 positions at all times, and a net exposure close to 0. How can a very diversified market neutral portfolio with a gross exposure close to 100% lose around 14% between December 28 and December 29 2016. Something is really weird! If only I could understand the magic done by the Optimize API, it would be a lot easier to debug this type of algo. I guess the first step would be to look at all the individual positions held when this big drawdown occured, but that seems quite time consuming with close to 400 positions...

Any advice on this would be greatly appreciated.

Best,

Mattias

13
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""This algorithm demonstrates the concept of long-short equity.
It uses two fundamental factors to rank equities in our universe.
It then longs the top of the ranking and shorts the bottom.
For information on long-short equity strategies, please see the corresponding
lecture on our lectures page:

https://www.quantopian.com/lectures

WARNING: These factors were selected because they worked in the past over the specific time period we choose.
We do not anticipate them working in the future. In practice finding your own factors is the hardest
part of developing any long-short equity strategy. This algorithm is meant to serve as a framework for testing your own ranking factors.

This algorithm was developed as part of
Quantopian's Lecture Series. Please direct any
questions, feedback, or corrections to [email protected]
"""

from quantopian.algorithm import attach_pipeline, pipeline_output, order_optimal_portfolio
from quantopian.pipeline import Pipeline
from quantopian.pipeline.factors import CustomFactor, SimpleMovingAverage, AverageDollarVolume, RollingLinearRegressionOfReturns, Returns
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.data import morningstar
from quantopian.pipeline.filters.morningstar import IsPrimaryShare
from quantopian.pipeline.classifiers.morningstar import Sector
import talib
import numpy as np
import pandas as pd

from quantopian.pipeline.filters import Q1500US
import quantopian.optimize as opt

# Constraint Parameters
MAX_GROSS_EXPOSURE = 1.0
NUM_LONG_POSITIONS = 300
NUM_SHORT_POSITIONS = 300

# Here we define the maximum position size that can be held for any
# given stock. If you have a different idea of what these maximum
# sizes should be, feel free to change them. Keep in mind that the
# optimizer needs some leeway in order to operate. Namely, if your
# maximum is too small, the optimizer may be overly-constrained.
MAX_SHORT_POSITION_SIZE = 2*1.0/(NUM_LONG_POSITIONS + NUM_SHORT_POSITIONS)
MAX_LONG_POSITION_SIZE = 2*1.0/(NUM_LONG_POSITIONS + NUM_SHORT_POSITIONS)

# Risk Exposures
MAX_SECTOR_EXPOSURE = 0.30
MAX_BETA_EXPOSURE = 0.10

class Momentum(CustomFactor):
"""
Here we define a basic momentum factor using a CustomFactor. We take
the momentum from the past year up until the beginning of this month
and penalize it by the momentum over this month. We are tempering a
long-term trend with a short-term reversal in hopes that we get a
better measure of momentum.
"""
inputs = [USEquityPricing.close,
Returns(window_length=126)]
window_length = 252

def compute(self, today, assets, out, prices, returns):
out[:] = ((prices[-21] - prices[-252])/prices[-252])/ np.nanstd(returns, axis=0) #np.nanstd(talib.ROC(prices[~np.isnan(prices)],1))
print(prices)

class Growth(CustomFactor):
"""
Here we define a basic momentum factor using a CustomFactor. We take
the momentum from the past year up until the beginning of this month
and penalize it by the momentum over this month. We are tempering a
long-term trend with a short-term reversal in hopes that we get a
better measure of momentum.
"""
inputs = [morningstar.income_statement.ebit]
window_length = 252

def compute(self, today, assets, out, returns):
out[:] = returns[-1]/returns[-252]

def make_pipeline():
"""
Create and return our pipeline.

We break this piece of logic out into its own function to make it easier to
test and modify in isolation.

In particular, this function can be copy/pasted into research and run by itself.
"""

# Create our momentum, value, and quality factors
momentum = Momentum()
# By appending .latest to the imported morningstar data, we get builtin Factors
# so there's no need to define a CustomFactor
#value = morningstar.income_statement.ebit.latest / morningstar.valuation.enterprise_value.latest
growth = Growth()
quality = morningstar.operation_ratios.roe.latest

# Classify all securities by sector so that we can enforce sector neutrality later
sector = Sector()

# Screen out non-desirable securities by defining our universe.
# Removes ADRs, OTCs, non-primary shares, LP, etc.
# Also sets a minimum $500MM market cap filter and$5 price filter
mkt_cap_filter = morningstar.valuation.market_cap.latest >= 300000000
price_filter = USEquityPricing.close.latest >= 5
universe = Q1500US() & price_filter & mkt_cap_filter

# Construct a Factor representing the rank of each asset by our momentum,
# value, and quality metrics. We aggregate them together here using simple
#
# By applying a mask to the rank computations, we remove any stocks that failed
# to meet our initial criteria **before** computing ranks.  This means that the
# stock with rank 10.0 is the 10th-lowest stock that was included in the Q1500US.

combined_rank = (
rank_momentum * 3 +
)
#combined_rank = combined_rank.rank.zscore()

# Build Filters representing the top and bottom 150 stocks by our combined ranking system.
# We'll use these as our tradeable universe each day.
longs = combined_rank.top(NUM_LONG_POSITIONS)
shorts = combined_rank.bottom(NUM_SHORT_POSITIONS)

# The final output of our pipeline should only include
# the top/bottom 300 stocks by our criteria
long_short_screen = (longs | shorts)  #or operator

# Define any risk factors that we will want to neutralize
# We are chiefly interested in market beta as a risk factor so we define it using
# Bloomberg's beta calculation
beta = 0.66*RollingLinearRegressionOfReturns(
target=sid(8554),
returns_length=5,
regression_length=260,
).beta + 0.33*1.0

# Create pipeline
pipe = Pipeline(columns = {
'longs':longs,
'shorts':shorts,
'combined_rank':combined_rank,
'quality':quality,
'value':growth,
'momentum':momentum,
'sector':sector,
'market_beta':beta
},
screen = long_short_screen)
return pipe

def initialize(context):
# Here we set our slippage and commisions. Set slippage
# and commission to zero to evaulate the signal-generating
# ability of the algorithm independent of these additional
# costs.
set_slippage(slippage.VolumeShareSlippage(volume_limit=1, price_impact=0))
context.spy = sid(8554)

attach_pipeline(make_pipeline(), 'long_short_equity_template')

# Schedule my rebalance function
schedule_function(func=rebalance,
date_rule=date_rules.month_start(),
time_rule=time_rules.market_open(hours=0,minutes=30),
half_days=True)
# record my portfolio variables at the end of day
schedule_function(func=recording_statements,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_close(),
half_days=True)

# Call pipeline_output to get the output
# Note: this is a dataframe where the index is the SIDs for all
# securities to pass my screen and the columns are the factors
# added to the pipeline object above
context.pipeline_data = pipeline_output('long_short_equity_template')

def recording_statements(context, data):
# Plot the number of positions over time.
#record(num_positions=len(context.portfolio.positions))
record(gross=context.account.leverage)

# Called at the start of every month in order to rebalance
# the longs and shorts lists
def rebalance(context, data):
### Optimize API
pipeline_data = context.pipeline_data

### Extract from pipeline any specific risk factors you want
# to neutralize that you have already calculated
risk_factor_exposures = pd.DataFrame({
'market_beta':pipeline_data.market_beta.fillna(1.0)
})
# We fill in any missing factor values with a market beta of 1.0.
# We do this rather than simply dropping the values because we have
# want to err on the side of caution. We don't want to exclude
# a security just because it's missing a calculated market beta,
# so we assume any missing values have full exposure to the market.

### Here we define our objective for the Optimize API. We have
# selected MaximizeAlpha because we believe our combined factor
# ranking to be proportional to expected returns. This routine
# will optimize the expected return of our algorithm, going
# long on the highest expected return and short on the lowest.
objective = opt.MaximizeAlpha(pipeline_data.combined_rank)

### Define the list of constraints
constraints = []
# Constrain our maximum gross leverage
constraints.append(opt.MaxGrossExposure(MAX_GROSS_EXPOSURE))
# Require our algorithm to remain dollar neutral
constraints.append(opt.DollarNeutral())
# Add a sector neutrality constraint using the sector
# classifier that we included in pipeline
constraints.append(
opt.NetGroupExposure.with_equal_bounds(
labels=pipeline_data.sector,
min=-MAX_SECTOR_EXPOSURE,
max=MAX_SECTOR_EXPOSURE,
))
# Take the risk factors that you extracted above and
# list your desired max/min exposures to them -
# Here we selection +/- 0.01 to remain near 0.
neutralize_risk_factors = opt.FactorExposure(
min_exposures={'market_beta':-MAX_BETA_EXPOSURE},
max_exposures={'market_beta':MAX_BETA_EXPOSURE}
)
constraints.append(neutralize_risk_factors)

# With this constraint we enforce that no position can make up
# greater than MAX_SHORT_POSITION_SIZE on the short side and
# no greater than MAX_LONG_POSITION_SIZE on the long side. This
# ensures that we do not overly concentrate our portfolio in
# one security or a small subset of securities.
constraints.append(
opt.PositionConcentration.with_equal_bounds(
min=-MAX_SHORT_POSITION_SIZE,
max=MAX_LONG_POSITION_SIZE
))

# Put together all the pieces we defined above by passing
# them into the order_optimal_portfolio function. This handles
# all of our ordering logic, assigning appropriate weights
# to the securities in our universe to maximize our alpha with
# respect to the given constraints.
order_optimal_portfolio(
objective=objective,
constraints=constraints,
)

There was a runtime error.
29 responses

With a monthly snapshot I'm seeing HERO as an outlier in pnl as of 12-2015 at -40k (with next nearest CONN at -3.3k). It delisted 6-2016.
At the dip toward the end, PVAC shows up at -152k when CONN is -5.4k. PVAC is short while the price goes from 5 to 49.
Maybe verify data with any splits are ok on both of those and then experiment with modifying the factors and try other factors.
If factors for example had PVAC going long instead of short, that would have been great.
Stocks on both ends of the pnl win/loss spectrum tend to be short more than long.

As a temporary diagnostic test just to see the difference it would make you can tell optimize to ignore those two stocks:

constraints.append(opt.CannotHold([symbol('HERO'), symbol('PVAC')]))


Thanks for taking the time to take a look at this crazy backtest.

I definitely need to play around with the constraints. Also, probably having some kind of stop loss on the shorts would make sense as right now the model just picks stocks once per month and does nothing until a month later.

Shorts inherently will have bigger negative tail risks and lower positive tail outcomes than longs (unlimited loss, limited gain vs unlimited gain, limited loss for longs). Risk control methods to limit large losses on shorts usually pay off over time...

Right now, I have to say that I'm not yet that comfortable with the optimize() API. I'd prefer to first develop an equal weighted strategy that works with order_percent(), and only after I can see that it works try to get it working with optimize().

Hi Mattias,

You're running into a few interesting problems simultaneously.

As others have noted in the thread, your two big drawdowns are caused by short positions in penny stocks (HERO and PVAC), that experienced major spikes upward while your algo held a short position. That should be surprising, however, because your algorithm is trying to filter down to stocks with prices greater than 5 dollars during its universe construction, so your algorithm probably wasn't meant to be holding those positions in the first place.

Your algorithm ends up holding shorts in those penny stocks because of how the optimizer handles positions that are in your portfolio but not in the vector of alphas you passed to MaximizeAlpha. The optimizer needs to assume some alpha value for the assets that dropped out of your universe, because we need to calculate a new target portfolio, which might have to include those assets (we can't simply exit those positions because you might have, say, a MaxTurnover constraint). The current implementation assumes an alpha value of zero for assets that weren't provided to MaximizeAlpha. That assumption works out poorly for your algorithm, because all of your alpha values are positive (as a side note, generally speaking you want negative alpha values for your expected shorts and positive alphas for your expected longs), so the optimizer ends up using the assets that have dropped out of your portfolio as hedges against your other positions. Needless to say, this isn't the behavior anyone wants or expects. We're currently investigating ways of making MaximizeAlpha handle these cases better.

The other problem you're running into is that PVAC was actually delisted for about a year during the period where your algorithm held it. Normally the backtester would automatically liquidate the delisted position, but we don't currently model the case where an asset is listed, gets delisted, and then subsequently gets re-listed (the backtester just sees this as a long period of zero volume and no price changes). When PVAC re-lists, it does so at a totally different price, likely because the company re-capitalized and/or experienced other corporate actions that we don't have because they occurred while the company wasn't publicly traded.

I've attached a notebook with a more in-depth analysis of the issues I just outlined. The notebook also makes use of the pretty wide variety of tools available for investigating backtests in research, which hopefully makes for an interesting case study.

18
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Wow! Thank you Scott! You spent a lot of time on this!

I have never used the backtest tearsheets before, but I can immediately see their huge value in this scenario. I'm going to study your notebook carefully to use it as a template in the future. In addition to the largest long and short positions of the backtest period, it would be cool to have a table that could give me individual positions responsible for largest P&L moves (they are usually correlated but not exactly the same as largest positions). The rolling Fama single factors chart is unbelievably cool! Really useful to identify which factors are working and when they are working!

Now, I understand why things started going wrong when I removed the zscore() in order to make my quadratic manipulation. In the workshop last week, one of the people in my table commented to Delaney that Rank and Zscore were serving the same purpose in terms of normalizing the values and were not both really necessary in this case. In reality, the Zscore() call was used as a quick way to transform half the rankings around a 0 mean with half the population with negative values...

While I understand that optimize() needs to keep track of old universe values to effectively get rid of them after they are no longer part of the universe, I think it would make a lot of sense to have a constraint that can force optimize() to only have active positions in stocks that do not have an absolute 0 vector passed into MaximizeAlpha().

Thanks again for spending so much time to respond to my query.

Best,

Mattias

While I understand that optimize() needs to keep track of old universe values to effectively get rid of them after they are no longer part of the universe, I think it would make a lot of sense to have a constraint that can force optimize() to only have active positions in stocks that do not have an absolute 0 vector passed into MaximizeAlpha().

My current thinking on this is that MaximizeAlpha needs to have a separate (much more strongly-weighted) term in its optimization to account for the desire to exit positions in assets no longer in the the trading universe. Right now, the core of MaximizeAlpha builds a CVXPY objective by doing, roughly:

cvxpy_objective = cvx.Maximize(new_weights.T * alphas)


There's really no way in this formulation to express a desire to not hold an asset. Better would be something like:

expected_alpha = new_weights.T * alphas
# Penalize continuing to hold positions in assets that aren't in our alphas.
penalty_coefficients = [0 if asset in alphas.index else LARGE_NUMBER
for asset in current_weights.index]
non_universe_penalty = cvx.abs(new_weights.T) * np.array(penalty_coefficients)
cvxpy_objective = cvx.Maximize(expected_alpha - non_universe_penalty)


The idea here is to use the non_universe_penalty term to penalize the magnitude of any held position for which an alpha wasn't supplied. If LARGE_NUMBER is sufficiently large relative to alpha values (doing this correctly probably requires re-scaling of alphas, which is safe as long as you just multiply every alpha by the same constant), then in practice the penalty operates like a constraint unless incurring the penalty is necessary to satisfy a constraint.

Holy cow. Scott, that notebook is amazing.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

@Scott, that is really a great notebook. Thanks.

Hi Scott -

generally speaking you want negative alpha values for your expected shorts and positive alphas for your expected longs

I haven't digested all of this, but I'd assumed that the Optimize API could be fed normalized alphas that are not demeaned. But I think you are saying if the input is not demeaned, then stocks that should be reduced in weight will be highly weighted (since you are setting the alpha values of stocks not in the universe to zero)? Is this the correct interpretation?

Is a work-around simply to demean the input to the Optimize API?

Also, what does order_optimal_portfolio do? I'd expected that it would attempt to close out any positions of stocks not in the current universe, with a separate set of orders:

for stock in context.portfolio.positions.keys():
if stock not in universe:
order_target_percent(stock,0)


But apparently not. I guess I'm unclear how stocks get dropped from the universe without code similar to what I have above.

Wouldn't you simply run the optimize API on the current universe only, and then close out any positions not in the universe? Or is the paradigm that changes in the universe shouldn't dictate changes in positions?

If one wanted to assure stocks dropped from the universe are actually dropped, would it mean running a separate routine, prior to the final optimize API run, to close out positions of stocks no longer in the universe?

Regarding the listing-de-listing-listing issue, when a stock gets listed, shouldn't it be assigned a new SID? Wouldn't this solve the problem? If a stock gets de-listed, then when it is re-listed, I'd think a new SID would be appropriate. Even if it subsequently trades under the same symbol, as soon as it disappears, within Quantopian, one has no visibility of it. It is effectively a new issue when it gets re-listed, right?

Injecting stocks into the pipeline from within order_optimal_portfolio with zero alphas can have a lot of unintended and random consequences. Please fix this issue or alternately expose a flag to only rank pipeline and not add positions to it (which should be default behavior everybody expects).

Not sure why in the code we are adding existing positions to optimize routine. If we do optimize on just the pipeline stocks and add a routine at the end to close out any positions that are not in pipeline, it should achieve the intuitive and expected behavior of order_optimal_portfolio.

It all depends on what one wants to achieve with the Optimize API and order_optimal_portfolio. Looking at https://blog.quantopian.com/a-professional-quant-equity-workflow/, there is an element of risk management in this step, so even though the universe changes, it may be too risky to make a dramatic change all at once. Optimization might require holding onto stocks no longer in the universe (but then how does one eventually exit them definitively?). However, this could end up being belt-and-suspenders, since presumably Q is cobbling together the orders from various Q fund algos and applying another layer of constrained optimization, where they could apply such risk management, and also their prime broker could do it, too, depending on what functionality is included in the Execution step on https://blog.quantopian.com/a-professional-quant-equity-workflow/.

I'd say keep the backtester straightforward and intuitive, and push the business of how to close out positions no longer in the universe to the fund construction and execution steps.

What everybody wants and expects should be supported in a code path via the default behavior or through a flag.

We rather deal with the problem of ranking stocks in the pipeline according to our algorithm and getting a resultant portfolio based solely on what we ranked, rather than have to deal with the complexity of non-pipeline stocks lingering around in the portfolio for reasons we cannot possibly comprehend or deal with in our ranking algorithms.

Here is an example strategy that illustrates why it will be problematic to optimize by merging positions into pipeline using zero alpha.

Say your pipeline filter is for stocks > 5 equity price plus some other filters
Your alphas/pipeline weights are 1 day returns
You enter short 10 of the smallest returns and long 10 of the largest returns
If the stocks from prior day positions fall below 5 then they are filtered in the pipeline and not ranked
If optimizer sets zero alphas to them (<5 priced prior day stock positions) then on days when there is up market and all stocks in the rest of your universe are up we will be entering shorts on positions from prior day that are < 5 and zero alpha (meaning lowest weight). That will be contrary to the algorithm strategy and no way to work around it.

Hmm I always thought order_optimal_portfolio() would liquidate all positions not in the weights vector.

@ Scott -

Within CVXPY, wouldn't it simply be a matter of adding hard constraints? For example, x[0] == 0, where x[0] is the weight for a stock that should not be held? Wouldn't this force zeros where they should be, or result in an exception if a solution can't be found? It just seems like it would be a more definite approach, unless you are seeing the need to provide users with the ability to set the LARGE_NUMBER you describe above?

@Grant

I haven't digested all of this, but I'd assumed that the Optimize API could be fed normalized alphas that are not demeaned. But I think you are saying if the input is not demeaned, then stocks that should be reduced in weight will be highly weighted (since you are setting the alpha values of stocks not in the universe to zero)? Is this the correct interpretation?

The answer to this question depends on what constraints are being applied.

In general, the behavior of MaximizeAlpha is that, given a vector of alpha values, a[0], a[1], ..., a[N], we find a vector of new target portfolio weights, w[0], w[1], ..., w[N] that maximizes a[0] * w[0] + a[1] * w[1], ... a[N] * w[N] while respecting any additional constraints that have been supplied.

Some observations that fall out of that mathematical definition are:

1. Without any additional constraints, MaximizeAlpha can't produce a portfolio at all, because there's no upper bound on the function it's trying to maximize.
2. With just a MaxGrossLeverage constraint, MaximizeAlpha will allocate all of its capital to the asset with the largest alpha magnitude, going short if the sign of that alpha is negative, and going long if the sign of that alpha is positive.
3. With just MaxGrossLeverage and PositionConcentration, MaximizeAlpha will proceed through the assets in order of alpha magnitude, allocating the maximum-size long or short position to each, depending on the sign of the alpha value. One implication of this is that if you use ranks as your "alphas" you'll always end up with a long-only portfolio with these constraints.
4. With MaxGrossLeverage, PositionConcentration, and DollarNeutral constraints, the behavior of MaximizeAlpha ends up being roughly what you'd get from an algorithm like: "Choose the N largest and N smallest assets by alpha value and allocate the maximum-size long to the largest assets and the maximum-size short to the smallest assets." This is where we run afoul of MaximizeAlpha's current behavior of assuming an alpha value of zero for assets with no alpha provided. If all the provided alphas are greater than zero, then the "N smallest assets" are all the zeros that we inserted for the assets that weren't provided, which makes them the most attractive targets for short positions.

As you add in additional constraints, the behavior becomes more complex to describe succinctly. One intuition that I find helpful is to think of each constraint as defining one or more "budgets" that positions contribute to at different rates, and to think of the alpha vector you provide to MaximizeAlpha as defining the amount of "value" you get from holding a unit of each asset. The goal of the optimizer when using MaximizeAlpha is to produce the portfolio with the largest amount of value that still respects all the budgets impose by the constraints.

One thing that's important to stress is that the sign and magnitude of alphas matter, because they determine how attractive MaximizeAlphaconsiders each asset as it's deciding how best to make use of its budgets. In particular, assets with positive alpha values are attractive to hold, assets with negative alpha values are attractive to hold short.

The best way I've found for building intuition about how the optimizer responds to different inputs is to set up simple test cases in a notebook (like the ones in the middle section of the notebook I posted above) and see how the result of the optimization changes as I change inputs.

A few people in the thread have asked why the optimizer doesn't just close out positions in any assets not provided in the alpha vector. In general, we can't do that because there may be constraints (e.g. a user-provided MaxTurnover constraint) that prevent us from completely closing out an existing position. The penalty term version I've outlined above has the same behavior as a hard constraint in all the cases where it's possible to do so, but it also makes a best effort in cases where completely closing out the target positions would conflict with another hard constraint.

Thanks Scott -

One thing that's important to stress is that the sign and magnitude of alphas matter

It seems like you'd want a standard preprocessing step for alphas (consistent with what is done in ML). For example:

from sklearn import preprocessing
import numpy as np

def preprocess(a):
a = np.nan_to_num(a - np.nanmean(a))
# return preprocessing.robust_scale(a) # security violation
return preprocessing.scale(a)


Lots of alternatives listed here:

http://scikit-learn.org/stable/modules/preprocessing.html

(although perhaps not all available yet on Q)

The penalty term version I've outlined above has the same behavior as a hard constraint in all the cases where it's possible to do so, but it also makes a best effort in cases where completely closing out the target positions would conflict with another hard constraint.

It would seem that the scale of the penalty term would matter (e.g. for a sufficiently large penalty, I'd think that the turnover constraint wouldn't work on non-universe stocks). How would users know how to set it?

Seems overly complicated. The universe shouldn't be changing that frequently anyway, so its contribution to turnover should be minimal. It would be more intuitive simply to close positions no longer in the universe.

Is it possible to detect if user has set MaxTurnover in constraints and only do special case handling (of adding positions to alphas) if it is set? Is it possible to demean alphas if adding positions to alphas with alpha=zero?

I'd think that the turnover constraint wouldn't work on non-universe stocks

Constraints are unconditional requirements that must be satisfied by the new portfolio, so no change in the implementation of the objective will ever cause the optimizer to emit a portfolio that violates a constraints.

What does matter in the context of a penalty term is that the scale of the alpha-maximization term in the objective is smaller than the scale of the "close out alpha-less positions" term. The easiest way to achieve that is to ensure that the scale of the alpha-maximization term is fixed by re-scaling alphas to fall between -1 and 1. There are a couple ways you could imagine doing that, but the only semantics-preserving rescaling is to divide all the alphas by the absolute value of the largest-magnitude alpha, which is safe because the w that maximizes dot(w, a) also maximizes dot(w, a * N) as long as N is a positive constant.

How would users know how to set it?

My expectation is that users shouldn't have to set it, unless they're doing something unusual. The goals is for the behavior of MaximizeAlpha to be what most people in this thread already assumed, which is that positions in assets with no alphas provided are always closed out unless doing so is impossible. The only case where scaling the penalty term would matter is if you want fine-grained control over what happens in the tiny percentage of cases where closing out the undesired positions isn't possible

Is it possible to detect if user has set MaxTurnover in constraints and only do special case handling (of adding positions to alphas) if it is set?

There are more cases than just MaxTurnover that can prevent us from closing a position. In general, an arbitrarily complex combination of constraints can conspire to make it impossible to close an existing position, and the only robust way to find that out is to do the full optimization (or something morally equivalent to it). To take an extreme example, you could encode a Boolean Satisfiability problem as a set of Basket constraints.

Is it possible to demean alphas if adding positions to alphas with alpha=zero?

I don't think it would be appropriate for an objective to automatically de-mean alphas. Consider, for example, that providing all-positive alphas will yield a long-only portfolio in the absence of a DollarNeutral (or similar) constraint, but the same alphas de-meaned will yield a long-short portfolio.

Wouldn't the median (or mean?) alpha be a better choice for positions than an arbitrary number like zero when there is no expectation that algorithm developers demean their alphas prior to calling order_optimal_portfolio for their long short algorithms?

I'm glad that my original post created this interesting debate, when clearly the reason my backtest created these "anomalies" was because my constraints were diagonally opposed to each other - "Go right! No, Go Left!" (my alpha vectors were all positive, and I simultaneously set constraints for a dollar neutral portfolio and a Beta between -0.1 and +0.1). It does make sense that Alpha vectors should not be "a preference ranking" but should instead be "a real expected Alpha over a given time horizon". There should however be some optional constraint that prevents non-pipeline output from being in your portfolio - Who's in control of this algo I just wrote? Is it optimize() or is it me!? ;)

I have to say that this is a completely new way of thinking about a trading system's construction and it's hard for old trading school guys like me to adapt (my current live models have big arrays to keep track of state, and they are all based on conditional triggers causing a Buy, a Sell, a Take Profit or a Stop Loss signal for each individual asset) .

I can immediately see the edge that optimize has over the "classic signal trigger methodology": the trigger methodology is by definition focused on building a portfolio at the micro level without regard to the macro portfolio level. For example, a momentum strategy on the Q500US with moving average triggers could very well buy 30 energy energy stocks on the same day because of a big oil price move - this probably wouldn't make much sense on a risk/reward basis for the entire portfolio. Optimize() would clearly allow you to cap your risk more efficiently given certain constraints rather than creating sector position limits using "classic signal trigger methodology". I'm curious: is there a constraint that would allow you to combine MaximizeAlpha and "MinimizePortfolioVariance" so that you could combine an Alpha vector with a portfolio co-variance ranking? (i.e.: if 2 assets have a similar Alpha vector, Optimize() might as well try to overweight the asset which is more likely to act as a dampener on overall portfolio volatility).

Interesting observation @Mattias. I wonder if you could just combine the alpha vector and co-variance weightings with some type of averaging?

You could, for example, just multiply the alpha vector by the covariance weightings for a final portfolio output that optimize could use.

Hi Cheng,

It was nice to meet you at QuantCon Singapore. You showed me during the Hackathon that you can read Python code faster than I can read English! I was very impressed how you could read code structure and debug someone else's code in the blink of an eye...

Since my last post, I thought a little bit about this issue and it's a really complicated problem (which I'm sure many smart guys have already tried to tackle - as it would seem like a natural thing one would want to optimize). I remember Delaney briefly mentioning the complexity of a similar optimization problem during the workshop.

While I'm sure there are various manipulations that could be done to the Alpha vector using co-variances, I don't think your suggestion would work. An asset with close to zero covariance to all the assets in the universe and a very high expected Alpha should be a pretty sought after asset for any strategy's long book, but multiplying them together would give such an asset a middle close to 0 ranking, when it should get a very high ranking.

One either has to use 1/ a covariance matrix with all the other assets in the universe over a given time frame (that's a seriously big matrix to optimise a theoritical Sharpe ratio), or 2/ one has to iterate through each of the single covariances with each permutation of the portfolio minus each asset (you get a different covariance for each permutation of weights in the n-1 portfolio universe).

It would appear that there are 2 different ways to approach this problem: A/ one optimises the matrix mentioned in 1/ above to find out what weights would have produced the highest Sharpe Ratio using historical co-variances and Alpha vectors instead of historical performances, or B) one uses a recursive technique to approximate a historical weight "sweet spot". As Approach A is seeking to find an exact optimization of the past to make an approximation of the future, I would have thought that Approach B will be just as good at estimating what will be the best weights for the future. Depending on the number of iterations required to get a decent approximation, Approach B could also have much less computational requirements than Approach A, for a similar end result.

For approach B, I would probably try to use Fisher Black's "combination of 2 risky assets" formula (Variance_2assetportfolio = ((x1^2)(s1^2)) + (2*x1*x2*r12*s1*s2) + ((x2^2)(s2^2)) where x is weights, s is standard deviations and r is correlation). I would start with an equally weighed portfolio and then parse through the portfolio multiple times optimizing weights one at a time. I would suspect that even after 5 iterations of parsing through the entire portfolio, I would have a pretty good setting for my weights...

Great Point. The approach I mentioned fails to account for the correlation between the alpha vector and covariance, and will likely lead to lackluster results. That said, the approach B you mentioned seems very sound since it looks at both the variances and correlation in both vectors.

Let me know if that works out for you!

In case you haven't seen this yet, this post talks a lot about portfolio diversification techniques that could be useful: https://www.quantopian.com/posts/hierarchical-risk-parity-comparing-various-portfolio-diversification-techniques

Thanks for sharing this post. I had missed it.

Hi Scott,

Regarding the non_universe_penalty term you mentioned earlier in a post, I had an idea yesterday that we could vary the non_universe_penalty in such a way that positions that have been longer in the portfolio and not in current alpha vector get higher penalty than those positions that have been added recently (and not in alpha vector). This would ensure starvation problem doesn't occur while satisfying the max turnover constraint. Similar to how some caching mechanisms work getting rid of the oldest cache item to free up space for a new item.

Thanks,
Leo