Back to Community
Please help, why do my short positions keep rising

Hi,

I am trying a simple long short algorithm. When I print the number of securities in universe, it shows 300. But when I plot long and short positions the short positions keep rising. Could someone please help.

Best regards,
Pravin

Clone Algorithm
5
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 599799ee29c3df50fb77d677
There was a runtime error.
6 responses

I believe this line had a mistake, but interestingly fixing it didn't alleviate the problem:

        elif context.portfolio.positions[sid].amount < 0:  
            shorts += 1  

Hi,

Anyone from Quantopian support? I wonder why my short positions keep rising?

Bet regards,
Pravin

There is a subtlety in the order_optimal_portfolio method which often goes un-noticed. You noticed it.

The universe of stocks this method uses, in its efforts to come up with an optimal solution, is defined as

set(current_nonempty_positions).union(assets_referenced_by_objective) 

Notice the universe INCLUDES current holdings. The optimize method uses any current holdings along with any new securities referenced in the objective . It does NOT assume you want to close current holdings just because they are not referenced in the objective (ie the pipeline output). So what's going on here is that the most optimal portfolio keeps a lot of short positions. They keep growing.

Now, what one typically may want (which I believe is the case here) is to close any current holdings not found in the current pipeline output. This can be done by using the vanilla PositionConcentration method. Something like this:

not_in_pipe_weights = pd.Series(0, index=not_in_pipe_securities)

constraints.append(  
        opt.PositionConcentration(  
            min_weights = not_in_pipe_weights,  
            max_weights = not_in_pipe_weights,  
            default_min_weight = -MAX_SHORT_POSITION_SIZE,  
            default_max_weight = MAX_SHORT_POSITION_SIZE,  
        ))

This will place a zero min and max position (ie no position) for all held securities which are not in the pipe output. Any others (ie those in the pipeline) will get the default values. Using the positionConcentration.with_equal_bounds method will allow for non-zero weights of existing assets. There are other ways to close out existing positons (eg using the CannotHold method) but this highlights the issue.

See attached backtest and code.

Many thanks to @Burrito Dan for originally pointing out this behavior (see https://www.quantopian.com/posts/optimize-api-now-available-in-algorithms)

Clone Algorithm
3
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5999a307365bbe51085a0e21
There was a runtime error.

Maybe we could have Q team's @Scott Sanderson to reply to this post to clarify if the order_optimal_portfolio expects the portfolio positions to be all present in the ranked pipeline output and if they are not present then those positions continue to be kept (which would be a big divergence from expected behavior imho). The fact that something is not there in the pipeline output is because it is not ranked any more by the algorithm and the intent is that they disappear from the portfolio (i.e weight = 0) .

@Dan thanks for the help. What happens if a security is both in the alpha series and the past position series. Which weight will take effect?

@ Aqua Rooster

"What happens if a security is both in the alpha series and the past position series. Which weight will take effect?"

One can program the weights and what to hold pretty much however one wants. However, with the implementation I did it was simply anything currently held and NOT in the alpha series (ie not in the pipeline output) would get a weight of zero. Everything else, including anything currently held and ALSO in the pipeline output, get's the default weight.

One could, however, make a few tweaks to this logic. The pipeline returns stocks selected for long and stocks selected for short. One could force the optimizer to respect these choices by setting appropriate weights for each set. One could set a max weight of 0 for shorts and a min weight of 0 for longs. In the current implementation one could expect a stock to be shorted but the optimizer may choose to go long.

As an aside, your choices for min and max weights may be a be too stringent. They are being set as 1 / NUM_LONG_POSITIONS (and similar for shorts). This is really saying optimize all you want but in the end we want equal weighting. Maybe try setting the min and max to 2x or even 5x this value to give the optimizer more leeway and see what happens.