There is a subtlety in the
order_optimal_portfolio method which often goes un-noticed. You noticed it.
The universe of stocks this method uses, in its efforts to come up with an optimal solution, is defined as
Notice the universe INCLUDES current holdings. The optimize method uses any current holdings along with any new securities referenced in the objective . It does NOT assume you want to close current holdings just because they are not referenced in the objective (ie the pipeline output). So what's going on here is that the most optimal portfolio keeps a lot of short positions. They keep growing.
Now, what one typically may want (which I believe is the case here) is to close any current holdings not found in the current pipeline output. This can be done by using the vanilla
PositionConcentration method. Something like this:
not_in_pipe_weights = pd.Series(0, index=not_in_pipe_securities)
min_weights = not_in_pipe_weights,
max_weights = not_in_pipe_weights,
default_min_weight = -MAX_SHORT_POSITION_SIZE,
default_max_weight = MAX_SHORT_POSITION_SIZE,
This will place a zero min and max position (ie no position) for all held securities which are not in the pipe output. Any others (ie those in the pipeline) will get the default values. Using the
positionConcentration.with_equal_bounds method will allow for non-zero weights of existing assets. There are other ways to close out existing positons (eg using the
CannotHold method) but this highlights the issue.
See attached backtest and code.
Many thanks to @Burrito Dan for originally pointing out this behavior (see https://www.quantopian.com/posts/optimize-api-now-available-in-algorithms)