Back to Community
mean reversion w/ scipy.optimize.minimize

This is an adaptation of the algorithm described in:

Li, Bin, and Steven HOI. "On-Line Portfolio Selection with Moving Average Reversion." The 29th International Conference on Machine Learning (ICML2012), 2012.
http://icml.cc/2012/papers/168.pdf

To solve the portfolio optimization problem, it uses the Python module scipy.optimize.minimize instead of the analytic solution applied by the author.

Please note that the backtest is biased by the selection of some high-performance securities (i.e. intentional look-ahead bias).

Some things to try:

  • Different securities
  • Change the value of epsilon
context.eps = 1.005 # change epsilon here  
  • Change the length of the trailing window of prices:
    prices = history(10,'1d','price').as_matrix(context.stocks)[0:-1,:]
  • Re-allocate but don't re-balance, by changing to:
    if np.dot(res.x,x_tilde) - context.eps > 0: rebalance_portfolio(context,allocation) else: return
  • Compare the result to the algorithm used in the paper (do a Quantopian forum search on 'OLMAR' for applicable code)

Grant

Clone Algorithm
406
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54e0883c0ded0c0f099c5606
There was a runtime error.
31 responses

Here's the same algo as above, but with:

    # equal weighting  
    rebalance_portfolio(context, context.b_t)  
    return  

The portfolio is equal weight and re-balanced daily.

Clone Algorithm
406
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54e0951f203c160ee3bdcd3e
There was a runtime error.

Very interesting Grant. Thanks for sharing. The sharpe @4 is amazing but the beta is high.

Thanks Satyapravin,

I'm still working on it. I'm not confident that the optimizer doesn't sometimes end up off in the weeds, resulting in spurious values. This code is a Band-Aid:

    allocation = res.x  
    allocation[allocation<0] = 0  
    allocation = allocation/np.sum(allocation)  

It eliminates negative values in the allocation, which shouldn't result based on the bounds.

Grant

Grant: Very cool. Just to make sure, here you're trying to replicate the OLMAR algorithm? I suppose that's a good approach as once that is working it's easier to change to solve a slightly different optimization problem (e.g. a market-neutral one).

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hello Thomas,

Yes, I am trying to replicate the OLMAR algorithm. I haven't attempted an side-by-side comparisons, yet. And yes, the idea is to have more flexibility in the optimization problem. Just out of curiosity, what are the various optimizers available on Quantopian? Did CVXOPT ever get up and working?

Grant

@Grant, yeah, that's a tight little group you've got there...

def initialize(context):  
    set_symbol_lookup_date('2015-01-01')  
    context.stocks = symbols("VRTX","AKAM","BIIB","MNST","AMGN","MSFT","ROST","INTC","NFLX","WYNN",  
                             "TSLA","MU","CELG","FB","GILD","YHOO","WDC","ILMN","PCLN","AAPL")  
def handle_data(context, data):  
    if bool(get_open_orders()): return  
    for stock in context.stocks: order_target_percent(stock, 1.0 / len(context.stocks))

Did I miss something? Were you using OLMAR to pick them? Is that what your code ended up doing? If so, wicked! What universe did you use to pick them from?

Hello Market Tech,

Nothing fancy to pick the securities. As I said above:

Please note that the backtest is biased by the selection of some high-performance securities (i.e. intentional look-ahead bias).

As I recall, I did a Google search for the top Nasdaq 100 stocks of 2013 & 2014. I found that a couple of them did not trade all the way into 2015, so I just replaced them with a couple big tech names off the top of my head.

At this point, the algo only handles long positions, so I figured why not work with a portfolio that has some umph.

Besides the look-ahead bias, there may be a risk of over-fitting here, too. There are enough knobs to turn that with high-performing stocks, a little tweak here and there could end up making a big difference in returns. So, it would be reasonable to wonder if the algo is working at all. I've played around with a very different set of securities, and the performance is ugly:

    context.stocks =  [sid(19662),  # XLY Consumer Discrectionary SPDR Fund  
                       sid(19656),  # XLF Financial SPDR Fund  
                       sid(19658),  # XLK Technology SPDR Fund  
                       sid(19655),  # XLE Energy SPDR Fund  
                       sid(19661),  # XLV Health Care SPRD Fund  
                       sid(19657),  # XLI Industrial SPDR Fund  
                       sid(19659),  # XLP Consumer Staples SPDR Fund  
                       sid(19654),  # XLB Materials SPDR Fund  
                       sid(19660)]  # XLU Utilities SPRD Fund  

Grant

Here's one that seems to work with:

   context.stocks =  [sid(19662),  # XLY Consumer Discrectionary SPDR Fund  
                       sid(19656),  # XLF Financial SPDR Fund  
                       sid(19658),  # XLK Technology SPDR Fund  
                       sid(19655),  # XLE Energy SPDR Fund  
                       sid(19661),  # XLV Health Care SPRD Fund  
                       sid(19657),  # XLI Industrial SPDR Fund  
                       sid(19659),  # XLP Consumer Staples SPDR Fund  
                       sid(19654),  # XLB Materials SPDR Fund  
                       sid(19660)]  # XLU Utilities SPRD Fund  

There's a drawdown at the end (due to oil?), but otherwise it does o.k.

Grant

Clone Algorithm
406
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54e321a98df4410ee256ca07
There was a runtime error.

I suppose what I was wondering was that since your results in the second test above are precisely the results that that spot of code I posted produced, and rightly so, rebalance everyday for 3 years, what was the optimize code bringing to the table? If the straight even-weight beats the optimized, then what am I missing? Even with the X ETFs above, a straight rebalance, even weight beats the test.

Are you searching for the efficient frontier in continuously rebalanced weights? Are you sub-selecting from your group? Your statement "To solve the portfolio optimization problem" would seem to want to re-weight every instrument, on the fly, and then use those weights to rebalance. Is that the intent? Hey, if this is an ongoing WIP then just ignore me (my wife does (and my kids)).

Market Tech,

As you said, it's a work in progress. If you have a peek at Section 4.2 of the paper I posted above, there is an optimization problem described there. The basic idea is to minimize the difference between a new portfolio allocation vector and an existing one, subject to an inequality constraint. In this case, the constraint should result in distributing the capital such that under-performing securities relative to their mean prices are weighted more heavily than ones at or above their means. So, every security could get tweaked when there is an update. It may be that the strategy fundamentally doesn't work, or I have made mistakes in the code or my application of it. Since there's no great secret, I thought I'd post it for the masses to review.

Regarding the S&P 500 sector ETFs above, and an equal-weighted portfolio re-balanced every day, I've attached a backtest. It's what I'd expect---basically tracks the SPY benchmark. The optimization algo sputters at the end, but seems to be doing something right otherwise. Not too sexy, but interesting.

Grant

Clone Algorithm
406
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54e3bc9610c6310efe70cec6
There was a runtime error.

Here's another installment. I think I'll try to replicate the result Thomas W. published on https://www.quantopian.com/posts/olmar-3-dot-0-using-new-order-and-history-features. Note that this algo based on scipy.optimize.minimize does not yet have the averaging over the trailing window that Thomas has in his algo. --Grant

Clone Algorithm
406
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54e47b0fb14bb20efd0101a9
There was a runtime error.

Here's another tweak, and I added more high-flying stocks and SH, to reduce the volatility. Not sure if it helps, but note that I added the derivative of the objective function. --Grant

Clone Algorithm
406
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54ed21855baf880f19b80b3a
There was a runtime error.

Hey, grant. I had a quick question, what do you mean when you say your algorithm has equal weight and that you rebalance it daily. What sort of benefit does this give you?

Hello Jason,

I posted it as a crude way of showing that the algorithm might be working. If the return of the algo is greater than a simplistic equal weight portfolio, then maybe the algo is actually taking advantage of mean reversion.

Grant

Why can't you start a backtest on, say 1 Jan 2012? It gets an error from before that. The error stops between 1 Jan 2012 and 1 Jan 2013.

This is the error:
"There was a runtime error. AttributeError: 'SIDData' object has no attribute 'price'
... USER ALGORITHM:51, in tradeGo to IDE
context.b_t[i] = context.portfolio.positions[stock].amount*data[stock].price"

Thanks,
Nick

This error usually means you try to trade a symbol that does not have price date available. Usually because it's just trading, stopped trading or has no data for other reasons. Use the debugger to trace the error

That's exactly right - the algo tried to access data on a minute bar the stock didn't trade. So there wasn't a price available in "data[stock].price".

So fix that you can add the guard,

for i, stock in enumerate(context.stocks):  
  if stock in data:  
    context.b_t[i] = context.portfolio.positions[stock].amount*data[stock].price  
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

@Grant Great Algorithm, Could you elaborate what are you doing in this 2 lines?

 cons = ({'type': 'eq', 'fun': lambda x:  np.sum(x)-1.0},  
            {'type': 'ineq', 'fun': lambda x:  np.dot(x,x_tilde) - context.eps})  
 res= optimize.minimize(norm_squared, context.b_0, args=context.b_t,jac=norm_squared_deriv,method='SLSQP',constraints=cons,bounds=bnds)

Hello Erick,

The first line provides the constraints for the optimization. The first equality constraint corresponds to the normalization of the portfolio vector, x; it sums to 1. The second inequality constraint comes from the paper referenced in the algo (see section 4.2 of http://icml.cc/2012/papers/168.pdf). It is meant to maximize the expected return of the portfolio, based on the assumption of mean reversion.

The second line is the call to the optimization routine (see http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html). The arguments are the objective function, initial value, additional arguments for the objective function, derivatives of the objective function, the optimization method, contraints, and bounds.

There's another example of optimization in https://www.quantopian.com/posts/long-only-minimum-variance-portfolio-using-scipy-dot-optimize-dot-minimize, if you're interested.

Grant

Thanks Grant, for the explanation. By the way congrats, you truly deserved to be the first winner. Changing topic I like the approach of this strategy however it looks like as you mention needs to have good set of securities that behave well in mean reversion.

EG

Thanks Erick,

Yes, it may be the case that there needs to be an overall upward trend for the portfolio. Over the long-haul, one would hope that the market trends upward, so that if the portfolio is sufficiently diversified, there will be an edge.

Grant

Here's another update, using get_fundamentals. Sorry, the code is a bit of a wreck, but I'm still struggling to understand how get_fundamentals works. In any case, it seems to be working, and the performance looks decent. --Grant

Clone Algorithm
749
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54fce63caa421c0f0e2cb24a
We have migrated this algorithm to work with a new version of the Quantopian API. The code is different than the original version, but the investment rationale of the algorithm has not changed. We've put everything you need to know here on one page.
There was a runtime error.

Hi Grant,

Today I cloned your algo(last one with fundamentals) and tried to run it without changin anything, but got following error :

ValueError: shapes (1,20) and (1,20) not aligned: 20 (dim 1) != 1 (dim 0)  
USER ALGORITHM:119, in norm_squared  
return 0.5*np.dot(delta_b,delta_b)  

Is it possible that this error is related to this Quantopian upgrade?

Yes, that's certainly related to the update (numpy in this case). I fixed the dot() call and it should run with this version.

Clone Algorithm
103
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 550c26521cc1f82f3d2a3a1c
There was a runtime error.

Thanks Thomas,

What would happen, if I had put this algo to live trading?

Hasan

We would have emailed you before the upgrade and helped you to fix your algorithm. This is what we did for all live and contest algorithms.

Hi Thomas,

In one spot in the updated code, I see:

np.dot(delta_b, delta_b.T)  

But in other spots, I see:

np.dot(x,x_tilde)  
np.dot(allocation,x_tilde)  

Is the transpose of one of the vectors needed in some cases, but not in others?

Grant

Hi Grant,
Why the results change when backtesting in daily mode? Is this because of different price value when trade method is executed?
Is there any solution to make them the same?

Hello Hasan,

Well, I wouldn't recommend expending energy running on daily bars, since Quantopian doesn't support live trading in this mode. Is there a reason you've gone this route?

Grant

Thanks Grant,
There is no special reason, I just backtested in daily mode, and when decided to go for paper testing, confused by different results.

I guess I'd circle back and do some backtesting on minute bars. --Grant