Back to Community
Universal portfolios

This algo is an implementation of Universal Portfolios, described in paper by Professor Thomas M. Cover from Stanford. His book is one of the standard textbooks in Information Theory. For implementation tricks to bring the theory into practice, please refer to comments in the code.

This model makes no statistical assumption about distribution of asset prices (e.g. normal, log-normal). Also, it leaves few rooms for backtest-fitting, as only parameters are choice of (i) assets in portfolio and (ii) look back period. I chose very basic 8 ETFs and one year period.

Interestingly, in late 2008 the model decided to own none of the 8 ETFs and just held cash during financial crisis. Even more interestingly, the model again sold all ETFs on August 26, 2015, so we just have $3M in cash today (started with $1M 10 years ago)...

Note: It is long-only and with high beta - not suitable as for contest. But it could be potentially a useful framework in allocating funds to algos in hedge fund.

Clone Algorithm
711
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 55f318d38c67cb0dfdbeebac
We have migrated this algorithm to work with a new version of the Quantopian API. The code is different than the original version, but the investment rationale of the algorithm has not changed. We've put everything you need to know here on one page.
There was a runtime error.
18 responses

Thanks for the well-written algorithm with comprehensive comments!

Hi Naoki,

Thank you very much for sharing this, it looks really nice.
I was wondering though: How is it possible that the script chooses to keep the cash and not buy any ETF's?
The bins only consider the money to be put in at least one of the ETF's, not none, am I right?

Also, just as a remark, there is a numpy function np.nan_to_num(arr), doing exactly what it implies.

At least one ETF get the money, and in all such C(10,8) portfolios all of the funds are invested. But when taking the average of portfolio holdings weighted by the return of each portfolio, return happened to be negative across most portfolios. The script decides not to buy any ETFs if all holdings turns negative in this way.

Thanks for the tip on nan_to_num!

That makes sense..

Something else I was wondering; The bins you are generating are basically just linear combinations of the identity matrix (all money in one of the securities)
Why does the outcome differ (it does) when you consider all these bins separately, and not just using the identity matrix as 'bins'?

Even turning off the 'clipping' operations such as zeroing negative weights and disregarding trades under 1000, there still is a difference between the two..

The result from portfolios with all money in each security, and result of all possible portfolio is different, because we rebalance it daily. Return of average of 100% stock and 100% bond is different from the return of 50% bond/50% stock if you rebalance it (if you don't rebalance at all, it should be the same).

In terms of algebra, we are taking S = X B where matrix X (252 days of return * m securities) dotted by matrix B (m securities * all possible portfolios), which will produce matrix S (252 days of return * all possible portfolio). So using B = all portfolios or B = I (identity matrix) will produce completely different S.

Thank you for sharing. It's very interesting this script holds cash during downturn. The high beta and return are mainly due to leverage. Can you elaborate why you choose leverage of 2?

@Joe Black

I dig this strategy a little bit. There is an explanation here[1]. In general, this strategy does not perform well when the volatility of the underlying securities is low. Hence we need to improve the performance of this strategy by amplifying the volatility with a leverage. You could try a leverage of 10, that has a lot a fun :)

[1]http://epchan.blogspot.sg/2007/01/universal-portfolios.html

I did some light reading on the algorithm and it seems to be a bit slow in reaching the optimal return with hindsight. Very cool nonetheless. Doubly nice that it holds cash on a downturn.

Hi Joe, I chose leverage of 2 because of more attractive return, with still reasonable drawdown.

I reviewed the paper again and realized I misinterpreted one thing.

In line 94, after calculating returns for all portfolio, the script subtract 1 (e.g. 1% loss is -0.01). Then it takes the weighted average of all portfolio. However, what the paper meant was not subtracting 1 (e.g. 1% loss is 0.99) and taking the weighted average.

Now I did removed the -1 and backtested, but 1 year is too short and 'universal portfolio' ends up almost just the equal weight. So as in the original paper, this version below takes the the power 20 of the 1 year return to emulate 20-year return, and takes the weighted average.

Because there is no negative returns (just the returns less than 1) you never get negative weights in universal portfolio. So no need to skip shorting ETFs in trading according to the theory. I was checking the paper for how to treat negative weight and realized the above discrepancy.

But interestingly, the original -1 version has better return and drawdown. Would this -1 universal portfolio still approach the optimal, even if you subtract 1 before integration?

Clone Algorithm
711
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 55f8dd3cea2cd40e0029c2bd
We have migrated this algorithm to work with a new version of the Quantopian API. The code is different than the original version, but the investment rationale of the algorithm has not changed. We've put everything you need to know here on one page.
There was a runtime error.

"But interestingly, the original -1 version has better return and drawdown"

As expected both versions have about the same risk-adjusted Return in "normal" situations. I made a backtest post-crisis (starting 1 January 2010) and the new/corrected version has higher beta (hence the bigger DD) and higher return, but about the same Shape-ratio.

The diference comes from the 1st version going to cash on the Great Recession correction. You should probably focus on that and to try to understand why/diferences.

Hi Naoki,
Yes the paper doesn't subtract 1 from the return. Your first script also used sum(abs()) as denominator, which leads to negative weights. The idea is to weight performance for all possible constant rebal portfolio. Subtract 1 or not doesn't change the idea of weighting based on performance, however that sum(abs()) did make a difference.
Also I think the paper doesn't suggest use a certain look back period. You could start on any day with equal weight, and keep re-balancing since that day based on actual performance.

A couple of things.
I tried this algorithm from 2004 and it didn't seem to do well, any idea why?
Also, I haven't tried all of this, but can we extend this algorithm with the following etfs?
1. And spx short etf for the bear market. This didn't work for me.
2. Some international etfs, like Asia pac, Europe, emerging markets, etc etf

Any thoughts?

hi this is my first comment on quantopian . nice to meet you here

Hi Saravanan, it does not work well for 2004 because some ETFs were't there back then (the algo needs 1 additional year of back data as well). SPX does not work well because it's an inverse of SPY. The Universal Portfolio considers every possible combination of stocks in the universe. But putting large cap (close to SPY) and SPX together distorts calculation.

By the way, this is not the kind of algo Quantopian is looking for at the moment. I would say there are 3 type of algo trading: portfolio optimization (like this one), momentum (go with trend) and arbitrage (buy and sell something at the same time). Quantopian is looking for arbitrage type algo (i.e. low-beta) because you can create value regardless of the stock market going up or down, and that's what the institutional investors want in hedge funds, to diversify their portfolio.

Nevertheless, Universal Portfolio is a cool theory with mathematical backing, and it is an interesting problem to code. I hope techniques and tricks used in the script would be useful in creating up a great algo.

Starting with 1 million, the first algo exchanges 4.5 million dollars to profit 2 million for a return (taking leverage into account) of only 45%. The second is 41%. Aside from that, the flatline in '08 looks great. Sure would be good to even make money during that downturn, more difficult than expected when I restrict myself to only what I think can truly do well going forward.

Naoki,

I like this code a lot and wanted to experiment . It worked for me the other day, but now I get following error

Something went wrong. Sorry for the inconvenience. Try using the built-in debugger to analyze your code. If you would like help, send us an email.
ValueError: cannot convert float NaN to integer
There was a runtime error on line 171.

Line 171 points to your execute function. It appears an NaN is creeping into the data, probably on this line

tradingvolume = history(3, frequency='1d', field='volume', ffill=True).mean()

but doesn't mean adjust for NaN's be using a 0 value?

I then inserted following

 if  pd.isnull(tradingvolume[sec]):  
      continue

but this did not eliminate error.

Before I go crazy with this I figure somebody else has already caught an fixed this bug. btw, comes up in run for dates 01/01/2003 - 01/01/2005

I know this is really old but how can I port over the logic that determines the market is in a downturn and chooses to hold cash?

I would like to use it for other Pipeline Algos I have and especially the one attached. I know the attached uses XIV but we could change that to prove the logic works during 2008. Thanks!

Clone Algorithm
96
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 571a67e9ec97390f81cb1689
There was a runtime error.

Here is another Algo I would like to use that logic in. Would it be possible to implement it with the attached?

Clone Algorithm
144
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 571ae122dd59d50f7894e936
There was a runtime error.