Back to Community
Quantopian Lecture Series: The Good, the Bad, and the Correlated

Correlation analysis is ubiquitous in many fields of quantitative study. In the field of quantitative finance, it is especially useful for diversifying your portfolio by minimizing correlation between your assets' return streams. We will provide a primer on correlation analysis and discuss its relation to covariance.

In addition, we have released a notebook on Spearman Rank Correlation here. Spearman Rank Correlation is robust to differing scales in the underlying data and non-normal distributions.

This is part of Quantopian’s Summer Lecture Series. We are currently developing a quant finance curriculum and will be releasing clone-able notebooks and algorithms to teach key concepts. This notebook will be presented at this meetup. Stay tuned for more.

Credit for the notebooks goes to Evgenia 'Jenny' Nitishinskaya, and credit for the algorithms goes to David Edwards.

Loading notebook preview...
Notebook previews are currently unavailable.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

23 responses

We implemented a correlation reduction algorithm from this paper. The idea behind correlation reduction is that it helps diversify your returns streams and implicitly reduce risk exposure. Here is a backtest from an long-short momentum strategy that uses correlation reduction as a filter before its final buy/sell orders.

Clone Algorithm
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Long/Short Cross-Sectional Momentum

Author: David Edwards 

This algorithm implements a long/short strategy that looks at
an N day window of M day returns. It buys the assets that have 
had the most consistent returns relative to other assets and shorts
assets that have consistently underperformed.

Asset weights can be optionally scaled to reduce the correlations within
each basket. Even weights are used if the correlation reduction is not used.

reference for minimum correlation algorithm.
    http://cssanalytics.com/doc/MCA%20Paper.pdf
   
This algorithm was developed by David Edwards as part of 
Quantopian's 2015 summer lecture series. Please direct any 
questions, feedback, or corrections to [email protected]
"""

import numpy as np
import scipy
import pandas as pd


def initialize(context):
    
    context.lookback = 300
    context.return_window = 50
    context.longleverage = 0.5
    context.shortleverage = -0.5
    context.reduce_correlation = True
    
    # There's bad data for this security so I ignore it
    context.ignores = [sid(7143)]
    schedule_function(trade,
                      date_rule=date_rules.month_start(),
                      time_rule=time_rules.market_open(minutes=20))
    

    
def handle_data(context, data):
    leverage=context.account.leverage
    exposure=context.account.net_leverage
    record(leverage=leverage, exposure=exposure)
    
def trade(context, data):
    
    prices = np.log(history(context.lookback, '1d', 'price').dropna(axis=1))
    
    R = (prices / prices.shift(context.return_window)).dropna()
    
    # Subtract the cross-sectional average out of each data point on each day. 
    ranks = (R.T - R.T.mean()).T.mean()
    # Take the top and botton percentiles for the long and short baskets 
    lower, upper = ranks.quantile([.05, .95])
    shorts = ranks[ranks <= lower]
    longs = ranks[ranks >= upper]
    
    # Get weights that reduce the correlation within each basket
    if context.reduce_correlation:
        daily_R = prices.pct_change().dropna()
        longs = get_reduced_correlation_weights(daily_R[longs.index])
        shorts = get_reduced_correlation_weights(daily_R[shorts.index])
    else:
        # Use even weights
        longs = longs.abs()
        longs /= longs.sum()
        
        shorts = shorts.abs()
        shorts /= shorts.sum()
        
    for stock in data:
        if stock in context.ignores:
            continue
        try:
            if stock in shorts.index:
                order_target_percent(stock, 
                                     context.shortleverage * shorts[stock])
            elif stock in longs.index:
                order_target_percent(stock, 
                                     context.longleverage * longs[stock])
            else:
                order_target(stock, 0)
        except:
            log.warn("[Failed Order] stock = %s"%stock.symbol)
            


def get_reduced_correlation_weights(returns, risk_adjusted=True):
    """
    Implementation of minimum correlation algorithm.
    ref: http://cssanalytics.com/doc/MCA%20Paper.pdf
    
    :Params:
        :returns <Pandas DataFrame>:Timeseries of asset returns
        :risk_adjusted <boolean>: If True, asset weights are scaled
                                  by their standard deviations
    """
    correlations = returns.corr()
    adj_correlations = get_adjusted_cor_matrix(correlations)
    initial_weights = adj_correlations.T.mean()

    ranks = initial_weights.rank()
    ranks /= ranks.sum()

    weights = adj_correlations.dot(ranks)
    weights /= weights.sum()

    if risk_adjusted:
        weights = weights / returns.std()
        weights /= weights.sum()
    return weights

def get_adjusted_cor_matrix(cor):
    values = cor.values.flatten()
    mu = np.mean(values)
    sigma = np.std(values)
    distribution = scipy.stats.norm(mu, sigma)
    return 1 - cor.apply(lambda x: distribution.cdf(x))


def before_trading_start(context):
    num_stocks = 500
    fundamental_df = get_fundamentals(
        query(
            # To add a metric. Start by typing "fundamentals."
            fundamentals.valuation.market_cap,
        )
        .filter(fundamentals.valuation.market_cap > 1e8)
        .order_by(fundamentals.valuation.market_cap.desc())
        .limit(num_stocks)
    )
    update_universe(fundamental_df)
    
    
There was a runtime error.

Here, as a control, is the same strategy without the correlation reduction filter. Notice how the sharpe ratio is lower.

Clone Algorithm
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Long/Short Cross-Sectional Momentum

Author: David Edwards 

This algorithm implements a long/short strategy that looks at
an N day window of M day returns. It buys the assets that have 
had the most consistent returns relative to other assets and shorts
assets that have consistently underperformed.

Asset weights can be optionally scaled to reduce the correlations within
each basket. Even weights are used if the correlation reduction is not used.

reference for minimum correlation algorithm.
    http://cssanalytics.com/doc/MCA%20Paper.pdf
   
This algorithm was developed by David Edwards as part of 
Quantopian's 2015 summer lecture series. Please direct any 
questions, feedback, or corrections to [email protected]
"""

import numpy as np
import scipy
import pandas as pd


def initialize(context):
    
    context.lookback = 300
    context.return_window = 50
    context.longleverage = 0.5
    context.shortleverage = -0.5
    context.reduce_correlation = False
    
    # There's bad data for this security so I ignore it
    context.ignores = [sid(7143)]
    schedule_function(trade,
                      date_rule=date_rules.month_start(),
                      time_rule=time_rules.market_open(minutes=20))
    

    
def handle_data(context, data):
    leverage=context.account.leverage
    exposure=context.account.net_leverage
    record(leverage=leverage, exposure=exposure)
    
def trade(context, data):
    
    prices = np.log(history(context.lookback, '1d', 'price').dropna(axis=1))
    
    R = (prices / prices.shift(context.return_window)).dropna()
    
    # Subtract the cross-sectional average out of each data point on each day. 
    ranks = (R.T - R.T.mean()).T.mean()
    # Take the top and botton percentiles for the long and short baskets 
    lower, upper = ranks.quantile([.05, .95])
    shorts = ranks[ranks <= lower]
    longs = ranks[ranks >= upper]
    
    # Get weights that reduce the correlation within each basket
    if context.reduce_correlation:
        daily_R = prices.pct_change().dropna()
        longs = get_reduced_correlation_weights(daily_R[longs.index])
        shorts = get_reduced_correlation_weights(daily_R[shorts.index])
    else:
        # Use even weights
        longs = longs.abs()
        longs /= longs.sum()
        
        shorts = shorts.abs()
        shorts /= shorts.sum()
        
    for stock in data:
        if stock in context.ignores:
            continue
        try:
            if stock in shorts.index:
                order_target_percent(stock, 
                                     context.shortleverage * shorts[stock])
            elif stock in longs.index:
                order_target_percent(stock, 
                                     context.longleverage * longs[stock])
            else:
                order_target(stock, 0)
        except:
            log.warn("[Failed Order] stock = %s"%stock.symbol)
            


def get_reduced_correlation_weights(returns, risk_adjusted=True):
    """
    Implementation of minimum correlation algorithm.
    ref: http://cssanalytics.com/doc/MCA%20Paper.pdf
    
    :Params:
        :returns <Pandas DataFrame>:Timeseries of asset returns
        :risk_adjusted <boolean>: If True, asset weights are scaled
                                  by their standard deviations
    """
    correlations = returns.corr()
    adj_correlations = get_adjusted_cor_matrix(correlations)
    initial_weights = adj_correlations.T.mean()

    ranks = initial_weights.rank()
    ranks /= ranks.sum()

    weights = adj_correlations.dot(ranks)
    weights /= weights.sum()

    if risk_adjusted:
        weights = weights / returns.std()
        weights /= weights.sum()
    return weights

def get_adjusted_cor_matrix(cor):
    values = cor.values.flatten()
    mu = np.mean(values)
    sigma = np.std(values)
    distribution = scipy.stats.norm(mu, sigma)
    return 1 - cor.apply(lambda x: distribution.cdf(x))


def before_trading_start(context):
    num_stocks = 500
    fundamental_df = get_fundamentals(
        query(
            # To add a metric. Start by typing "fundamentals."
            fundamentals.valuation.market_cap,
        )
        .filter(fundamentals.valuation.market_cap > 1e8)
        .order_by(fundamentals.valuation.market_cap.desc())
        .limit(num_stocks)
    )
    update_universe(fundamental_df)
    
    
There was a runtime error.

Hi everyone,

We will be running a live webinar on this content next Thursday, July 16th at 12pm EDT. You can register here: http://bit.ly/thegoodthebadthecorrelatedwebinar. Once the webinar is wrapped up, we will share the recording here.

Hope you can join us!

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Here is the recording from our live webinar today on "The Good, the Bad, and The Correlated": http://bit.ly/thegoodbadcorrelatedvideo. We hope you enjoy it!

When I clone and back-test the algorithm I get a runtime error message.

There was a runtime error.
TypeError: 'float' object is not iterable
... USER ALGORITHM:57, in trade
lower, upper = ranks.quantile([.05, .95])

Could you please investigate?

Many thanks!

I take it back, it seems to be working (again). My apologies.

Inspite of my blunder, allow me another burning question, please.

If I change the trading frequency to every_day, i.e., if I replace

schedule_function(trade,  
                  date_rule=date_rules.month_start(),  
                  time_rule=time_rules.market_open(minutes=20))  

with

schedule_function(trade,  
                  date_rule=date_rules.every_day(),  
                  time_rule=time_rules.market_open(minutes=20))

the leverage and the exposure inevitably go haywire at some point.

How is this possible, especially when dealing with the algorithm version which does not perform correlation reduction,
because in this case the following lines get executed:

    longs = longs.abs()  
    longs /= longs.sum()  

    shorts = shorts.abs()  
    shorts /= shorts.sum()

followed by

for stock in data:  
    if stock in context.ignores:  
        continue  
    try:  
        if stock in shorts.index:  
            order_target_percent(stock,  
                                 context.shortleverage * shorts[stock])  
        elif stock in longs.index:  
            order_target_percent(stock,  
                                 context.longleverage * longs[stock])  
        else:  
            order_target(stock, 0)  
    except:  
        log.warn("[Failed Order] stock = %s"%stock.symbol)

where

context.longleverage = 0.5  
context.shortleverage = -0.5

Shouldn't this guarantee that the leverage remains (almost) exactly unity under any circumstances?

Hi Tim,
I know exactly what you are talking about, the issue is that the universe goes haywire through the financial crisis period. I don't have an exact reason for you, but my hunch is that it has to do with the massive number of bankruptcies, mergers, and other corporate events.

This sort of thing becomes a problem in backtesting whenever you are dealing with a dynamic universe. If I recall the leverage/exposure goes nuts even at a monthly frequency. Unfortunately I don't have a simple solution. You will have to write some helper functions to make sure all of the stocks you are buying/selling are still being actively traded. Maybe looking at their volumes is a good place to start. Sorry I don't have an easy answer, hope this helps though.

David

Many thanks for your explanation, David

In the meantime I have applied the following change, which seems to help quite a bit:

    try:  
        if get_open_orders(sid = stock):  
            continue  
        elif stock in shorts.index:  
            order_target_percent(stock,  
                                 context.shortleverage * shorts[stock])  
        elif stock in longs.index:  
            order_target_percent(stock,  
                                 context.longleverage * longs[stock])  
        else:  
            order_target(stock, 0)  
    except:  
        log.warn("[Failed Order] stock = %s"%stock.symbol)

instead of the original

    try:  
        if stock in shorts.index:  
            order_target_percent(stock,  
                                 context.shortleverage * shorts[stock])  
        elif stock in longs.index:  
            order_target_percent(stock,  
                                 context.longleverage * longs[stock])  
        else:  
            order_target(stock, 0)  
    except:  
        log.warn("[Failed Order] stock = %s"%stock.symbol)  

The open orders check is a great one, Tim. Alisa covered it in this list of ways to make your algorithm more robust: https://www.quantopian.com/posts/tips-for-writing-robust-algorithms-for-the-hedge-fund

My personal recommendation is that any algorithm you submit to the contest or live trade should be run through her checklist.

when i try to run this algo i get this error:

AttributeError: 'float' object has no attribute 'rank'
... USER ALGORITHM:105, in get_reduced_correlation_weightsGo to IDE
ranks = initial_weights.rank()

Hi Giuseppe,

It works for me, any chance you're running in daily mode? It needs to be minutely as far as I know.

Thanks,
Delaney

I tried to increase the leverage to 1, which I supposed the exposure should still be zero (someone please correct me if I was wrong), But at some point of backtesting, the exposure changed to -1 over night, what could be the case?

Clone Algorithm
21
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Long/Short Cross-Sectional Momentum

Author: David Edwards 

This algorithm implements a long/short strategy that looks at
an N day window of M day returns. It buys the assets that have 
had the most consistent returns relative to other assets and shorts
assets that have consistently underperformed.

Asset weights can be optionally scaled to reduce the correlations within
each basket. Even weights are used if the correlation reduction is not used.

reference for minimum correlation algorithm.
    http://cssanalytics.com/doc/MCA%20Paper.pdf
   
This algorithm was developed by David Edwards as part of 
Quantopian's 2015 summer lecture series. Please direct any 
questions, feedback, or corrections to [email protected]
"""

import numpy as np
import scipy
import pandas as pd


def initialize(context):
    
    context.lookback = 300
    context.return_window = 50
    context.longleverage = 1
    context.shortleverage = -1
    context.reduce_correlation = True
    
    # There's bad data for this security so I ignore it
    context.ignores = [sid(7143)]
    schedule_function(trade,
                      date_rule=date_rules.month_start(),
                      time_rule=time_rules.market_open(minutes=20))
    

    
def handle_data(context, data):
    leverage=context.account.leverage
    exposure=context.account.net_leverage
    record(leverage=leverage, exposure=exposure)
    
def trade(context, data):
    
    prices = np.log(history(context.lookback, '1d', 'price').dropna(axis=1))
    
    R = (prices / prices.shift(context.return_window)).dropna()
    
    # Subtract the cross-sectional average out of each data point on each day. 
    ranks = (R.T - R.T.mean()).T.mean()
    # Take the top and botton percentiles for the long and short baskets 
    lower, upper = ranks.quantile([.05, .95])
    shorts = ranks[ranks <= lower]
    longs = ranks[ranks >= upper]
    
    # Get weights that reduce the correlation within each basket
    if context.reduce_correlation:
        daily_R = prices.pct_change().dropna()
        longs = get_reduced_correlation_weights(daily_R[longs.index])
        shorts = get_reduced_correlation_weights(daily_R[shorts.index])
    else:
        # Use even weights
        longs = longs.abs()
        longs /= longs.sum()
        
        shorts = shorts.abs()
        shorts /= shorts.sum()
        
    for stock in data:
        if stock in context.ignores:
            continue
        try:
            if stock in shorts.index:
                order_target_percent(stock, 
                                     context.shortleverage * shorts[stock])
            elif stock in longs.index:
                order_target_percent(stock, 
                                     context.longleverage * longs[stock])
            else:
                order_target(stock, 0)
        except:
            log.warn("[Failed Order] stock = %s"%stock.symbol)
            


def get_reduced_correlation_weights(returns, risk_adjusted=True):
    """
    Implementation of minimum correlation algorithm.
    ref: http://cssanalytics.com/doc/MCA%20Paper.pdf
    
    :Params:
        :returns <Pandas DataFrame>:Timeseries of asset returns
        :risk_adjusted <boolean>: If True, asset weights are scaled
                                  by their standard deviations
    """
    correlations = returns.corr()
    adj_correlations = get_adjusted_cor_matrix(correlations)
    initial_weights = adj_correlations.T.mean()

    ranks = initial_weights.rank()
    ranks /= ranks.sum()

    weights = adj_correlations.dot(ranks)
    weights /= weights.sum()

    if risk_adjusted:
        weights = weights / returns.std()
        weights /= weights.sum()
    return weights

def get_adjusted_cor_matrix(cor):
    values = cor.values.flatten()
    mu = np.mean(values)
    sigma = np.std(values)
    distribution = scipy.stats.norm(mu, sigma)
    return 1 - cor.apply(lambda x: distribution.cdf(x))


def before_trading_start(context):
    num_stocks = 500
    fundamental_df = get_fundamentals(
        query(
            # To add a metric. Start by typing "fundamentals."
            fundamentals.valuation.market_cap,
        )
        .filter(fundamentals.valuation.market_cap > 1e8)
        .order_by(fundamentals.valuation.market_cap.desc())
        .limit(num_stocks)
    )
    update_universe(fundamental_df)
    
    
There was a runtime error.

@ Giuseppe, is it possible that you are limiting the percentile of selected stocks so that only one stock is in the long or short baskets? If so a length 1 series will coerce to a float and you would get the error you're describing.

@ Yachi, your assumption that the exposure should still be 0 is correct. I am not sure what happened in your situation (I'll take a closer look when I find a minute). There are some issues with stocks being delisted around the 08 debacle that cause similar problems. You will have to hone in on the trades the algo made that caused it to go nutty. I find it interesting that the allocations never went back to normal after another rebalance.

Thanks David and Delaney, i actually was running the algo in daily mode... this was the issue

@David, Thank you for your explanation, I take into consideration of delisting in my code this time by adding a "if get_open_orders(sid = stock): /n continue "statement, and it did alleviates the frenzy of leverage. However, it did not explain the abruptly decreasing of leverage. I also tried out the code in different period, interestingly, I found out the leverage changing is not consistent in different timeframe, I tried to figured out if it is caused by the algo... For example, if took timeframe from 2008-05-01 to 2008-12-31, the deleveraging seemed to occurred periodically

Clone Algorithm
21
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Long/Short Cross-Sectional Momentum

Author: David Edwards 

This algorithm implements a long/short strategy that looks at
an N day window of M day returns. It buys the assets that have 
had the most consistent returns relative to other assets and shorts
assets that have consistently underperformed.

Asset weights can be optionally scaled to reduce the correlations within
each basket. Even weights are used if the correlation reduction is not used.

reference for minimum correlation algorithm.
    http://cssanalytics.com/doc/MCA%20Paper.pdf
   
This algorithm was developed by David Edwards as part of 
Quantopian's 2015 summer lecture series. Please direct any 
questions, feedback, or corrections to [email protected]
"""

import numpy as np
import scipy
import pandas as pd


def initialize(context):
    
    context.lookback = 300
    context.return_window = 50
    context.longleverage = 1
    context.shortleverage = -1
    context.reduce_correlation = True
    

    
    # There's bad data for this security so I ignore it
    context.ignores = [sid(7143)]
    schedule_function(trade,
                      date_rule=date_rules.month_start(),
                      time_rule=time_rules.market_open(minutes=20))
    

    
def handle_data(context, data):
    leverage=context.account.leverage
    exposure=context.account.net_leverage
    record(leverage=leverage, exposure=exposure)
    
def trade(context, data):
    
    prices = np.log(history(context.lookback, '1d', 'price').dropna(axis=1))
    
    R = (prices / prices.shift(context.return_window)).dropna()
    
    # Subtract the cross-sectional average out of each data point on each day. 
    ranks = (R.T - R.T.mean()).T.mean()
    # Take the top and botton percentiles for the long and short baskets 
    lower, upper = ranks.quantile([.05, .95])
    shorts = ranks[ranks <= lower]
    longs = ranks[ranks >= upper]
    
    # Get weights that reduce the correlation within each basket
    if context.reduce_correlation:
        daily_R = prices.pct_change().dropna()
        longs = get_reduced_correlation_weights(daily_R[longs.index])
        shorts = get_reduced_correlation_weights(daily_R[shorts.index])
    else:
        # Use even weights
        longs = longs.abs()
        longs /= longs.sum()
        
        shorts = shorts.abs()
        shorts /= shorts.sum()
        
    for stock in data:
        try:

     
            if get_open_orders(sid = stock):  
                continue 
            
            elif stock in shorts.index:
                order_target_percent(stock, 
                                     context.shortleverage * shorts[stock])
            elif stock in longs.index:
                 order_target_percent(stock, 
                                         context.longleverage * longs[stock])
            else:
                  order_target(stock, 0)
        except:    
                log.warn("[Failed Order] stock = %s"%stock.symbol)
            


def get_reduced_correlation_weights(returns, risk_adjusted=True):
    """
    Implementation of minimum correlation algorithm.
    ref: http://cssanalytics.com/doc/MCA%20Paper.pdf
    
    :Params:
        :returns <Pandas DataFrame>:Timeseries of asset returns
        :risk_adjusted <boolean>: If True, asset weights are scaled
                                  by their standard deviations
    """
    correlations = returns.corr()
    adj_correlations = get_adjusted_cor_matrix(correlations)
    initial_weights = adj_correlations.T.mean()

    ranks = initial_weights.rank()
    ranks /= ranks.sum()

    weights = adj_correlations.dot(ranks)
    weights /= weights.sum()

    if risk_adjusted:
        weights = weights / returns.std()
        weights /= weights.sum()
    return weights

def get_adjusted_cor_matrix(cor):
    values = cor.values.flatten()
    mu = np.mean(values)
    sigma = np.std(values)
    distribution = scipy.stats.norm(mu, sigma)
    return 1 - cor.apply(lambda x: distribution.cdf(x))


def before_trading_start(context):
    num_stocks = 500
    fundamental_df = get_fundamentals(
        query(
            # To add a metric. Start by typing "fundamentals."
            fundamentals.valuation.market_cap,
        )
        .filter(fundamentals.valuation.market_cap > 1e8)
        .order_by(fundamentals.valuation.market_cap.desc())
        .limit(num_stocks)
    )
    update_universe(fundamental_df)
    
    
There was a runtime error.

What confused me is if I extend the timeframe from 2008-02-01 to 2009-04-30, the leveraging only occurred once, on Jun,1 2008,
what could be the cause of this situation?

Clone Algorithm
21
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Long/Short Cross-Sectional Momentum

Author: David Edwards 

This algorithm implements a long/short strategy that looks at
an N day window of M day returns. It buys the assets that have 
had the most consistent returns relative to other assets and shorts
assets that have consistently underperformed.

Asset weights can be optionally scaled to reduce the correlations within
each basket. Even weights are used if the correlation reduction is not used.

reference for minimum correlation algorithm.
    http://cssanalytics.com/doc/MCA%20Paper.pdf
   
This algorithm was developed by David Edwards as part of 
Quantopian's 2015 summer lecture series. Please direct any 
questions, feedback, or corrections to [email protected]
"""

import numpy as np
import scipy
import pandas as pd


def initialize(context):
    
    context.lookback = 300
    context.return_window = 50
    context.longleverage = 1
    context.shortleverage = -1
    context.reduce_correlation = True
    

    
    # There's bad data for this security so I ignore it
    context.ignores = [sid(7143)]
    schedule_function(trade,
                      date_rule=date_rules.month_start(),
                      time_rule=time_rules.market_open(minutes=20))
    

    
def handle_data(context, data):
    leverage=context.account.leverage
    exposure=context.account.net_leverage
    record(leverage=leverage, exposure=exposure)
    
def trade(context, data):
    
    prices = np.log(history(context.lookback, '1d', 'price').dropna(axis=1))
    
    R = (prices / prices.shift(context.return_window)).dropna()
    
    # Subtract the cross-sectional average out of each data point on each day. 
    ranks = (R.T - R.T.mean()).T.mean()
    # Take the top and botton percentiles for the long and short baskets 
    lower, upper = ranks.quantile([.05, .95])
    shorts = ranks[ranks <= lower]
    longs = ranks[ranks >= upper]
    
    # Get weights that reduce the correlation within each basket
    if context.reduce_correlation:
        daily_R = prices.pct_change().dropna()
        longs = get_reduced_correlation_weights(daily_R[longs.index])
        shorts = get_reduced_correlation_weights(daily_R[shorts.index])
    else:
        # Use even weights
        longs = longs.abs()
        longs /= longs.sum()
        
        shorts = shorts.abs()
        shorts /= shorts.sum()
        
    for stock in data:
        try:

     
            if get_open_orders(sid = stock):  
                continue 
            
            elif stock in shorts.index:
                order_target_percent(stock, 
                                     context.shortleverage * shorts[stock])
            elif stock in longs.index:
                 order_target_percent(stock, 
                                         context.longleverage * longs[stock])
            else:
                  order_target(stock, 0)
        except:    
                log.warn("[Failed Order] stock = %s"%stock.symbol)
            


def get_reduced_correlation_weights(returns, risk_adjusted=True):
    """
    Implementation of minimum correlation algorithm.
    ref: http://cssanalytics.com/doc/MCA%20Paper.pdf
    
    :Params:
        :returns <Pandas DataFrame>:Timeseries of asset returns
        :risk_adjusted <boolean>: If True, asset weights are scaled
                                  by their standard deviations
    """
    correlations = returns.corr()
    adj_correlations = get_adjusted_cor_matrix(correlations)
    initial_weights = adj_correlations.T.mean()

    ranks = initial_weights.rank()
    ranks /= ranks.sum()

    weights = adj_correlations.dot(ranks)
    weights /= weights.sum()

    if risk_adjusted:
        weights = weights / returns.std()
        weights /= weights.sum()
    return weights

def get_adjusted_cor_matrix(cor):
    values = cor.values.flatten()
    mu = np.mean(values)
    sigma = np.std(values)
    distribution = scipy.stats.norm(mu, sigma)
    return 1 - cor.apply(lambda x: distribution.cdf(x))


def before_trading_start(context):
    num_stocks = 500
    fundamental_df = get_fundamentals(
        query(
            # To add a metric. Start by typing "fundamentals."
            fundamentals.valuation.market_cap,
        )
        .filter(fundamentals.valuation.market_cap > 1e8)
        .order_by(fundamentals.valuation.market_cap.desc())
        .limit(num_stocks)
    )
    update_universe(fundamental_df)
    
    
There was a runtime error.

At the leverage drop from 1.86 to 1.08, you can examine the trades at that time by adding track_orders(). You might want to set a start date in its options for convenience and to avoid overwhelming the logging window with a boatload of output, no stop date required, just let 'er rip and see what happens.

Edit: I had a bug in PvR code now fixed where it wasn't always accounting for all shorting as risk and have had to drop some of what I said here previously about this code. Note this code only spends ~1/4 of initial capital however it shorts beyond initial capital ($2.3M) and that's extra risk. The beta at way below zero is often (or always, I'm not sure) a sign of copious shorting. If you adjust to go long more, to use more of that starting capital, and short less, the mountain during the '08 crash will be less and I think the long-term separation above SPY will be correspondingly higher. Consider/try increasing longleverage and decreasing shortleverage and see what the differences would be thru 2016. That'll take hours. :/ :) You can run two or more at the same time.

I was seeing a flatline for about a year, all in cash, to look into, a bit beyond the range of the test above.

I have a question about the use of log prices in this algo. Because the prices are converted to log on line 50, shouldn't they be subtracted rather than divided on line 52?
Replacing:

    prices = np.log(history(context.lookback, '1d', 'price').dropna(axis=1))  
    R = (prices / prices.shift(context.return_window)).dropna()  

with:

    prices = np.log(history(context.lookback, '1d', 'price').dropna(axis=1))  
    R = prices.diff(context.return_window).dropna()  

The pct_change calculation on line 63 would also need a similar change.

I played around in the research environment and found the answer to my own question. By dividing the log prices the results are normalized. Subtracting them gives a ranking that's weighted toward higher priced stocks, which may have a higher log return but lower percent return. But that brings me to the question, why use log prices? What advantage does it offer over just dividing the normal prices?

The algorithms in this thread may cease working briefly as we're doing maintenance during the switch to Q2. Please let me know if this causes any issues.

Do you by chance have a fresh link to the paper?

Looks like you can get it here: https://www.scribd.com/document/106570475/CSS-Analytics-Minimum-Correlation-Algorithm#from_embed