Back to Community
Questions on Beta Hedging Example of Quantopian Lecture Series

Hi Everyone,

I am trying to follow Quantopian Lecture Series and I have some questions regarding this beautifully written algorithm that I cloned from "Beta Hedging" Example of Quantopian Lecture 4.

First question:

def buy_assets(context, data):  
    all_prices = history(1, '1d', 'price', ffill=True)  
    eligible_assets = [asset for asset in all_prices  
                       if asset not in context.dont_buys  
                       and asset != context.index]  
    pct_per_asset = 1.0 / len(eligible_assets)  
    context.pct_per_asset = pct_per_asset  
    for asset in eligible_assets:  
        # Some assets might cause a key error due to being delisted  
        # or some other corporate event so we use a try/except statement  
        try:  
            if get_open_orders(sid=asset):  
                continue  
            order_target_percent(asset, pct_per_asset)  
        except:  
            log.warn("[Failed Order] asset = %s"%asset.symbol)  

In the above buy_assets() function, if there is open orders left from the previous period ( in this case, month ) for a particular asset, the algorithm leaves it alone. Won't that be

potentially problematic ? Because order_target_percent(asset, pct_per_asset) was called in the last period, and "pct_per_asset" for the current period would most likely be different from

that of the previous period.

Second Question:

def before_trading_start(context, data):  
    # Number of stocks to find  
    num_stocks = 100  
    fundamental_df = get_fundamentals(  
        query(  
            # To add a metric. Start by typing "fundamentals."  
            fundamentals.valuation_ratios.earning_yield,  
            fundamentals.valuation.market_cap,  
        )  
        .filter(fundamentals.valuation.market_cap > 1e9)  
        .order_by(fundamentals.valuation_ratios.earning_yield.desc())  
        .limit(num_stocks)  
    )  
    update_universe(fundamental_df)  

My understanding is that when the universe is updated, stocks that qualify that fundamental metrics, ( columns of fundamental_df) are added and the ones that don't qualify would be kicked

out of the universe. If so, there should be only around 100 stocks any given day. My test shows that is not the case as you can see in custom data plot as

stocks_in_universe  

What happens to the stocks that are already in the portfolio ? Will they still be part of the universe ? When do stocks leave universe?

And, would those stocks, ( stocks that are already in the portfolio, but didn't pass the fundamental test) show up when
all_prices = history(1, '1d', 'price', ffill=True) is called ? If this is the case, they can still be part of
eligible_assets even though they didn't pass the fundamental test. Won't that be against the underlying strategy ?

Third Question:

Beta of the strategy is calculated by taking the average of beta of individual assets. What if we calculate beta using the returns of the whole portfolio? What are the pros and cons ?

Clone Algorithm
2
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
import pandas as pd
import numpy as np
import statsmodels.api as sm
import math


def initialize(context):
    use_beta_hedging = True # Set to False to trade unhedged
    # Initialize a universe of liquid assets
    schedule_function(buy_assets,
                      date_rule=date_rules.month_start(),
                      time_rule=time_rules.market_open())
    if use_beta_hedging:
        # Call the beta hedging function one hour later to
        # make sure all of our orders have gone through.
        schedule_function(hedge_portfolio,
                          date_rule=date_rules.week_start(),
                          time_rule=time_rules.market_open(hours=1))
    # trading days used for beta calculation
    context.lookback = 150
    # Used to aviod purchasing any leveraged ETFs 
    context.dont_buys = security_lists.leveraged_etf_list
    # Current allocation per asset
    context.pct_per_asset = 0
    context.index = symbol('SPY')
    
def before_trading_start(context, data):
    # Number of stocks to find
    num_stocks = 100
    fundamental_df = get_fundamentals(
        query(
            # To add a metric. Start by typing "fundamentals."
            fundamentals.valuation_ratios.earning_yield,
            fundamentals.valuation.market_cap,
        )
        .filter(fundamentals.valuation.market_cap > 1e9)
        .order_by(fundamentals.valuation_ratios.earning_yield.desc())
        .limit(num_stocks)
    )
    update_universe(fundamental_df)
    
def buy_assets(context, data):
    all_prices = history(1, '1d', 'price', ffill=True)
    record(stocks_in_universe=all_prices.shape[1])
    eligible_assets = [asset for asset in all_prices
                       if asset not in context.dont_buys 
                       and asset != context.index]
    pct_per_asset = 1.0 / len(eligible_assets)
    context.pct_per_asset = pct_per_asset
    for asset in eligible_assets:
        # Some assets might cause a key error due to being delisted 
        # or some other corporate event so we use a try/except statement
        try:
            if get_open_orders(sid=asset):
                continue
            order_target_percent(asset, pct_per_asset)
        except:
            log.warn("[Failed Order] asset = %s"%asset.symbol)
           
    
def hedge_portfolio(context, data):
    """
    This function places an order for "context.index" in the 
    amount required to neutralize the beta exposure of the portfolio.
    Note that additional leverage in the account is taken on, however,
    net market exposure is reduced.
    """
    factors = get_alphas_and_betas(context, data)
    beta_exposure = 0.0
    count = 0
    for asset in context.portfolio.positions:
        if asset in factors and asset != context.index:
            if not np.isnan(factors[asset].beta):
                beta_exposure += factors[asset].beta
                count += 1
    beta_hedge = -1.0 * beta_exposure / count
    dollar_amount = context.portfolio.portfolio_value * beta_hedge
    record(beta_hedge=beta_hedge)
    if not np.isnan(dollar_amount):
        order_target_value(context.index, dollar_amount)
    
def get_alphas_and_betas(context, data):
    """
    returns a dataframe of 'alpha' and 'beta' exposures 
    for each asset in the current universe.
    """
    prices = history(context.lookback, '1d', 'price', ffill=True)
    returns = prices.pct_change()[1:]
    index_returns = returns[context.index]
    factors = {}
    for asset in context.portfolio.positions:
        try:
            y = returns[asset]
            factors[asset] = linreg(index_returns, y)
        except:
            log.warn("[Failed Beta Calculation] asset = %s"%asset.symbol)
    return pd.DataFrame(factors, index=['alpha', 'beta'])
    
def linreg(x, y):
    # We add a constant so that we can also fit an intercept (alpha) to the model
    # This just adds a column of 1s to our data
    X = sm.add_constant(x)
    model = sm.OLS(y, X).fit()
    return model.params[0], model.params[1]

def handle_data(context, data):
    record(net_exposure=context.account.net_leverage,
           leverage=context.account.leverage)
    
There was a runtime error.
6 responses

Hey there,

I'll try to answer your questions here.

  1. It seems like you've found a bug in the algorithm. Open orders should be handled correctly and we don't here. I have made an issue to fix the algorithm and will get to it as soon as I have some free time. For an algorithm that handles open orders, please see this one.
  2. Same as 1., it seems like the algorithm does not correctly close out old positions, leading to the creep you recorded. I'll also try to fix this issue. For examples on how to fix these types of issues, please see Alisa's post here.
  3. The beta of a portfolio is the weighted mean of the betas of its positions, so the two approaches should be mathematically equivalent. It would be more efficient to compute the beta once between the portfolio returns and SPY, but since we've already computed all the position betas, it's more efficient to average them.

Hope this helps.

Thanks,
Delaney

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Delaney,

Thanks for the reply. Would the Algorithm I have cloned from Quantopian Lecture Series page update itself after you fixed the bugs ? Or I need to clone it again ?

Please also reply to this post once you or someone from Quantopian fix the bugs. I am very interested in seeing your approach.

Best,

You'd have to clone it again. I'll let you know once I get around to fixing the issues. I'm pretty busy these days so no guarantees about time, but I'll get to it as soon as possible.

@Nyan

Delaney requested I take a look at the Beta Hedging algo and see what I can do. Shockingly, we weren't closing out any old positions and we definitely weren't handling open orders correctly. I've taken care of those issues by iterating through the portfolio and removing ineligible securities, and canceling all open orders at the end of the day, respectively. You will notice a little bit of position creep over time, this is due to delisted securities that are still in our portfolio. I didn't want to make any assumptions about how to handle that case - so just be aware I suppose. We'll get this fixed algo up on the Lectures page as soon as possible, in the mean time I thought I'd post it here so you can carry on.

Clone Algorithm
8
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
import pandas as pd
import numpy as np
import statsmodels.api as sm
import math


def initialize(context):
    use_beta_hedging = True # Set to False to trade unhedged
    # Initialize a universe of liquid assets
    schedule_function(rebalance,
                      date_rule=date_rules.month_start(),
                      time_rule=time_rules.market_open())
    schedule_function(cancel_open_orders, date_rules.every_day(),
                      time_rules.market_close())
    
    if use_beta_hedging:
        # Call the beta hedging function one hour later to
        # make sure all of our orders have gone through.
        schedule_function(hedge_portfolio,
                          date_rule=date_rules.week_start(),
                          time_rule=time_rules.market_open(hours=1))
    # trading days used for beta calculation
    context.lookback = 150
    # Used to aviod purchasing any leveraged ETFs 
    context.dont_buys = security_lists.leveraged_etf_list
    # Current allocation per asset
    context.pct_per_asset = 0
    context.index = symbol('SPY')
    
def before_trading_start(context, data):
    # Number of stocks to find
    num_stocks = 100
    fundamental_df = get_fundamentals(
        query(
            # To add a metric. Start by typing "fundamentals."
            fundamentals.valuation_ratios.earning_yield,
            fundamentals.valuation.market_cap,
        )
        .filter(fundamentals.valuation.market_cap > 1e9)
        .order_by(fundamentals.valuation_ratios.earning_yield.desc())
        .limit(num_stocks)
    )
    context.eligible_assets = fundamental_df.columns
    
    update_universe(context.eligible_assets)
    
def rebalance(context, data):
    all_prices = history(1, '1d', 'price', ffill=True)
    
    #  Sell assets no longer eligible.
    for asset in context.portfolio.positions:
        if asset in data and asset not in context.eligible_assets:
            if get_open_orders(asset):
                continue
            order_target_percent(asset, 0)
            
    #  Buy eligible assets.
    context.pct_per_asset = 1.0 / len(context.eligible_assets)
    for asset in context.eligible_assets:
        if asset in data:
            if get_open_orders(asset):
                continue
            order_target_percent(asset, context.pct_per_asset)
           
    
def hedge_portfolio(context, data):
    """
    This function places an order for "context.index" in the 
    amount required to neutralize the beta exposure of the portfolio.
    Note that additional leverage in the account is taken on, however,
    net market exposure is reduced.
    """
    factors = get_alphas_and_betas(context, data)
    beta_exposure = 0.0
    count = 0
    for asset in context.portfolio.positions:
        if asset in factors and asset != context.index:
            if not np.isnan(factors[asset].beta):
                beta_exposure += factors[asset].beta
                count += 1
    beta_hedge = -1.0 * beta_exposure / count
    dollar_amount = context.portfolio.portfolio_value * beta_hedge
    record(beta_hedge=beta_hedge)
    if not np.isnan(dollar_amount):
        order_target_value(context.index, dollar_amount)
    
def get_alphas_and_betas(context, data):
    """
    returns a dataframe of 'alpha' and 'beta' exposures 
    for each asset in the current universe.
    """
    prices = history(context.lookback, '1d', 'price', ffill=True)
    returns = prices.pct_change()[1:]
    index_returns = returns[context.index]
    factors = {}
    for asset in context.portfolio.positions:
        try:
            y = returns[asset]
            factors[asset] = linreg(index_returns, y)
        except:
            log.warn("[Failed Beta Calculation] asset = %s"%asset.symbol)
    return pd.DataFrame(factors, index=['alpha', 'beta'])
    
def linreg(x, y):
    # We add a constant so that we can also fit an intercept (alpha) to the model
    # This just adds a column of 1s to our data
    X = sm.add_constant(x)
    model = sm.OLS(y, X).fit()
    return model.params[0], model.params[1]

def handle_data(context, data):
    record(net_exposure=context.account.net_leverage,
           leverage=context.account.leverage,
           num_pos=len(context.portfolio.positions), 
           oo=len(get_open_orders()))
           
def cancel_open_orders(context, data):
    for security in get_open_orders():
        for order in get_open_orders(security):
            cancel_order(order)
           
           
           
    
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thanks for making those changes James, the universe size was a huge bug. I plan to make a pass over the other lecture algos as well, they were tossed together and can be improved. I remember leaving open orders alone because if a market order didn't fill in a month, then the stock is probably not traded anymore.

@Nyan, to elaborate on your second question, open positions are always added to your universe, even if they are not included in the list passed to update_universe. Since this algorithm was not closing positions, the universe just got bigger. Thanks for bringing this to our attention, please pick apart the other lecture algorithms as well.

What's the correct way to replace history with data.history in this context (for quantopian 2 migration)?

data.history(context.portfolio.positions, 'price', context.lookback, '1d')  

seems like it's almost correct, but positions may not include context.index and nothing I tried could get a workable union here.
even cheating and trying to set a target of 0 on context.index if it's not in the portfolio positions. I also tried doing a separate
call for the index:
data.history(context.index, 'price', context.lookback, '1d')

which seems cleaner if less efficient, but the errors that come from that are difficult to interpret - they point at the caller, not the code.

I'm also assuming from the documentation that we don't need ffill=True any more, as it says price always is.