Back to Community
Sector Rotation Strategy....

Hi All -

Some of you know that I am a professional money manager for a small firm, focused on retirement accounts. We've been operating in a very qualitative, fundamental analysis fashion and I'm trying to take the firm into the 21st century by introducing some algorithmic trading to our client accounts.

I had been working on a backtester/optimizer similar to Quantopian for algorithmic trading for several months (prior to my employment, even) when I was turned on to Quantopian and realized that alot of my work had already been done for me. I coded one of the algorithms I was working on and found substantial excess return from SPY in Quantopian but had trouble reproducing the backtest in my in-house backtester.

The algorithm is not a typical high-frequency trade algorithm but rather a quarterly sector rotation. I used dividend adjusted data to calculate a sector ranking, which I uploaded into Quantopian using Fetcher. I buy and hold the two sectors with the highest relative strength and then rotate after 63 trading days.

Here are the results....

15 responses

So the backtest in Quantopian is attached to this post and this is the output from my backtester --> https://dl.dropboxusercontent.com/s/gk3bzt7wmdc4bbm/Sector_Rotation_test.jpg

Clone Algorithm
119
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
#Use SPY 15 day MA to determine when to go to cash
#When not in cash buy and hold the most undervalued sectors

import datetime
import math
import numpy
import pandas

def clean_col(df):
    df = df.fillna(method='ffill')
    df = df[['rank', 'sid']]
    log.info(' \n %s % df.head()')
    return df

def initialize(context):
    #
    #
    #Variables for later
    context.day=None
    context.pauseUntil=0
    context.timestep=0
    context.margin_req=0 
    context.totalShorts=0
    context.totalLongs=0
    context.cash=0
    context.oldRanks=None
    context.timer=0
    context.holdNmanyDays=63
    context.liquid=1
          
    #
    #
    #Set constraints on borrowing
    context.pct_invested_threshold=1 #Set limit on percent invested (as a decimal)
    context.init_margin=.01 #Set initial margin requirement    
    context.maint_margin=.01 #Set the maintenance margin requirement   


    #
    #
    #Read universe
    context.SPY=sid(8554)
    context.Sectors=[sid(19662), sid(19659), sid(19656), sid(19661), sid(19655), sid(19658), sid(19660), sid(19654), sid(19657)]  
    
    #Read the rankings
    fetch_csv('https://dl.dropboxusercontent.com/s/7shk9xm1xi311at/Ranks_QuantopianTest.csv',
              date_column='Date',
              post_func=clean_col)

    
    

def handle_data(context, data):

    update_newFrame(context, data)

    #Set Liquidate to false
    Liquidate = 0
    
    #If its before the pause time
#    if context.timestep == 1:
#        nshares=context.cash / data[sid(8554)].price
#        order(sid(8554), nshares)
#        context.liquid=0
#        return
#    elif context.timestep == (context.pauseUntil-1):
#        Liquidate = 1
#        return

#    if context.timestep < context.pauseUntil: #context.PauseUntil:
#        return
    
    if context.timer >= context.holdNmanyDays: #reset the timer to 0 after the specified holding period
        #log.debug ("Liquidating and rotating sectors now because "+str(context.timer)+" indicates we have reached "+str(context.holdNmanyDays))
        Liquidate = 1
        context.timer=0
         

    #
    #
    #Its time to trade!
        
    if context.liquid == 1: #if we are in cash    
              
        ts1=None
        tr1=-100
        ts2=None
        tr2=-100
        ts3=None
        tr3=-100

        for stock in context.Sectors:
            if 'rank' in data[stock]:
                if data[stock]['rank']>tr1:
                    ts3=ts2
                    tr3=tr2
                    ts2=ts1
                    tr2=tr1
                    tr1=data[stock]['rank']
                    ts1=stock
 
                elif data[stock]['rank']>tr2:
                    ts3=ts2
                    tr3=tr2
                    tr2=data[stock]['rank']
                    ts2=stock
                
    #            elif data[stock]['rank']>tr3:
    #                tr3=data[stock]['rank']
    #                ts3=stock
        else:
            log.debug ("WARNING: rank not found for ticker "+str(stock))

                            
        if ts1!=None:
            nshares=context.cash*0.49/data[ts1].price
            log.debug ("Buying most undervalued stock "+str(ts1)+" with score of "+str(tr1))
            generate_order(ts1, nshares, context, data) 
            context.liquid=0 #we are not in cash

        if ts2!=None:
            nshares=context.cash*0.49/data[ts2].price
            log.debug ("Buying second most undervalued stock "+str(ts2)+" with score of "+str(tr2))
            generate_order(ts2, nshares, context, data) 
            context.liquid=0 #we are not in cash
        
#        if ts3!=None:
#            nshares=context.cash*0.3333/data[ts3].price
#            generate_order(ts3, nshares, context, data) 
#            context.liquid=0 #we are not in cash
#            log.debug ("Changed context.liquid to "+str(context.liquid))
            
    if Liquidate == 1:
        for i in context.Sectors:
            liquidate_position(i, context)
        liquidate_position(context.SPY, context)
        context.liquid=1
    

    
#
#
#    Supporting Functions
#
#

    

def update_newFrame(context, data):
    #
    context.cash = context.portfolio.cash
    context.portvalue = context.portfolio.positions_value
    context.totalShorts=0
    for sym in data.keys():
        if context.portfolio.positions[sym].amount < 0:
            context.totalShorts += (context.portfolio.positions[sym].amount * data[sym].price)
        else:
            context.totalLongs += (context.portfolio.positions[sym].amount * data[sym].price)

    update_portvals(context)
    
    #Handle assigning the timestep number (1 day is 1 timestep)
    if get_datetime().day <> context.day: #look for a new day
        context.day=get_datetime().day
        context.timestep += 1
        context.timer += 1
        #log.info ( "Cash: "+str(context.cash)+"; Margin Req: "+str(context.margin_req)+" Avail Cash:"+str(context.cash - context.margin_req) )
        if context.timestep>context.pauseUntil:
            if context.cash < context.margin_req: #check for margin calls daily
                generate_marginCall(context, data)
            
    
def update_portvals(context):
    #Update account information when this function is called
    context.total_equity = context.cash + context.portvalue
    context.pct_invested = (context.totalLongs-context.totalShorts) / context.total_equity
    context.pct_cash = context.cash / context.total_equity
    context.margin_req = abs(context.totalShorts * context.maint_margin)
    
        
def generate_order(sym, size, context, data):    
    #log.info("Call to generate_order")
    if size>0: #Opening a long position    

        #log.info("Buy long "+str(size)+" shares "+str(sym) )
        #log.info("Cash = "+str(context.cash)+"; Current Margin Req.="+str(context.margin_req) )                 

        #Is there enough available cash to buy the position
        if (context.cash-context.margin_req) < size * data[sym].price:
            #log.info("Trade canceled : Insufficient funds.")
            return

        #Deduct price from cash and add to portfolio value
        context.cash -= size * data[sym].price
        context.portvalue += size * data[sym].price
        context.totalLongs += size * data[sym].price
        update_portvals(context)

        #Abort the transaction if the percent invested is greater than the threshold
        #before slippage and commissions
        if context.pct_invested > context.pct_invested_threshold:
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalLongs -= size * data[sym].price
            update_portvals(context)

#            if size>100:
#                log.info("Re-generating order for "+str(size*context.pct_invested_threshold)+" instead of "+str(size))
#                generate_order(sym,size*context.pct_invested_threshold, context,data)
#            return
        
        #Abort the transaction if the investment would generate a margin call
        if context.cash < context.margin_req:
#            log.info("Invest would generate a margin call")
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalLongs -= size * data[sym].price
            update_portvals(context)
            return
    
        order(sym,size)

    else: #Opening a short position
        
        #log.info("Generating a short order for "+str(size)+" shares of "+str(sym)+" and context.cash="+str(context.cash)+" and context.margin_req="+str(context.margin_req) )
        #Is there at least enough available cash to cover the initial maintenance margin
        if (context.cash-context.margin_req) < abs(size * data[sym].price * context.init_margin):
            #log.info("Trade canceled")
            return
        
        #Deduct price from cash and add to portfolio value (note that size is negative)
        context.cash -= size * data[sym].price
        context.portvalue += size * data[sym].price
        context.totalShorts += size * data[sym].price
        update_portvals(context)
        
        #Abort the transaction if the percent invested is greater than the threshold
        #before slippage and commission
        if context.pct_invested > context.pct_invested_threshold:
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalShorts -= size * data[sym].price
            update_portvals(context)
            #log.info("Trade canceled")
            return
            
        #Abort the transaction if the investment would generate a margin call
        if context.cash < context.margin_req:
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalShorts -= size * data[sym].price
            update_portvals(context)
            #log.info("Trade canceled")
            return
        
        order(sym,size)
            
        
def generate_marginCall(context,data):
    #This function should be coded to address margin calls
    #log.info("Margin call")
    return(0)

def liquidate_position(sym,context):
    if context.portfolio.positions[sym].amount is not 0:
        log.debug("Liquidating "+str(context.portfolio.positions[sym].amount)+" shares of position "+str(sym))
        order(sym, -context.portfolio.positions[sym].amount)
        
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

As you can see, Quantopian indicates the algorithm yields a 150% premium over SPY and my backtester indicates barely 20% over the same time period. There are a number of discrepancies between the two that I am trying to rectify and I'm cataloging my results here for any insight and also to share my conclusions:

1) Quantopian handles dividends "realistically" meaning that price data is used in the backtester and dividend payments are credited to cash. My backtester expects the price data to be dividend adjusted. Right now, this is my leading suspicion for the difference in results and the implications, if this is the cause, would be AMAZING. Note that dividend adjusted data was used to rank the sectors and so the transaction log SHOULD (i'll get to this next) be the same. This would be mean that 'dividend adjusting' data significantly alters the performance, i.e. the adjustment is not accurate to reality. (ON A SIDE NOTE, I stand by prior posts I've made that dividend adjusted data must be used to code technical-based algos. I'll continue to use dividend adjusted data from my other data feed until it is available here.)

2) My backtester allows the user to perform the ranking using closing data and then purchase at the closing price. I understand this is a source of look-ahead bias, technically, but my feeling was that using the NEXT day's closing price was probably farther from the actual execution price I would get than the same day's closing price.... I'm going to change this in my program.

3) In my Quantopian algorithm I buy on day 0, liquidate the entire portfolio on day 63, and then buy new securities on day 64. In my backtester securities are liquidated and repurchased on day 63.

I can't find any other discrepancies and, as I said, points 2 and 3 shouldn't cause THAT much of a difference (the cost basis and sale price change by a day on either end). At this point, I'm thinking dividends are the issue.

Will keep you all posted.

Daniel, that's a fascinating algo. I spent 5-10 minutes looking at it. It looks very well coded. I couldn't see any of the "common" errors like over-borrowing or other things that falsely inflate returns.

Of your 3 possible sources of error, I'm most concerned about #2, simply because I know least about how you're generating your rankings. For instance, one possible way to err would be to say Q2 = April 1, but Q2 is composed of data from April 1 to June 30. When you're importing data you have to be extra careful that that your Q2 rank data is only used within the algorithm on July 1 or later.

I don't have a strong feeling either way about #1. #3 feels like a non-issue, but I agree it's possible.

Absolutely fascinating. I hope it holds up to more scrutiny!

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Me too! If my backtester is wrong and Quantopian is right then I actually have a viable trading strategy.

So... I'm still trying to deconstruct both backtesters to find the difference. One thing I noticed in my most recent backtest (below) on lines 3 and 4 of the backtester were transactions made on 04/07/2002 (a Sunday) and 07/04/2002 (Independence Day, markets are closed). This is probably not causing the differences obviously but any idea why trades were placed when the market was closed?

NOTE: Different dates are displayed in the position value tab. Maybe a bug in the transaction log?

Clone Algorithm
119
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
#Use SPY 15 day MA to determine when to go to cash
#When not in cash buy and hold the most undervalued sectors

import datetime
import math
import numpy
import pandas

def clean_col(df):
    df = df.fillna(method='ffill')
    df = df[['rank', 'sid']]
    log.info(' \n %s % df.head()')
    return df

def initialize(context):
    #
    #
    #Variables for later
    context.day=None
    context.pauseUntil=0
    context.timestep=0
    context.margin_req=0 
    context.totalShorts=0
    context.totalLongs=0
    context.cash=0
    context.oldRanks=None
    context.timer=0
    context.holdNmanyDays=63
    context.liquid=1
          
    #
    #
    #Set constraints on borrowing
    context.pct_invested_threshold=1 #Set limit on percent invested (as a decimal)
    context.init_margin=.01 #Set initial margin requirement    
    context.maint_margin=.01 #Set the maintenance margin requirement   


    #
    #
    #Read universe
    context.SPY=sid(8554)
    context.Sectors=[sid(19662), sid(19659), sid(19656), sid(19661), sid(19655), sid(19658), sid(19660), sid(19654), sid(19657)]  
    
    #Read the rankings
    fetch_csv('https://dl.dropboxusercontent.com/s/7shk9xm1xi311at/Ranks_QuantopianTest.csv',
              date_column='Date',
              post_func=clean_col)

    
    

def handle_data(context, data):

    update_newFrame(context, data)

    #Set Liquidate to false
    Liquidate = 0
    
    #If its before the pause time
#    if context.timestep == 1:
#        nshares=context.cash / data[sid(8554)].price
#        order(sid(8554), nshares)
#        context.liquid=0
#        return
#    elif context.timestep == (context.pauseUntil-1):
#        Liquidate = 1
#        return

#    if context.timestep < context.pauseUntil: #context.PauseUntil:
#        return
    
    if context.timer >= context.holdNmanyDays: #reset the timer to 0 after the specified holding period
        #log.debug ("Liquidating and rotating sectors now because "+str(context.timer)+" indicates we have reached "+str(context.holdNmanyDays))
        Liquidate = 1
        context.timer=0
         

    #
    #
    #Its time to trade!
        
    if context.liquid == 1: #if we are in cash    
              
        ts1=None
        tr1=-100
        ts2=None
        tr2=-100
        ts3=None
        tr3=-100

        for stock in context.Sectors:
            if 'rank' in data[stock]:
                if data[stock]['rank']>tr1:
                    ts3=ts2
                    tr3=tr2
                    ts2=ts1
                    tr2=tr1
                    tr1=data[stock]['rank']
                    ts1=stock
 
                elif data[stock]['rank']>tr2:
                    ts3=ts2
                    tr3=tr2
                    tr2=data[stock]['rank']
                    ts2=stock
                
    #            elif data[stock]['rank']>tr3:
    #                tr3=data[stock]['rank']
    #                ts3=stock
        else:
            log.debug ("WARNING: rank not found for ticker "+str(stock))

                            
        if ts1!=None:
            nshares=context.cash*0.99/data[ts1].price
            log.debug ("Buying most undervalued stock "+str(ts1)+" with score of "+str(tr1))
            generate_order(ts1, nshares, context, data) 
            context.liquid=0 #we are not in cash

#        if ts2!=None:
#            nshares=context.cash*0.49/data[ts2].price
#            log.debug ("Buying second most undervalued stock "+str(ts2)+" with score of "+str(tr2))
#            generate_order(ts2, nshares, context, data) 
#            context.liquid=0 #we are not in cash
        
#        if ts3!=None:
#            nshares=context.cash*0.3333/data[ts3].price
#            generate_order(ts3, nshares, context, data) 
#            context.liquid=0 #we are not in cash
#            log.debug ("Changed context.liquid to "+str(context.liquid))
            
    if Liquidate == 1:
        for i in context.Sectors:
            liquidate_position(i, context)
        liquidate_position(context.SPY, context)
        context.liquid=1
    

    
#
#
#    Supporting Functions
#
#

    

def update_newFrame(context, data):
    #
    context.cash = context.portfolio.cash
    context.portvalue = context.portfolio.positions_value
    context.totalShorts=0
    for sym in data.keys():
        if context.portfolio.positions[sym].amount < 0:
            context.totalShorts += (context.portfolio.positions[sym].amount * data[sym].price)
        else:
            context.totalLongs += (context.portfolio.positions[sym].amount * data[sym].price)

    update_portvals(context)
    
    #Handle assigning the timestep number (1 day is 1 timestep)
    if get_datetime().day <> context.day: #look for a new day
        context.day=get_datetime().day
        context.timestep += 1
        context.timer += 1
        #log.info ( "Cash: "+str(context.cash)+"; Margin Req: "+str(context.margin_req)+" Avail Cash:"+str(context.cash - context.margin_req) )
        if context.timestep>context.pauseUntil:
            if context.cash < context.margin_req: #check for margin calls daily
                generate_marginCall(context, data)
            
    
def update_portvals(context):
    #Update account information when this function is called
    context.total_equity = context.cash + context.portvalue
    context.pct_invested = (context.totalLongs-context.totalShorts) / context.total_equity
    context.pct_cash = context.cash / context.total_equity
    context.margin_req = abs(context.totalShorts * context.maint_margin)
    
        
def generate_order(sym, size, context, data):    
    #log.info("Call to generate_order")
    if size>0: #Opening a long position    

        #log.info("Buy long "+str(size)+" shares "+str(sym) )
        #log.info("Cash = "+str(context.cash)+"; Current Margin Req.="+str(context.margin_req) )                 

        #Is there enough available cash to buy the position
        if (context.cash-context.margin_req) < size * data[sym].price:
            #log.info("Trade canceled : Insufficient funds.")
            return

        #Deduct price from cash and add to portfolio value
        context.cash -= size * data[sym].price
        context.portvalue += size * data[sym].price
        context.totalLongs += size * data[sym].price
        update_portvals(context)

        #Abort the transaction if the percent invested is greater than the threshold
        #before slippage and commissions
        if context.pct_invested > context.pct_invested_threshold:
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalLongs -= size * data[sym].price
            update_portvals(context)

#            if size>100:
#                log.info("Re-generating order for "+str(size*context.pct_invested_threshold)+" instead of "+str(size))
#                generate_order(sym,size*context.pct_invested_threshold, context,data)
#            return
        
        #Abort the transaction if the investment would generate a margin call
        if context.cash < context.margin_req:
#            log.info("Invest would generate a margin call")
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalLongs -= size * data[sym].price
            update_portvals(context)
            return
    
        order(sym,size)

    else: #Opening a short position
        
        #log.info("Generating a short order for "+str(size)+" shares of "+str(sym)+" and context.cash="+str(context.cash)+" and context.margin_req="+str(context.margin_req) )
        #Is there at least enough available cash to cover the initial maintenance margin
        if (context.cash-context.margin_req) < abs(size * data[sym].price * context.init_margin):
            #log.info("Trade canceled")
            return
        
        #Deduct price from cash and add to portfolio value (note that size is negative)
        context.cash -= size * data[sym].price
        context.portvalue += size * data[sym].price
        context.totalShorts += size * data[sym].price
        update_portvals(context)
        
        #Abort the transaction if the percent invested is greater than the threshold
        #before slippage and commission
        if context.pct_invested > context.pct_invested_threshold:
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalShorts -= size * data[sym].price
            update_portvals(context)
            #log.info("Trade canceled")
            return
            
        #Abort the transaction if the investment would generate a margin call
        if context.cash < context.margin_req:
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalShorts -= size * data[sym].price
            update_portvals(context)
            #log.info("Trade canceled")
            return
        
        order(sym,size)
            
        
def generate_marginCall(context,data):
    #This function should be coded to address margin calls
    #log.info("Margin call")
    return(0)

def liquidate_position(sym,context):
    if context.portfolio.positions[sym].amount is not 0:
        log.debug("Liquidating "+str(context.portfolio.positions[sym].amount)+" shares of position "+str(sym))
        order(sym, -context.portfolio.positions[sym].amount)
        
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

Gah. I think you're running into the same bug you reported before, cropped up in a different screen.

When I clone and run your algo, the log output looks right.

2002-01-03handle_data:117DEBUGBuying most undervalued stock Security(19659 [XLP]) with score of -0.9473006  
2002-04-05handle_data:117DEBUGBuying most undervalued stock Security(19658 [XLK]) with score of -0.9180112  
2002-07-05handle_data:117DEBUGBuying most undervalued stock Security(19658 [XLK]) with score of -0.9425241  

What's happening in the transaction tab is that we're taking midnight UTC (which is the timestamp on minute bars) and casting that incorrectly to Eastern (which is why the transactions show at 1900 or 2000, depending on daylight savings time). The trades are happening on the "right" day, but they're being displayed in the transaction tab on the previous day. I don't know how I missed that when I was debugging it the last time!

Bottom line: that's a UI bug that we will fix, but the backtester itself is trading on the right day.

No worries - I figured that this wasn't the cause of the difference in returns. And I appreciate your response earlier regarding your words of caution about look ahead bias. I've been quite careful about this and do not think this is the issue. I'm reasonably certain that the issue is using dividend adjusted data and I think you folks are doing it correctly. I'm trying to find an example, though, of a substantial difference in returns between using dividend adjusted data (and only looking at the price difference) and using price data with the actual dividends credited to cash..... I'll report back.

Got something else of interest, possibly more fundamental....

So - let's forget about the algorithm and just look at the benchmark, SPY in the backtest I post below, which runs from Jan 3, 2002 to July 12, 2013.

On Jan 3, 2002 the closing price of SPY was 116.84 and on July 12, 2013 the closing price of SPY was 167.51 - a return of 43.37%. The 43.37% (Yahoo data) is very close to the value calculated by Quantopian, 45.5%. However, SPY also paid $26.997 per share in dividends over the same time period, meaning that the actual buy-and-hold return of SPY would be 66.47%! And this is without reinvesting dividends!

If we use the dividend adjusted price of $93.44 for Jan 3, 2002 we get a return of 79.3% up to July 12, 2013 (this would be the approximate return if dividends were reinvested).

Have I miscalculated here? If not, my recommendation moving forward would be 1) at the very least have the benchmark include it's dividends in the calculated percent return, the way it is included in the algo and, eventually, 2) provide the user the ability to reinvest dividends automatically (both in the benchmark and the algo).

Clone Algorithm
39
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
#Use SPY 15 day MA to determine when to go to cash
#When not in cash buy and hold the most undervalued sectors

import datetime
import math
import numpy
import pandas

def initialize(context):
    #
    #
    #Variables for later
    context.day=None
    context.pauseUntil=0
    context.timestep=0
    context.margin_req=0 
    context.totalShorts=0
    context.totalLongs=0
    context.cash=0
    context.oldRanks=None
    context.timer=0
    context.holdNmanyDays=63
    context.liquid=1
          
    #
    #
    #Set constraints on borrowing
    context.pct_invested_threshold=1 #Set limit on percent invested (as a decimal)
    context.init_margin=.01 #Set initial margin requirement    
    context.maint_margin=.01 #Set the maintenance margin requirement   


    #
    #
    #Read universe
    context.SPY=sid(8554)
    context.Sectors=[sid(19662), sid(19659), sid(19656), sid(19661), sid(19655), sid(19658), sid(19660), sid(19654), sid(19657)]  
    
 
    

def handle_data(context, data):

    update_newFrame(context,data)
    
    if context.timestep == 1:
        log.debug("Timestep: "+str(context.timestep)+" buying XLY")
        nshares=context.portfolio.cash*0.99/data[sid(19662)].price
        generate_order(sid(19660), nshares, context, data)
        

    
#
#
#    Supporting Functions
#
#

    

def update_newFrame(context, data):
    #
    context.cash = context.portfolio.cash
    context.portvalue = context.portfolio.positions_value
    context.totalShorts=0
    for sym in data.keys():
        if context.portfolio.positions[sym].amount < 0:
            context.totalShorts += (context.portfolio.positions[sym].amount * data[sym].price)
        else:
            context.totalLongs += (context.portfolio.positions[sym].amount * data[sym].price)

    update_portvals(context)
    
    #Handle assigning the timestep number (1 day is 1 timestep)
    if get_datetime().day <> context.day: #look for a new day
        context.day=get_datetime().day
        context.timestep += 1
        context.timer += 1
        #log.info ( "Cash: "+str(context.cash)+"; Margin Req: "+str(context.margin_req)+" Avail Cash:"+str(context.cash - context.margin_req) )
        if context.timestep>context.pauseUntil:
            if context.cash < context.margin_req: #check for margin calls daily
                generate_marginCall(context, data)
            
    
def update_portvals(context):
    #Update account information when this function is called
    context.total_equity = context.cash + context.portvalue
    context.pct_invested = (context.totalLongs-context.totalShorts) / context.total_equity
    context.pct_cash = context.cash / context.total_equity
    context.margin_req = abs(context.totalShorts * context.maint_margin)
    
        
def generate_order(sym, size, context, data):    
    #log.info("Call to generate_order")
    if size>0: #Opening a long position    

        #log.info("Buy long "+str(size)+" shares "+str(sym) )
        #log.info("Cash = "+str(context.cash)+"; Current Margin Req.="+str(context.margin_req) )                 

        #Is there enough available cash to buy the position
        if (context.cash-context.margin_req) < size * data[sym].price:
            #log.info("Trade canceled : Insufficient funds.")
            return

        #Deduct price from cash and add to portfolio value
        context.cash -= size * data[sym].price
        context.portvalue += size * data[sym].price
        context.totalLongs += size * data[sym].price
        update_portvals(context)

        #Abort the transaction if the percent invested is greater than the threshold
        #before slippage and commissions
        if context.pct_invested > context.pct_invested_threshold:
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalLongs -= size * data[sym].price
            update_portvals(context)

#            if size>100:
#                log.info("Re-generating order for "+str(size*context.pct_invested_threshold)+" instead of "+str(size))
#                generate_order(sym,size*context.pct_invested_threshold, context,data)
#            return
        
        #Abort the transaction if the investment would generate a margin call
        if context.cash < context.margin_req:
#            log.info("Invest would generate a margin call")
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalLongs -= size * data[sym].price
            update_portvals(context)
            return
    
        order(sym,size)

    else: #Opening a short position
        
        #log.info("Generating a short order for "+str(size)+" shares of "+str(sym)+" and context.cash="+str(context.cash)+" and context.margin_req="+str(context.margin_req) )
        #Is there at least enough available cash to cover the initial maintenance margin
        if (context.cash-context.margin_req) < abs(size * data[sym].price * context.init_margin):
            #log.info("Trade canceled")
            return
        
        #Deduct price from cash and add to portfolio value (note that size is negative)
        context.cash -= size * data[sym].price
        context.portvalue += size * data[sym].price
        context.totalShorts += size * data[sym].price
        update_portvals(context)
        
        #Abort the transaction if the percent invested is greater than the threshold
        #before slippage and commission
        if context.pct_invested > context.pct_invested_threshold:
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalShorts -= size * data[sym].price
            update_portvals(context)
            #log.info("Trade canceled")
            return
            
        #Abort the transaction if the investment would generate a margin call
        if context.cash < context.margin_req:
            context.cash += size * data[sym].price
            context.portvalue -= size * data[sym].price
            context.totalShorts -= size * data[sym].price
            update_portvals(context)
            #log.info("Trade canceled")
            return
        
        order(sym,size)
            
        
def generate_marginCall(context,data):
    #This function should be coded to address margin calls
    #log.info("Margin call")
    return(0)

def liquidate_position(sym,context):
    if context.portfolio.positions[sym].amount is not 0:
        log.debug("Liquidating "+str(context.portfolio.positions[sym].amount)+" shares of position "+str(sym))
        order(sym, -context.portfolio.positions[sym].amount)
        
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

Daniel,
Thanks for this very well written strategy. It gave me a lot of inspiration.
For the reinvestment problem, I think the best way to do it is to reinvest the dividend received manually. Quantopian will look at the dividend as cash increase in your portfolio, and you can access this number as "context.portfolio.cash"

Here is a very short sample of buying SPY and spending all the dividend to re-invest in SPY.

Clone Algorithm
11
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
import numpy as np
import datetime

def initialize(context):

    # context.stocks = [sid(19662),sid(19659),sid(19655),sid(19656),sid(19661),sid(19657),sid(19654),sid(19658),sid(19660), sid(8554)]
    context.stocks = [sid(8554)]
    context.m = len(context.stocks)
    context.b_t = np.ones(context.m) / context.m
    context.eps = 1  #change epsilon here
    context.init = True
    context.counter = 0
    
    set_slippage(slippage.VolumeShareSlippage(volume_limit=0.25, price_impact=0))
    set_commission(commission.PerShare(cost=0))
    
def handle_data(context, data):    
    context.counter += 1
    if context.counter == 1:
        buyAndHold(context, data)
    else:
        ReInvest(context, data)

def buyAndHold(context, data):
    for i, stock in enumerate(context.stocks):
        prices = np.zeros_like(context.b_t)
        desired_amount = np.zeros_like(prices)
        prices[i] = data[stock].price
        desired_amount[i] = np.round(context.portfolio.starting_cash / context.m / prices[i])
        log.info("Bought {ticker} @ {price} for {amount} shares".format(ticker = stock, price = prices[i], amount = desired_amount[i]))
        order(stock, desired_amount[i])

def ReInvest(context, data):
    for i, stock in enumerate(context.stocks):
        prices = np.zeros_like(context.b_t)
        desired_amount = np.zeros_like(prices)
        prices[i] = data[stock].price
        desired_amount[i] = np.round(context.portfolio.cash / context.m / prices[i])
        if desired_amount[i] >0:
            log.info("Bought {ticker} @ {price} for {amount} shares".format(ticker = stock, price = prices[i], amount = desired_amount[i]))
            order(stock, desired_amount[i])
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

@Jiaming - It is interesting that the reinvested dividends beats SPY by 40% when dividends are reinvested but let me be clear - the ~43% return of the benchmark does not include dividends AT ALL! That is the price return only! This is misleading at best and possibly a big oversight.

I think Daniel's right. We covered it over in this thread. Just putting the link here in case someone else is following this thread.

I'd like not only to clone your algorithm (this is my first visit/attempt - and I've done it!), but also to expand/change the ETF universe. I have no clue how to work about this idea (using Fetcher, create my own CSV file). Could you please give me some hints? Thank you.

Andrzejek,

You can include up to 100 specific securities in an algorithm. For example:

context.stocks = [(your list of securities goes here)]

Grant

Yes, but as long as they are not included in the ("Fetch") CSV file, they are omitted in the analysis.This is my understanding, maybe I'm wrong. I ran my clone algorithm several times (with context.stocks expanded) and only "XL_" ETFs were included in the final rapport. Thank you for your response.

@Andrzejek - thanks for taking an interest in my algorithm! Unfortunately the ranking system that I used is not open source. It is a proprietary algorithm developed by me on my firm's behalf. There may come a point where we publish it but that decision would have to be approved by our CEO and we would probably publish it in a formal white paper (I promise I'll post a link on Quantopian if that ever happens).

There is an issue with the algorithm - or more specifically just a concern I have... we have been using in-house software to test the algorithm and have been getting slightly different results than we get in Quantopian. I am working on figuring out the difference between the two. The biggest road block at the moment is an issue with fetcher's NAN filling routine. (https://quantopian.com/posts/fetcher-problems and https://quantopian.com/posts/help-with-fetcher-and-filling-nan-values) Once I resolve this issue I can use Quantopian to read my trade log from my other backtester and verify that the algo generates similar if not identical returns in both programs.