Back to Community
Backtest results different in minute and daily mode

Hi everybody!

I'm new to Quantopian and I'm noticing something really strange when testing my first algorithm. It seems to work perfectly when tested in minute mode, though in daily mode the results are a little bit different - slightly larger drawdowns and slightly lesser returns.

The strategy should only trade once per day (at the close), and am using the following code to restrict the trading:

exchange_time = get_datetime().astimezone(pytz.timezone('US/Eastern'))  
exchange_minute = (exchange_time.hour*60)+exchange_time.minute-570  
if (exchange_minute == 390):  

The only data I use is daily data from history:
closes = history(100, '1d', 'price') lows = history(100, '1d', 'low') highs = history(100, '1d', 'high')

Anyone faced the same problem or have an idea of what could be going on?

Any help would be awesome ~

Thanks!

13 responses

To bump and slightly restate the question - when does daily mode trade in the day? Does it trade at the close, and if so, how does it factor slippage? Does the order roll over into after-hours or the next mornings open?

Hey Param,

Trades in backtesting mode are filled, at the fastest, the next bar. So in this case (daily mode) of creating an order on, let's say, May 6th, it'll get filled at the close price of May 7th. This might be why you're experiencing different results from daily versus minute mode since you're getting a difference of a full day in price in daily mode versus a difference of a few minutes in minutely mode. As for slippage, there's a default slippage model applied to your order which uses the VolumeShareSlippage model.

And when you use 'if (exchange_minute == 390)', the order gets carried over till the next morning because of the next-bar-execution behavior of the backtester so using '380' will get you something closer to what you're looking for. Alisa did an amazing job of addressing these issues on this topic (https://www.quantopian.com/posts/why-cancel-orders-at-end-of-day) so if you'd like to find out more, feel free to check out the link.

Let me know if you have any more questions.

Seong

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thanks so much for the reply, Seong.

I don't understand why the daily mode wouldn't trade the same day as the signal. It seems like this one-day delay would be immensely problematic for most algorithms. I find it almost impossible to experiment in minute mode as backtesting is painfully slow (several hours for a 10 year backtest), and is extremely redundant given that I only want to trade at once-daily frequency. Daily mode is typically around 3-5 minutes.

Are you guys working toward a way to execute trades the same day in daily mode? Is there a workaround in the meantime that would allow me to use the close price from the day of the signal? It just feels extremely inefficient to have to test an at-the-close strategy on every minute-bar of the day.

Hello Param,

Anony Mole posted a solution to this problem, https://www.quantopian.com/posts/trade-at-the-open-slippage-model. He takes advantage of the documented feature of applying a custom slippage model (https://www.quantopian.com/help#sample-custom-slippage). I've attached an example.

There might be a way to allow you "to use the close price from the day of the signal" when running an algo on daily bars. I'll see if I can sort it out. It would amount to shifting the 'trade_bar' in the process_order method back one bar (i.e. the day the order was submitted). One then has to be cautious of look-ahead bias, since when handle_data is called your algo will have access to the closing price which will be used as the execution price.

Grant

Clone Algorithm
31
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
def initialize(context):
    context.spy = sid(8554)
    
    set_slippage(TradeAtTheOpenSlippageModel(0.1))
 
def handle_data(context, data):

    order(context.spy, 1)   
    
########################################################  
# Slippage model to trade at the open or at a fraction of the open - close range.  
class TradeAtTheOpenSlippageModel(slippage.SlippageModel):  
    '''Class for slippage model to allow trading at the open  
       or at a fraction of the open to close range.  
    '''  
    # Constructor, self and fraction of the open to close range to add (subtract)  
    #   from the open to model executions more optimistically  
    def __init__(self, fractionOfOpenCloseRange):

        # Store the percent of open - close range to take as the execution price  
        self.fractionOfOpenCloseRange = fractionOfOpenCloseRange

    def process_order(self, trade_bar, order):  
        openPrice = trade_bar.open_price  
        closePrice = trade_bar.price  
        ocRange = closePrice - openPrice  
        ocRange = ocRange * self.fractionOfOpenCloseRange  
        if (ocRange != 0.0):  
            targetExecutionPrice = openPrice + ocRange  
        else:  
            targetExecutionPrice = openPrice  
        log.info('\nOrder:{0} open:{1} close:{2} exec:{3} side:{4}'.format(  
            trade_bar.sid.symbol, openPrice, closePrice, targetExecutionPrice, order.direction))

        # Create the transaction using the new price we've calculated.  
        return slippage.create_transaction(  
            trade_bar,  
            order,  
            targetExecutionPrice,  
            order.amount  
        )  
There was a runtime error.

Hi Param,

Please disregard my post above, since when I look at the transaction details for the algo, it appears that Anony's custom slippage model is not working (or I have not applied it correctly).

Grant

Param,

Here's the fixed version. I had to add:

set_commission(commission.PerTrade(cost=0.0))  

Otherwise, there is a 0.03 commission difference that shows up in the Transaction Details report. For the attached backtest, you can see directly that Anony's model works.

Grant

Clone Algorithm
1
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
def initialize(context):
    context.spy = sid(8554)
    
    set_slippage(TradeAtTheOpenSlippageModel(0.2))
    set_commission(commission.PerTrade(cost=0.0))
 
def handle_data(context, data):

    order(context.spy, 1)   
    
########################################################  
# Slippage model to trade at the open or at a fraction of the open - close range.  
class TradeAtTheOpenSlippageModel(slippage.SlippageModel):  
    '''Class for slippage model to allow trading at the open  
       or at a fraction of the open to close range.  
    '''  
    # Constructor, self and fraction of the open to close range to add (subtract)  
    #   from the open to model executions more optimistically  
    def __init__(self, fractionOfOpenCloseRange):

        # Store the percent of open - close range to take as the execution price  
        self.fractionOfOpenCloseRange = fractionOfOpenCloseRange

    def process_order(self, trade_bar, order):  
        openPrice = trade_bar.open_price  
        closePrice = trade_bar.price  
        ocRange = closePrice - openPrice  
        ocRange = ocRange * self.fractionOfOpenCloseRange  
        if (ocRange != 0.0):  
            targetExecutionPrice = openPrice + ocRange  
        else:  
            targetExecutionPrice = openPrice  
        log.info('\nOrder:{0} open:{1} close:{2} exec:{3} side:{4}'.format(  
            trade_bar.sid.symbol, openPrice, closePrice, targetExecutionPrice, order.direction))

        # Create the transaction using the new price we've calculated.  
        return slippage.create_transaction(  
            trade_bar,  
            order,  
            targetExecutionPrice,  
            order.amount  
        )
There was a runtime error.

Hi Param,

The attached code should do what you want. You should be able to run on daily bars, and buy at the closing price, thus simulating a MOC order. Note that following Anony's example of a simple linear model for the price variation over the day, you can buy close to the close (by setting fractionOfOpenCloseRange=0.1). Let me know if you find bugs, or if it doesn't meet your needs. Again, please be aware that there is the potential for look-ahead bias. Also, from the standpoint of good coding practice, my use of globals may pose a risk. Perhaps some Python gurus out there know how to do a better coding job.

Here's the log output, which you can compare to the Transaction Details tab (after running the algo yourself):

2014-10-01PRINTSPY prior open/close: 196.7/194.34  
2014-10-01PRINTBND prior open/close: 81.97/82.12  
2014-10-02process_order:52INFO Order:SPY open:196.7 close:194.34 exec:194.34 side:1.0  
2014-10-02process_order:52INFO Order:BND open:81.97 close:82.12 exec:82.12 side:1.0  
2014-10-02PRINTSPY prior open/close: 194.18/194.39  
2014-10-02PRINTBND prior open/close: 82.18/82.05  
2014-10-03process_order:52INFO Order:SPY open:194.18 close:194.39 exec:194.39 side:1.0  
2014-10-03process_order:52INFO Order:BND open:82.18 close:82.05 exec:82.05 side:1.0  
2014-10-03PRINTSPY prior open/close: 195.68/196.53  
2014-10-03PRINTBND prior open/close: 81.97/82.03  

You can see that the orders are executing as MOC with daily bars, and not using the next day's close for execution, which is the standard slippage.

Grant

Clone Algorithm
31
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
priorOpen = None
priorClose = None

def initialize(context):
    context.stocks = [sid(8554),sid(33652)]
    
    set_slippage(TradeAtTheCloseSlippageModel(priorOpen,priorClose,0.0))
    set_commission(commission.PerTrade(cost=0.0))
 
def handle_data(context, data):
    
    for stock in context.stocks:
        order(stock,1)
    
    global priorOpen
    global priorClose
    
    priorOpen = {}
    priorClose = {}
    
    for sid in data:
        priorOpen[sid] = data[sid].open_price
        priorClose[sid] = data[sid].close_price
        print sid.symbol+' prior open/close: '+str(priorOpen[sid])+'/'+str(priorClose[sid])
    
########################################################  
# Slippage model to trade at the prior close or at a fraction of the prior open - close range.  
class TradeAtTheCloseSlippageModel(slippage.SlippageModel):  
    '''Class for slippage model to allow trading at the prior close  
       or at a fraction of the prior open to close range.  
    '''  
    # Constructor, self and fraction of the prior open to close range to add (subtract)  
    # from the prior open to model executions more optimistically  
    def __init__(self, priorOpen,priorClose,fractionOfOpenCloseRange):

        # Store the percent of prior open - close range to take as the execution price  
        self.priorOpen = priorOpen
        self.priorClose = priorClose
        self.fractionOfOpenCloseRange = fractionOfOpenCloseRange

    def process_order(self, trade_bar, order):
        
        openPrice = priorOpen[trade_bar.sid]  
        closePrice = priorClose[trade_bar.sid]  
        ocRange = closePrice - openPrice  
        ocRange = ocRange * self.fractionOfOpenCloseRange  
        if (ocRange != 0.0):  
            targetExecutionPrice = closePrice - ocRange  
        else:  
            targetExecutionPrice = closePrice  
        log.info('\nOrder:{0} open:{1} close:{2} exec:{3} side:{4}'.format(  
            trade_bar.sid.symbol, openPrice, closePrice, targetExecutionPrice, order.direction))

        # Create the transaction using the new price we've calculated.  
        return slippage.create_transaction(  
            trade_bar,  
            order,  
            targetExecutionPrice,  
            order.amount  
        )  
There was a runtime error.

Grant this is awesome

Thanks so much, Grant! This is an amazingly elegant solution and will certainly save me a lot of time!

You're welcome. If you have more questions or insights, just post here. --Grant

@Grant
Whew :O :D :)

@Param, @Fawce
Param said: "I don't understand why the daily mode wouldn't trade the same day as the signal"
Agreed. Long ago transactions were confirmed while still on the phone, today websites fill/confirm right away, not based on the price 24 hours later.
Daily mode fill next day is unreal.

Requesting Quantopian provide us with a switch to fill orders in Daily mode right away.
Couple of options maybe:
1. Reproduce what Grant has shown here with one line we invoke, or
2. The switch makes Daily price the 3:59 prices rather than close and fill at 4:00 same day.

Param said: "I find it almost impossible to experiment in minute mode as backtesting is painfully slow"
You can say that again.
My algo takes 45 seconds in Daily mode and about 3 hours in Minute mode (2009-01-01 to present).
Edit: Around 20 minutes with the exchange_minute code above.

Requesting Quantopian consider providing us with a switch to feed to handle_data() in Minute mode only the minutes a user specifies.
The option is for those who are just going to be discarding the other minutes' data.
Pulling them from the data provider, tasking the processors and transmitting them no more will save energy and time.
Roughly 1/390th the effort, around 1/4 of 1 percent.
Please make the consideration of that a priority, ask the appropriate person whether it is technically feasible.

Another twist here is that one needs to consider the definition of an OHLC daily bar. If I understand correctly, Quantopian (or their vendor) is doing this:

O: the price of the first trade of the day, regardless of when it occurs
H: the highest price of the day
L: the lowest price of the day
C: the last price of the day, regardless of when it occurs

So, for thinly traded securities, there is another caveat for the code I posted above, since the close for a given security may not occur at the market close. So, even if the algo uses opening prices, and you are assuming MOC orders placed shortly after the open, in reality, they may not get filled the same day. This mistake could be avoided if datetime stamps were provided for the individual OHLC values. The other approach would be to run a diagnostic algo on minute bars to check the trading frequencies and volumes of the relevant securities, to see that the MOC orders are realistic.

A further extension (/abuse) of the slippage model would be to figure out how to place multiple orders in the same day but at different effective times (e.g. close to the open & close to the close), using daily bars (straightforward, I think). This probably is not the best approach, but (painfully) calling handle_data 390 times a day may be overkill for the type of trading that is of interest to some folks here, where the capability to detect small minute-to-minute differences is irrelevant. The relevant timescales are days, weeks, and months for many, I suspect.

Grant

Hi,

I know it's a long time, but is the daily execution issue (order submitted today but actually executed next day) finally corrected yet in Quantopian and Zipline? or is there any plan at all to correct it?

Thanks,
ya