Back to Community
Why good daily backtests fail at minute backtests?

Hey all,

I've been noticing that with quite a few of the algos I've been making lately, they have had pretty solid results when backtested on daily data, but seem to just go belly up right away on the intraday backtests, and I had a few questions about it:

  1. Does this happen a lot to newbies?
  2. Are there any common mistakes or oversights that lead to this happening?
  3. Any general strategies for how to prevent it from happening?

If it helps at all, the strategies I've been working with lately have mostly been experiments using moving averages (simple and exponential) with pretty short periods. Would transaction costs be a part of it?

Any input would be greatly appreciated. Thanks in advance for any help!

14 responses

Hello Jacob,

Do you have a specific algorithm you could share as an example?

Grant

Jacob, I assume by "belly up" you mean that the algo has a runtime error and fails when you change the backtest from daily mode to minutely mode.

The answer is that yes, it happens a lot, to newbies and experts both. We try really hard to make algo writing easy, and we work hard to make the product "smart" so that you can easily get past obstacles and complexities of algo writing. This is a problem that we didn't anticipate as well as others. (The ones we anticipate well are ones that you never noticed!) We're working on a few different changes to make this one go away, in particular making sample algorithms all minutely-compatible and by building some smart "run-once-daily" functions that work in minute mode algos. The new history() command should help, too, by making trailing windows of data in minute mode more easily accessible.

Some general strategies:

  • Don't use brittle code like Booleans. It's tempting to say "if first run = True" but that fails in many ways. Find a more robust way to identify your current status.
  • Along the same lines, it's easy to just say "order 50 shares" but that is brittle in the face of market and algorithm status. You should think instead of "target position X" and then have helper code that assesses your current portfolio position and issues the commands that gets you to the new, target portfolio.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Dan,

Sorry, I should have been more precise. What I mean is that algorithms that earn solid returns (north of 200% for example) on daily backtests seem to virtually shoot straight down in minute backtests.

Grant, for an example, see the attached backtest.

Clone Algorithm
9
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
import math
sma = ta.SMA(timeperiod=2)

def initialize(context):
    context.stocks = {sid(8554), sid(26578), sid(24), sid(5061), sid(3735)}
    # context.min_notional = 1000.0
    # context.max_notional = 10000.0

def handle_data(context, data):
    for stock in context.stocks:
        sma_data = sma(data)
        stock_sma = sma_data[stock]
        
        price = data[stock].close_price
        shares = context.portfolio.positions[stock].amount
        cash = context.portfolio.cash
        
        if price > stock_sma and cash > price * 100.0: order(stock, 100)
        if price < stock_sma and shares > 0: order(stock, -100)
        
        # record(sec_price = price, EMA = stock_sma)
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

Hi Jacob,

The first thing I would check is your trading frequency and transaction costs. Unless you specify otherwise the Quantopian backtester applies (non-zero) defaults for transaction costs (slippage & commissions) to your strategy, if your strategy trades more frequently in minute mode than daily mode then you will incur more t-costs. This can very quickly overpower returns.

Best regards, Jess

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thanks Jacob,

Good example...no time now, but I'll try to have a look later. My sense is that it's gonna be non-trivial to replicate your algorithm exactly running on minute bars.

Grant

Jacob,

Here's an example you can tinker with. I tried to get TA-lib to work, with no success. Perhaps someone else can do a better job, and actually show the TA-lib implementation, with the history API.

Grant

Clone Algorithm
4
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
from pytz import timezone

def initialize(context):
    context.stocks = [sid(8554), sid(26578), sid(24), sid(5061),
                      sid(3735)]

def handle_data(context, data):
    
    # limit trading to once per day
    if not intradingwindow_check(context):
        return
    
    prices = history(2, '1d', 'price')
    
    for stock in context.stocks:
        
        stock_sma = prices[stock].mean(axis=1)
        
        price = data[stock].close_price
        shares = context.portfolio.positions[stock].amount
        cash = context.portfolio.cash
        
        if price > stock_sma and cash > price * 100.0:
            order(stock, 100)
        if price < stock_sma and shares > 0:
            order(stock, -100)

def intradingwindow_check(context):
    # Converts all time-zones into US EST to avoid confusion
    loc_dt = get_datetime().astimezone(timezone('US/Eastern'))
    if loc_dt.hour == 15 and loc_dt.minute == 0:
        return True
    else:
        return False
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

Hello Grant,

Here is the same result using TA-Lib.

P.

Clone Algorithm
0
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
from pytz import timezone
import talib

def initialize(context):
    context.stocks = [sid(8554), sid(26578), sid(24), sid(5061),
                      sid(3735)]

def handle_data(context, data):
    
    # limit trading to once per day
    if not intradingwindow_check(context):
        return
    
    prices = history(2, '1d', 'price')
    
    for stock in context.stocks:
        
        stock_sma = talib.SMA(prices[stock], timeperiod=2)[-1:]
        
        price = data[stock].close_price
        shares = context.portfolio.positions[stock].amount
        cash = context.portfolio.cash
        
        if price > stock_sma and cash > price * 100.0:
            order(stock, 100)
        if price < stock_sma and shares > 0:
            order(stock, -100)

def intradingwindow_check(context):
    # Converts all time-zones into US EST to avoid confusion
    loc_dt = get_datetime().astimezone(timezone('US/Eastern'))
    if loc_dt.hour == 15 and loc_dt.minute == 0:
        return True
    else:
        return False
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

Thanks Peter...Grant

Here's the result over a longer time frame, using:

stock_sma = prices[stock].mean(axis=1)  

Based on Peter's code above, I'd expect the result to be the same with TA-lib.

Grant

Clone Algorithm
4
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
from pytz import timezone

def initialize(context):
    context.stocks = [sid(8554), sid(26578), sid(24), sid(5061),
                      sid(3735)]

def handle_data(context, data):
    
    # limit trading to once per day
    if not intradingwindow_check(context):
        return
    
    prices = history(2, '1d', 'price')
    
    for stock in context.stocks:
        
        stock_sma = prices[stock].mean(axis=1)
        
        price = data[stock].close_price
        shares = context.portfolio.positions[stock].amount
        cash = context.portfolio.cash
        
        if price > stock_sma and cash > price * 100.0:
            order(stock, 100)
        if price < stock_sma and shares > 0:
            order(stock, -100)

def intradingwindow_check(context):
    # Converts all time-zones into US EST to avoid confusion
    loc_dt = get_datetime().astimezone(timezone('US/Eastern'))
    if loc_dt.hour == 15 and loc_dt.minute == 0:
        return True
    else:
        return False
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

Hello Grant,
Could you please explain if there is any difference in performance, accuracy .. etc. by using talib to generate the SMA instead of python's "mean" or vice versa?

Best regards,
Omar

Hello Omar,

I can't comment on performance of the TA-Lib but it has the advantage of being a quite extensive library of functions including candle pattern recognition. Implementing a moving average in Python/pandas is trivial but implementing the whole library is not. That's not to say people don't waste their time trying - see https://code.google.com/p/ultra-finance/wiki/pyTaLib

P.

Wow, thanks for all the help everyone. Grant – thanks for putting that algo & backtest together, looks like only trading once a day made it work much better. Transaction costs must have been what was killing me, like Jessica said.

Thanks Peter,
I thought it was as you said, but had to check to see if I missed something.

Hello Jacob,
Maybe you should check that your chosen stocks are not biased. I have tried a different choice and got very different result. I also tried "set_universe" but that backtest hanged.

Clone Algorithm
1
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
from pytz import timezone

def initialize(context):
    # context.stocks = [sid(8554), sid(26578), sid(24), sid(5061),
    #                   sid(3735)]
    
    # set_universe(universe.DollarVolumeUniverse(floor_percentile=98.0, ceiling_percentile=99.0))
    
    context.stocks = {sid(8554), sid(19920), sid(21785), sid(23921), sid(32381), sid(24705), sid(12915), sid(33655), sid(33748)
                      , sid(19656), sid(21555), sid(32275), sid(25593), sid(22445), sid(21652), sid(28074), sid(26807), sid(28320)
                      , sid(22463), sid(19655) , sid(32279)}

def handle_data(context, data):
    
    # limit trading to once per day
    if not intradingwindow_check(context):
        return
    
    prices = history(2, '1d', 'price')
    
    # for stock in context.stocks:
    for stock in data:
        
        stock_sma = prices[stock].mean(axis=1)
        
        price = data[stock].close_price
        shares = context.portfolio.positions[stock].amount
        cash = context.portfolio.cash
        
        if price > stock_sma and cash > price * 100.0:
            order(stock, 100)
        if price < stock_sma and shares > 0:
            order(stock, -100)

def intradingwindow_check(context):
    # Converts all time-zones into US EST to avoid confusion
    loc_dt = get_datetime().astimezone(timezone('US/Eastern'))
    if loc_dt.hour == 15 and loc_dt.minute == 0:
        return True
    else:
        return False
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.