Back to Community
Futures data.history problem for JY & JE

I am trying to calculate the volatility (standard deviation, ATR) for futures contracts. However I ran into a snag for JY & JE in 2012. The problem is: the results from data.history in a backtest show every day having the same value for a stretch of time, this occurs for over 100 days in the summer of 2012. The actual price did not look like this. It appears that the date is getting truncated to 3 significant digits. What is interesting is that this only happens for days earlier than t-1.

Here is an example where on 2012-09-24 I fetch the last 100 days data for the continuous future JY. (See the attached backtest)

2012-09-24 06:31  PRINT                               close      high       low     price  
2012-09-18 00:00:00+00:00  0.012000  0.012000  0.012000  0.012000  
2012-09-19 00:00:00+00:00  0.012000  0.012000  0.012000  0.012000  
2012-09-20 00:00:00+00:00  0.012000  0.012000  0.012000  0.012000  
2012-09-21 00:00:00+00:00  0.012000  0.012000  0.012000  0.012000  
2012-09-24 00:00:00+00:00  0.012827  0.012835  0.012803  0.012827  
2012-09-24 06:31  PRINT ATRs: {'JE': 0.0, 'JY': 0.0}  
2012-09-24 06:31  PRINT SDs: {'JE': 8.716877585302702e-18, 'JY': 8.716877585302702e-18}  

I take the data from the last 100 days and calculate the ATR and Standard deviation, for many weeks these values are effectively 0.
I remember reading on one post on futures that the values were only displayed to three significant digits, but this was only a display issue. That doesn't seem to be the case as any math on t-1 values shows that there are only three significant digits in there.

Does anyone know why these values are incorrect? If you plot the values of JY in Research you do not get this.

Thanks for the help.

Clone Algorithm
Total Returns
Max Drawdown
Benchmark Returns
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
from quantopian.algorithm import order_optimal_portfolio
import quantopian.experimental.optimize as opt

month_idx = 0

ATR_fields=['high', 'low', 'close', 'price']
ATR_LEN = 100

available_futures = {
    'debug': {
        'jpy': 'JY', # good
        'jpy_emini': 'JE', # good

def make_futures(context):
    # populates context.futures, by flattening available_futures
    context.futures = []
    #for sector in available_futures:
    for sector in ["debug"]:
        for future in available_futures[sector].values():
                roll='volume',     # default is volume 

def initialize(context):
    schedule_function(func=rebalance, date_rule=date_rules.week_start())
    schedule_function(record_vars, date_rule=date_rules.every_day())

def can_trade(today, ftr):
    has_started = < today 
    has_not_ended = today <
    return has_started and has_not_ended

def rebalance(context, data):
    today =  #
    tradeable_contacts = [val for val in context.futures if can_trade(today, val)]
    atrs = calc_atrs(tradeable_contacts, data)
    print "ATRs: {}".format(pretty_d(atrs))
    sds = calc_stds(tradeable_contacts, data)
    print "SDs: {}".format(pretty_d(sds))
    current_contracts = [data.current(future, 'contract') for future in tradeable_contacts]
    targets = {current_contracts[0]: -0.5}
    objective = opt.TargetPortfolioWeights(targets)
    order_optimal_portfolio(objective, constraints=[])

def record_vars(context, data):
    price = data.history(context.futures[0], ATR_fields, 2, '1d')
    # record t-1 close price
    # can replace close with open, low
def calc_stds(contracts, data):
    prices = data.history(contracts, ATR_fields, ATR_LEN+2, '1d')
    ret_sds = {}
    for key in prices.minor_axis:
        ret_sds[key] = prices.minor_xs(key)['price'].ix[:-1].std()
    return ret_sds
def calc_atrs(contracts, data):
    """calculate ATR for futures we want to trade"""
    prices_atr = data.history(contracts, ATR_fields, ATR_LEN+2, '1d')
    # loop over all minor axes in panel (each future)
    ret_atrs = {}
    for key in prices_atr.minor_axis:
        ret_atrs[key] = calc_atr(prices_atr.minor_xs(key))
    return ret_atrs
def calc_atr(past_hlc):
    """calculates the Average True Range"""
    tr = calc_tr(past_hlc)
    return tr.mean()
def calc_tr(dfin):
    """calculates True Range for the data given in dfin"""
    print dfin.tail()
    dfin['closetm1'] = dfin['close'].shift()
    dfin['maxhc'] = dfin[['high','closetm1']].max(axis=1)
    dfin['minlc'] = dfin[['low','closetm1']].min(axis=1)
    dfin['tr'] = dfin['maxhc'] - dfin['minlc']
    return dfin['tr'].ix[1:-1]

def pretty_d(d):
    """replaces the contract with root symbol for easy reading"""
    return {k.root_symbol: v for k, v in d.iteritems()}
There was a runtime error.
1 response

As far as I can see the data precision for futures prices provided on 'd' level is too low for doing maths with futures. Prices for the continuous futures have the least precision, contracts seem to have 1 decimeter more, but still insufficient. E.g. for JY you will need 6 decimeters.

I have seen posts back on this without any solution. You'll get correct data if you retrieve by "data.history(future_contract, 'open', 60, '1m')" on minute level. However, doing maths for days will not be possible like this. Unless this is fixed any serious algo on some futures won't work (depending on their tick size).

Would be great to know if this issue is planned to be solved any time soon.