Futures data.history problem for JY & JE

I am trying to calculate the volatility (standard deviation, ATR) for futures contracts. However I ran into a snag for JY & JE in 2012. The problem is: the results from data.history in a backtest show every day having the same value for a stretch of time, this occurs for over 100 days in the summer of 2012. The actual price did not look like this. It appears that the date is getting truncated to 3 significant digits. What is interesting is that this only happens for days earlier than t-1.

Here is an example where on 2012-09-24 I fetch the last 100 days data for the continuous future JY. (See the attached backtest)

2012-09-24 06:31  PRINT                               close      high       low     price
2012-09-18 00:00:00+00:00  0.012000  0.012000  0.012000  0.012000
2012-09-19 00:00:00+00:00  0.012000  0.012000  0.012000  0.012000
2012-09-20 00:00:00+00:00  0.012000  0.012000  0.012000  0.012000
2012-09-21 00:00:00+00:00  0.012000  0.012000  0.012000  0.012000
2012-09-24 00:00:00+00:00  0.012827  0.012835  0.012803  0.012827
2012-09-24 06:31  PRINT ATRs: {'JE': 0.0, 'JY': 0.0}
2012-09-24 06:31  PRINT SDs: {'JE': 8.716877585302702e-18, 'JY': 8.716877585302702e-18}


I take the data from the last 100 days and calculate the ATR and Standard deviation, for many weeks these values are effectively 0.
I remember reading on one post on futures that the values were only displayed to three significant digits, but this was only a display issue. That doesn't seem to be the case as any math on t-1 values shows that there are only three significant digits in there.

Does anyone know why these values are incorrect? If you plot the values of JY in Research you do not get this.

Thanks for the help.

3
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
from quantopian.algorithm import order_optimal_portfolio
import quantopian.experimental.optimize as opt

month_idx = 0

ATR_fields=['high', 'low', 'close', 'price']
ATR_LEN = 100

#   https://www.quantopian.com/posts/continuous-future-data-lifespans
available_futures = {
'debug': {
'jpy': 'JY', # good
'jpy_emini': 'JE', # good
}
}

def make_futures(context):
# populates context.futures, by flattening available_futures
context.futures = []
#for sector in available_futures:
for sector in ["debug"]:
for future in available_futures[sector].values():
context.futures.append(continuous_future(
future,
offset=month_idx,
roll='volume',     # default is volume
)
)

def initialize(context):
make_futures(context)
schedule_function(func=rebalance, date_rule=date_rules.week_start())
schedule_function(record_vars, date_rule=date_rules.every_day())

has_started = ftr.start_date.date() < today
has_not_ended = today < ftr.end_date.date()
return has_started and has_not_ended

def rebalance(context, data):
today = data.current_session.date()  # datetime.date

print "ATRs: {}".format(pretty_d(atrs))

print "SDs: {}".format(pretty_d(sds))

current_contracts = [data.current(future, 'contract') for future in tradeable_contacts]

targets = {current_contracts[0]: -0.5}
objective = opt.TargetPortfolioWeights(targets)
order_optimal_portfolio(objective, constraints=[])

def record_vars(context, data):
price = data.history(context.futures[0], ATR_fields, 2, '1d')
# record t-1 close price
# can replace close with open, low
record(JY_close_tm1=price.ix[0]['close'])

def calc_stds(contracts, data):
prices = data.history(contracts, ATR_fields, ATR_LEN+2, '1d')
ret_sds = {}
for key in prices.minor_axis:
ret_sds[key] = prices.minor_xs(key)['price'].ix[:-1].std()
return ret_sds

def calc_atrs(contracts, data):
"""calculate ATR for futures we want to trade"""
prices_atr = data.history(contracts, ATR_fields, ATR_LEN+2, '1d')
# loop over all minor axes in panel (each future)
ret_atrs = {}
for key in prices_atr.minor_axis:
ret_atrs[key] = calc_atr(prices_atr.minor_xs(key))
return ret_atrs

def calc_atr(past_hlc):
"""calculates the Average True Range"""
tr = calc_tr(past_hlc)
return tr.mean()

def calc_tr(dfin):
"""calculates True Range for the data given in dfin"""
print dfin.tail()
dfin['closetm1'] = dfin['close'].shift()
dfin['maxhc'] = dfin[['high','closetm1']].max(axis=1)
dfin['minlc'] = dfin[['low','closetm1']].min(axis=1)
dfin['tr'] = dfin['maxhc'] - dfin['minlc']
return dfin['tr'].ix[1:-1]

def pretty_d(d):
"""replaces the contract with root symbol for easy reading"""
return {k.root_symbol: v for k, v in d.iteritems()}
There was a runtime error.
1 response

As far as I can see the data precision for futures prices provided on 'd' level is too low for doing maths with futures. Prices for the continuous futures have the least precision, contracts seem to have 1 decimeter more, but still insufficient. E.g. for JY you will need 6 decimeters.

I have seen posts back on this without any solution. You'll get correct data if you retrieve by "data.history(future_contract, 'open', 60, '1m')" on minute level. However, doing maths for days will not be possible like this. Unless this is fixed any serious algo on some futures won't work (depending on their tick size).

Would be great to know if this issue is planned to be solved any time soon.