How to Pick Intraday Market Direction – The 80% Rule

Hello,

Have anyone code this rule or play with it? using the Value area

The Value Area: The range of prices where 70% of yesterday's volume took place. For instance, if the value area in the S&Ps is 138500-139000, then 70% of the previous day's volume took place between the prices of 138500-139000.

The 80% Rule: When the market opens above or below the value area, and then gets in the value area for two consecutive half-hour periods. The market then has an 80% chance of filling the value area.

13 responses

I always thought of working with "value area" these act as major support and resistances.
So let me try something on this. It will sharpen my pencil atleast. .. . .

By the way I need to understand more about the 80% rule so some questions here

1. If open is outside value area do we want to take position in direction of value area.
2. by filling the value area do you mean that you will want to stay in position till the other end of value area.

ie
if value area is 195~196 and open is 194.5 do you want to buy right away
and
you want to exit at 196 or at 195.5 ?

till that time I will find out how to find value area from he 1m panda

Here it is a nice paper that explains more in detail

@Erick

Attached is my implementation of the value area from the file you provided, scenario 2 and 3 were tested. close all orders 15 minutes before the close.

98
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
import pandas as pd

def initialize(context):

set_universe(universe.DollarVolumeUniverse(floor_percentile=95.0, ceiling_percentile=100.0))

schedule_function(
func=get_value_area,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_open(),
half_days=True
)

schedule_function(
func=check_open_price,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_open(),
half_days=True
)

schedule_function(
date_rule=date_rules.every_day(),
time_rule=time_rules.market_open(minutes=60),
half_days=True
)

schedule_function(
func=clear_lists,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_close(minutes=5),
half_days=True
)

schedule_function(
func=close_orders,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_close(minutes=15),
half_days=True
)

context.value_area_dict = {}
context.possible_shorts = []
context.possible_longs = []

# Will be called on every trade event for the securities you specify.
def handle_data(context, data):
pass
"""
for stock in data:
if get_open_orders(stock):
return

order_share = context.portfolio.positions[stock].amount

if order_share > 0:
print(order_share)
#if stock in context.value_area_dict:
if data[stock].price >= context.value_area_dict[stock.symbol]['high']:
order_target(stock,0)
log.info('exiting %s profit at %s' % (stock.symbol,data[stock].price))

elif data[stock].price <= context.value_area_dict[stock.symbol]['low']:
order_target(stock,0)
log.info('exiting %s loss at %s' % (stock.symbol,data[stock].price))

elif order_share < 0:
print(order_share)
if data[stock].price <= context.value_area_dict[stock.symbol]['low']:
order_target(stock,0)
log.info('exiting %s profit at %s' % (stock.symbol,data[stock].price))

elif data[stock].price >= context.value_area_dict[stock.symbol]['high']:
order_target(stock,0)
log.info('exiting %s loss at %s' % (stock.symbol,data[stock].price))
"""
def clear_lists(context,data):

#context.value_area_dict = {}
context.possible_shorts = []
context.possible_longs = []

def close_orders(context,data):
"""
loop through the whole universe and close any open positions
"""
for sec in data:
order_share = context.portfolio.positions[sec].amount
#print("open order_share=",sec.symbol, " share=",order_share)

if order_share > 0:
order_target(sec, 0)
print ('exited long',sec.symbol)
elif order_share < 0:
order_target(sec,0)
print ('exited short', sec.symbol)

#check all close order success
for sec in data:
order_share = context.portfolio.positions[sec].amount
#print("check order_share=",sec.symbol," share=",order_share)

"""
Runs at 10:30 (two consecutives brackets of 30 minutes)
"""

for stock in data:
price_history  = history(60,'1m','price')[stock]

if stock.symbol in context.possible_shorts\
and (price_history < context.value_area_dict[stock.symbol]['high']).all():
order_target_percent(stock,-0.2)
log.info('shorting %s price at %s'% (stock.symbol,data[stock].price))
#order_target_percent(stock,0,style=LimitOrder(context.value_area_dict[stock.symbol]['low']))
#order_target_percent(stock,0,style=StopOrder(context.value_area_dict[stock.symbol]['high']-0.50))

elif stock.symbol in context.possible_longs \
and (price_history > context.value_area_dict[stock.symbol]['low']).all():
order_target_percent(stock,0.2)
log.info('buying %s price at %s'% (stock.symbol,data[stock].price))
#order_target_percent(stock,0,style=LimitOrder(context.value_area_dict[stock.symbol]['high']))
#order_target_percent(stock,0,style=StopOrder(context.value_area_dict[stock.symbol]['low']+0.50))

def check_open_price(context,data):
"""
Runs at the beginning of the trading day and checks if the open of each stock is above/below
The Value Area, if yes then append to the longs/shorts list
"""
for stock in data:
if data[stock].open_price > context.value_area_dict[stock.symbol]['high']:
context.possible_shorts.append(stock.symbol)

elif data[stock].open_price < context.value_area_dict[stock.symbol]['low']:
context.possible_longs.append(stock.symbol)

def get_value_area(context,data):
"""
Runs at the beginning of the trading day,and gets the value area high/low of the previous day
and store it into the value_area_dict
"""
for stock in data:
volume_history = history(391,'1m','volume')[stock]
closes_history = history(391,'1m','close_price')[stock]

# save the volume and closes history of yesterdays into a dataframe
stock_frame = pd.DataFrame({'volume':volume_history,'closes':closes_history})
# sort the frame by volume, to get the most spend volume during yeterday
stock_frame = stock_frame.sort(columns='volume')

# record the value area high and low for this specific stock
context.value_area_dict[stock.symbol] = {'high':stock_frame['closes'][273:-1].max(),'low':stock_frame['closes'][273:-1].min()}

There was a runtime error.

Not that I am a big fan of this "80% rule" but I was curious so I modified Adham's code (thank you Adham):
- value area calculation (I disagree with how it was calculated, and it might be worth double checking my implementation too)
- modified the test that decides whether to enter a new position (long or shot) for a stock (it was too strict, I made it more flexible)
- positions are closed whenever the price reach the value area borders (and not 15 minutes before trading close)
- don't let slippage kills the algorithm (200.000\$ for each stock was way too much)
- kept leverage and long/short exposure under control

I realized that what really matter is how to determine when a stock should be traded. The article says:

"It stated that there is an 80% chance when a market opens (or trades) above or below the value area, and then trades in the value area for two consecutive half hour periods, then the market has an 80% chance of filling the entire value area."

My opinion is that "and then trades in the value area for two consecutive half hour periods" is not clear enough. Depending on how you code this test you can get very different results. Also, there are lots of details to be taken care of to fully implement the paper, details that I didn't cover.

But here is the backtest and thanks to Adham for the original code.

105
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
import math
import random
import datetime
import pandas as pd
import numpy as np
from sqlalchemy import or_
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline import Pipeline
from quantopian.pipeline.factors import SimpleMovingAverage
from quantopian.pipeline.factors import Latest
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.data.morningstar import valuation

#################################################################
#As Q has a [limit of 500 securities handled per day,][1]  if the security basket returned
#by get_fundamentals or Pipeline is larger than 500 securities and you like to trade or scan
#it all, you need to spread it across several days.  Here is a class to do it.

class UniverseProvider(object):

def __init__(self, shuffle_results = False, sort_columns = None, ascending = False):
self.shuffle_results = shuffle_results
self.sort_columns    = sort_columns
self.ascending       = ascending
self.flush()

def flush(self):
self.results        = None
self.stocks         = None
self.output         = None
return self

def set_shuffle_results(self, shuffle_results):
self.shuffle_results = shuffle_results
return self

def set_sort_columns(self, sort_columns, ascending):
self.sort_columns    = sort_columns
self.ascending       = ascending
return self

def get_results(self):
return self.results

def get_sids(self):
return self.stocks

def next(self, how_many_results):

if self.output is None:
df = self.get_output()
if self.shuffle_results:
df = df.reindex(index=np.random.permutation(df.index))
elif self.sort_columns is not None:
df = df.sort(columns=self.sort_columns, ascending=self.ascending)
self.output      = df
self.output_used = 0

start = self.output_used
end   = self.output_used + how_many_results
self.results = self.output.iloc[start:end,:]
self.stocks  = list(self.results.index)

#log.debug('UniverseProvider retrieved stocks %d, selected %d, offset %d' % (len(self.output.index), len(self.stocks), self.output_used))

self.output_used += how_many_results
if self.output_used >= len(self.output.index):
self.output = None

return self

def get_output(self):
raise NotImplementedError("Subclass must implement 'get_output' method")

class GetFundamentals(UniverseProvider):

def __init__(self, query, filter_ordered_nulls = True):
UniverseProvider.__init__(self)
self.query                = query
self.filter_ordered_nulls = filter_ordered_nulls

def get_output(self):
df = get_fundamentals(self.query, self.filter_ordered_nulls)
df = df.transpose()
return df

class PipelineOutput(UniverseProvider):

def __init__(self, pipeline_name):
UniverseProvider.__init__(self)
self.pipeline_name = pipeline_name

def get_output(self):
df = pipeline_output(self.pipeline_name)
return df

#################################################################
# Keep track of leverage and long/short exposure
#
# One Class to rule them all, One Class to define them,
# One Class to monitor them all and in the bytecode bind them
#

class ExposureMngr(object):

def __init__(self, target_leverage = 1.0, target_long_exposure_perc = 0.50, target_short_exposure_perc = 0.50):
self.target_leverage            = target_leverage
self.target_long_exposure_perc  = target_long_exposure_perc
self.target_short_exposure_perc = target_short_exposure_perc
self.short_exposure             = 0.0
self.long_exposure              = 0.0
self.open_order_short_exposure  = 0.0
self.open_order_long_exposure   = 0.0

def current_leverage(self, context, consider_open_orders = True):
curr_cash = context.portfolio.cash - (self.short_exposure * 2)
if consider_open_orders:
curr_cash -= self.open_order_short_exposure
curr_cash -= self.open_order_long_exposure
curr_leverage = (context.portfolio.portfolio_value - curr_cash) / context.portfolio.portfolio_value
return curr_leverage

def long_short_exposure_pct(self, context, consider_open_orders = True, consider_unused_cash = True):
long_exposure   = self.long_exposure
short_exposure  = self.short_exposure
if consider_open_orders:
long_exposure  += self.open_order_long_exposure
short_exposure += self.open_order_short_exposure
total_cash      = long_exposure + short_exposure
if consider_unused_cash:
total_cash += self.get_available_cash(context, consider_open_orders)
long_exposure_pct   = long_exposure  / total_cash if total_cash > 0 else 0
short_exposure_pct  = short_exposure / total_cash if total_cash > 0 else 0
return (long_exposure_pct, short_exposure_pct)

def get_available_cash(self, context, consider_open_orders = True):
curr_cash = context.portfolio.cash - (self.short_exposure * 2)
if consider_open_orders:
curr_cash -= self.open_order_short_exposure
curr_cash -= self.open_order_long_exposure
cash_limit = context.portfolio.portfolio_value - (self.target_leverage * context.portfolio.portfolio_value)
return curr_cash - cash_limit

def get_available_cash_long_short(self, context, consider_open_orders = True):
total_available_cash  = self.get_available_cash(context, consider_open_orders)
long_exposure         = self.long_exposure
short_exposure        = self.short_exposure
if consider_open_orders:
long_exposure  += self.open_order_long_exposure
short_exposure += self.open_order_short_exposure
target_exposure       = long_exposure + short_exposure + total_available_cash
target_long_exposure  = target_exposure * self.target_long_exposure_perc
target_short_exposure = target_exposure * self.target_short_exposure_perc
long_available_cash   = target_long_exposure  - long_exposure
short_available_cash  = target_short_exposure - short_exposure

force_range = lambda value, clamp_min, clamp_max: min(clamp_max, max(clamp_min, value))

return (force_range(long_available_cash , 0, total_available_cash),
force_range(short_available_cash, 0, total_available_cash) )

def update(self, context, data):

#
# calculate cash needed to complete open orders
#
self.open_order_short_exposure  = 0.0
self.open_order_long_exposure   = 0.0
for stock, orders in  get_open_orders().iteritems():
if stock not in data:
continue
price = data[stock].price
amount = 0 if stock not in context.portfolio.positions else context.portfolio.positions[stock].amount
for oo in orders:
order_amount = oo.amount - oo.filled
if order_amount < 0 and amount <= 0:
self.open_order_short_exposure += (price * -order_amount)
elif order_amount > 0 and amount >= 0:
self.open_order_long_exposure  += (price * order_amount)

#
# calculate long/short positions exposure
#
self.short_exposure = 0.0
self.long_exposure  = 0.0
for stock in context.portfolio.positions:
amount = context.portfolio.positions[stock].amount
last_sale_price = context.portfolio.positions[stock].last_sale_price
if amount < 0:
self.short_exposure += (last_sale_price * -amount)
elif amount > 0:
self.long_exposure  += (last_sale_price * amount)

#################################################################

#
# Given the constraints:
#    volume_limit - maximum of current stock volume you want to trade
#    min_shares_order - minimum number of shares per stocks
#    long_available_cash - total cash for long position
#    short_available_cash - total cash for short position
# This function calculate how many shares to allocate to each stock present in stocks_to_trade
def calculate_positions(data, stocks_to_trade, long_available_cash, short_available_cash, volume_limit, min_shares_order):

#
# decide positions for each stock
#
position_per_stock = {}

available_cash = {
1  : long_available_cash,
-1 : short_available_cash,
}

while True:

burst_tot = {
1  : len( [v for v in stocks_to_trade.itervalues() if v ==  1 ] ) ,
-1 : len( [v for v in stocks_to_trade.itervalues() if v == -1 ] ) ,
}

cash_stock = {
1  :  (available_cash[1]  / burst_tot[1]  if burst_tot[1]  > 0 else 0. ) ,
-1 :  (available_cash[-1] / burst_tot[-1] if burst_tot[-1] > 0 else 0. ) ,
}

# exit loop if we used all money or there are no more stock to trade
if len(stocks_to_trade) == 0 or sum(available_cash.values()) <= 0:
break

positions = round(cash_stock[burst] / data[stock].price)
if positions <= 0:
continue

previous_position = abs(position_per_stock[stock]) if stock in position_per_stock else 0

if previous_position == 0 and positions < min_shares_order:
positions = min_shares_order

if (previous_position + positions) >= data[stock].volume * volume_limit:
positions = data[stock].volume * volume_limit - previous_position
if (previous_position + positions) < min_shares_order:
continue

available_cash[burst] -= positions * data[stock].price

if burst > 0:
position_per_stock[stock] = previous_position + positions
elif burst < 0:
position_per_stock[stock] = -(previous_position + positions)

return position_per_stock

#################################################################

#
# Fetch some new entries from our universe provider (can be pipeline or get_fundamentals)
#
context.stocks = context.universe.flush().next(context.handled_sec_limit).get_sids()
update_universe(context.stocks)

context.max_lev = 0.0

def initialize(context):

context.handled_sec_limit = 500 # Quantopian limit is 500

context.exposure = ExposureMngr(target_leverage = 1.0,
target_long_exposure_perc = 0.50,
target_short_exposure_perc = 0.50)

price = USEquityPricing.close.latest
volume = USEquityPricing.volume.latest

price_filter  = (price >= 5) & (price <= 1000)
volume_filter = volume.top(1500)

pipe = Pipeline()
pipe.set_screen(price_filter & volume_filter)

pipe = attach_pipeline(pipe, name='pipeline1')
context.universe = PipelineOutput('pipeline1')
#context.universe.set_sort_columns(sort_columns=['volume'], ascending=True)
context.universe.set_shuffle_results(shuffle_results = True)

schedule_function(
func=get_value_area,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_open(),
half_days=True
)

schedule_function(
func=check_open_price,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_open(),
half_days=True
)

schedule_function(
date_rule=date_rules.every_day(),
time_rule=time_rules.market_open(minutes=60),
half_days=True
)

schedule_function(
func=clear_lists,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_close(minutes=5),
half_days=True
)

schedule_function(
func=close_orders,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_close(minutes=15),
half_days=True
)

context.value_area_dict = {}
context.possible_shorts = []
context.possible_longs = []

context.max_lev = 0.0

# Will be called on every trade event for the securities you specify.
def handle_data(context, data):

if context.account.leverage > context.max_lev:
context.max_lev = context.account.leverage
record(leverage=context.account.leverage)

#
# monitor the stocks we traded and try to close some good trades
#
for stock in context.portfolio.positions:

if get_open_orders(stock):
continue

if stock not in context.value_area_dict:
log.error("This should never happen (%s not in context.value_area_dict)" % (stock.symbol))
continue

value_area_low  = context.value_area_dict[stock]['low']
value_area_high = context.value_area_dict[stock]['high']

amount = context.portfolio.positions[stock].amount
price = data[stock].price

threshold = (value_area_high - value_area_low) * 0.10

if amount > 0:
#
# We have a long position, sell when we are approachig the valure area high
#
if value_area_high - price < threshold:
order_target(stock, 0)

elif amount < 0:
#
# We have a short position, buy when we are approachig the valure area low
#
if price - value_area_low < threshold:
order_target(stock,  0)

def close_orders(context,data):
"""
loop through the whole universe and close any open positions still open
"""
for stock in context.portfolio.positions:
order_target(stock, 0)

"""
Runs at 10:30 (two consecutives brackets of 30 minutes)
"""

#
# keep track of portfolio exposure
#
context.exposure.update(context, data)

#
# Trade only if we have some cash
#
available_cash = context.exposure.get_available_cash(context)
if available_cash < 500:
return

price_history   = history(60,'1m','price')
volume_history  = history(60,'1m','volume')

#
# Decide what stock to buy or short
#
for stock in data:

if stock not in context.value_area_dict or \
(stock not in context.possible_shorts and stock not in context.possible_longs):
continue

value_area_low = context.value_area_dict[stock]['low']
value_area_high = context.value_area_dict[stock]['high']

# if the value area is too small we don't trade this stock
if (value_area_high - value_area_low) < 0.50:
continue

price = data[stock].price
threshold = (value_area_high - value_area_low) * 0.10

if stock in context.possible_shorts and \
(price_history[stock] < value_area_high).sum() > 52 and \
(value_area_high - price < threshold):
elif stock in context.possible_longs and \
(price_history[stock] > value_area_low).sum() > 52 and \
(price - value_area_low < threshold):

#
# Alternatively decide to enter a position based on comparison of current area value
# with previous day area value
#

#curr_value_area_low, curr_value_area_high = calculate_value_area_low_high(volume_history[stock], price_history[stock])

#threshold = (value_area_high - value_area_low) * 0.20

#if (value_area_high-curr_value_area_low > threshold) or (curr_value_area_high-value_area_low >  threshold):
#    if stock in context.possible_shorts:
#    elif stock in context.possible_longs:

#
# calculate the positions for each stock given the list of stocks to buy or short
#
long_available_cash, short_available_cash = context.exposure.get_available_cash_long_short(context)
position_per_stock = calculate_positions(data, stocks_to_trade, long_available_cash, short_available_cash, volume_limit = 0.15, min_shares_order = 1)

#
# finally enter orders
#
for stock, positions in position_per_stock.iteritems():
order(stock, positions)

def clear_lists(context,data):
context.value_area_dict = {}
context.possible_shorts = []
context.possible_longs = []

def check_open_price(context,data):
"""
Runs at the beginning of the trading day and checks if the open of each stock is above/below
The Value Area, if yes then append to the longs/shorts list
"""
for stock in data:

if stock not in context.value_area_dict:
continue

value_area_low  = context.value_area_dict[stock]['low']
value_area_high = context.value_area_dict[stock]['high']

if data[stock].open_price > value_area_high:
context.possible_shorts.append(stock)

elif data[stock].open_price < value_area_low:
context.possible_longs.append(stock)

def get_value_area(context,data):
"""
Runs at the beginning of the trading day,and gets the value area high/low of the previous day
and store it into the value_area_dict
"""

#
# Get yesterday history (excluding today entry, the last one)
#
volume_history = history(391,'1m','volume')[:-1]
price_history  = history(391,'1m','close_price')[:-1]

for stock in data:

value_area_low, value_area_high = calculate_value_area_low_high(volume_history[stock], price_history[stock])

# record the value area high and low for this specific stock
context.value_area_dict[stock] = {'high':value_area_high, 'low':value_area_low}

def calculate_value_area_low_high(volume_history, price_history):
"""
calculate_value_area_low_high
"""
stock_frame = pd.DataFrame( {'volume':volume_history, 'price':price_history } )
stock_frame = stock_frame.sort(columns='volume', ascending=False)

cumsum = stock_frame['volume'].cumsum()
_70_percent_activity = stock_frame['volume'].sum() * 0.70
_70_percent_activity_prices = stock_frame['price'][cumsum <= _70_percent_activity]

value_area_low  = _70_percent_activity_prices.min()
value_area_high = _70_percent_activity_prices.max()

return (value_area_low, value_area_high)
There was a runtime error.

oh, I forgot. A test without commission and slippage.

105
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
import math
import random
import datetime
import pandas as pd
import numpy as np
from sqlalchemy import or_
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline import Pipeline
from quantopian.pipeline.factors import SimpleMovingAverage
from quantopian.pipeline.factors import Latest
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.data.morningstar import valuation

#################################################################
#As Q has a [limit of 500 securities handled per day,][1]  if the security basket returned
#by get_fundamentals or Pipeline is larger than 500 securities and you like to trade or scan
#it all, you need to spread it across several days.  Here is a class to do it.

class UniverseProvider(object):

def __init__(self, shuffle_results = False, sort_columns = None, ascending = False):
self.shuffle_results = shuffle_results
self.sort_columns    = sort_columns
self.ascending       = ascending
self.flush()

def flush(self):
self.results        = None
self.stocks         = None
self.output         = None
return self

def set_shuffle_results(self, shuffle_results):
self.shuffle_results = shuffle_results
return self

def set_sort_columns(self, sort_columns, ascending):
self.sort_columns    = sort_columns
self.ascending       = ascending
return self

def get_results(self):
return self.results

def get_sids(self):
return self.stocks

def next(self, how_many_results):

if self.output is None:
df = self.get_output()
if self.shuffle_results:
df = df.reindex(index=np.random.permutation(df.index))
elif self.sort_columns is not None:
df = df.sort(columns=self.sort_columns, ascending=self.ascending)
self.output      = df
self.output_used = 0

start = self.output_used
end   = self.output_used + how_many_results
self.results = self.output.iloc[start:end,:]
self.stocks  = list(self.results.index)

#log.debug('UniverseProvider retrieved stocks %d, selected %d, offset %d' % (len(self.output.index), len(self.stocks), self.output_used))

self.output_used += how_many_results
if self.output_used >= len(self.output.index):
self.output = None

return self

def get_output(self):
raise NotImplementedError("Subclass must implement 'get_output' method")

class GetFundamentals(UniverseProvider):

def __init__(self, query, filter_ordered_nulls = True):
UniverseProvider.__init__(self)
self.query                = query
self.filter_ordered_nulls = filter_ordered_nulls

def get_output(self):
df = get_fundamentals(self.query, self.filter_ordered_nulls)
df = df.transpose()
return df

class PipelineOutput(UniverseProvider):

def __init__(self, pipeline_name):
UniverseProvider.__init__(self)
self.pipeline_name = pipeline_name

def get_output(self):
df = pipeline_output(self.pipeline_name)
return df

#################################################################
# Keep track of leverage and long/short exposure
#
# One Class to rule them all, One Class to define them,
# One Class to monitor them all and in the bytecode bind them
#

class ExposureMngr(object):

def __init__(self, target_leverage = 1.0, target_long_exposure_perc = 0.50, target_short_exposure_perc = 0.50):
self.target_leverage            = target_leverage
self.target_long_exposure_perc  = target_long_exposure_perc
self.target_short_exposure_perc = target_short_exposure_perc
self.short_exposure             = 0.0
self.long_exposure              = 0.0
self.open_order_short_exposure  = 0.0
self.open_order_long_exposure   = 0.0

def current_leverage(self, context, consider_open_orders = True):
curr_cash = context.portfolio.cash - (self.short_exposure * 2)
if consider_open_orders:
curr_cash -= self.open_order_short_exposure
curr_cash -= self.open_order_long_exposure
curr_leverage = (context.portfolio.portfolio_value - curr_cash) / context.portfolio.portfolio_value
return curr_leverage

def long_short_exposure_pct(self, context, consider_open_orders = True, consider_unused_cash = True):
long_exposure   = self.long_exposure
short_exposure  = self.short_exposure
if consider_open_orders:
long_exposure  += self.open_order_long_exposure
short_exposure += self.open_order_short_exposure
total_cash      = long_exposure + short_exposure
if consider_unused_cash:
total_cash += self.get_available_cash(context, consider_open_orders)
long_exposure_pct   = long_exposure  / total_cash if total_cash > 0 else 0
short_exposure_pct  = short_exposure / total_cash if total_cash > 0 else 0
return (long_exposure_pct, short_exposure_pct)

def get_available_cash(self, context, consider_open_orders = True):
curr_cash = context.portfolio.cash - (self.short_exposure * 2)
if consider_open_orders:
curr_cash -= self.open_order_short_exposure
curr_cash -= self.open_order_long_exposure
cash_limit = context.portfolio.portfolio_value - (self.target_leverage * context.portfolio.portfolio_value)
return curr_cash - cash_limit

def get_available_cash_long_short(self, context, consider_open_orders = True):
total_available_cash  = self.get_available_cash(context, consider_open_orders)
long_exposure         = self.long_exposure
short_exposure        = self.short_exposure
if consider_open_orders:
long_exposure  += self.open_order_long_exposure
short_exposure += self.open_order_short_exposure
target_exposure       = long_exposure + short_exposure + total_available_cash
target_long_exposure  = target_exposure * self.target_long_exposure_perc
target_short_exposure = target_exposure * self.target_short_exposure_perc
long_available_cash   = target_long_exposure  - long_exposure
short_available_cash  = target_short_exposure - short_exposure

force_range = lambda value, clamp_min, clamp_max: min(clamp_max, max(clamp_min, value))

return (force_range(long_available_cash , 0, total_available_cash),
force_range(short_available_cash, 0, total_available_cash) )

def update(self, context, data):

#
# calculate cash needed to complete open orders
#
self.open_order_short_exposure  = 0.0
self.open_order_long_exposure   = 0.0
for stock, orders in  get_open_orders().iteritems():
if stock not in data:
continue
price = data[stock].price
amount = 0 if stock not in context.portfolio.positions else context.portfolio.positions[stock].amount
for oo in orders:
order_amount = oo.amount - oo.filled
if order_amount < 0 and amount <= 0:
self.open_order_short_exposure += (price * -order_amount)
elif order_amount > 0 and amount >= 0:
self.open_order_long_exposure  += (price * order_amount)

#
# calculate long/short positions exposure
#
self.short_exposure = 0.0
self.long_exposure  = 0.0
for stock in context.portfolio.positions:
amount = context.portfolio.positions[stock].amount
last_sale_price = context.portfolio.positions[stock].last_sale_price
if amount < 0:
self.short_exposure += (last_sale_price * -amount)
elif amount > 0:
self.long_exposure  += (last_sale_price * amount)

#################################################################

#
# Given the constraints:
#    volume_limit - maximum of current stock volume you want to trade
#    min_shares_order - minimum number of shares per stocks
#    long_available_cash - total cash for long position
#    short_available_cash - total cash for short position
# This function calculate how many shares to allocate to each stock present in stocks_to_trade
def calculate_positions(data, stocks_to_trade, long_available_cash, short_available_cash, volume_limit, min_shares_order):

#
# decide positions for each stock
#
position_per_stock = {}

available_cash = {
1  : long_available_cash,
-1 : short_available_cash,
}

while True:

burst_tot = {
1  : len( [v for v in stocks_to_trade.itervalues() if v ==  1 ] ) ,
-1 : len( [v for v in stocks_to_trade.itervalues() if v == -1 ] ) ,
}

cash_stock = {
1  :  (available_cash[1]  / burst_tot[1]  if burst_tot[1]  > 0 else 0. ) ,
-1 :  (available_cash[-1] / burst_tot[-1] if burst_tot[-1] > 0 else 0. ) ,
}

# exit loop if we used all money or there are no more stock to trade
if len(stocks_to_trade) == 0 or sum(available_cash.values()) <= 0:
break

positions = round(cash_stock[burst] / data[stock].price)
if positions <= 0:
continue

previous_position = abs(position_per_stock[stock]) if stock in position_per_stock else 0

if previous_position == 0 and positions < min_shares_order:
positions = min_shares_order

if (previous_position + positions) >= data[stock].volume * volume_limit:
positions = data[stock].volume * volume_limit - previous_position
if (previous_position + positions) < min_shares_order:
continue

available_cash[burst] -= positions * data[stock].price

if burst > 0:
position_per_stock[stock] = previous_position + positions
elif burst < 0:
position_per_stock[stock] = -(previous_position + positions)

return position_per_stock

#################################################################

#
# Fetch some new entries from our universe provider (can be pipeline or get_fundamentals)
#
context.stocks = context.universe.flush().next(context.handled_sec_limit).get_sids()
update_universe(context.stocks)

context.max_lev = 0.0

def initialize(context):

context.handled_sec_limit = 500 # Quantopian limit is 500

context.exposure = ExposureMngr(target_leverage = 1.0,
target_long_exposure_perc = 0.50,
target_short_exposure_perc = 0.50)

price = USEquityPricing.close.latest
volume = USEquityPricing.volume.latest

price_filter  = (price >= 5) & (price <= 1000)
volume_filter = volume.top(1500)

pipe = Pipeline()
pipe.set_screen(price_filter & volume_filter)

pipe = attach_pipeline(pipe, name='pipeline1')
context.universe = PipelineOutput('pipeline1')
#context.universe.set_sort_columns(sort_columns=['volume'], ascending=True)
context.universe.set_shuffle_results(shuffle_results = True)

schedule_function(
func=get_value_area,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_open(),
half_days=True
)

schedule_function(
func=check_open_price,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_open(),
half_days=True
)

schedule_function(
date_rule=date_rules.every_day(),
time_rule=time_rules.market_open(minutes=60),
half_days=True
)

schedule_function(
func=clear_lists,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_close(minutes=5),
half_days=True
)

schedule_function(
func=close_orders,
date_rule=date_rules.every_day(),
time_rule=time_rules.market_close(minutes=15),
half_days=True
)

context.value_area_dict = {}
context.possible_shorts = []
context.possible_longs = []

context.max_lev = 0.0

set_commission(commission.PerShare(0.00))
set_slippage(slippage.VolumeShareSlippage(price_impact=0.0))

# Will be called on every trade event for the securities you specify.
def handle_data(context, data):

if context.account.leverage > context.max_lev:
context.max_lev = context.account.leverage
record(leverage=context.account.leverage)

#
# monitor the stocks we traded and try to close some good trades
#
for stock in context.portfolio.positions:

if get_open_orders(stock):
continue

if stock not in context.value_area_dict:
log.error("This should never happen (%s not in context.value_area_dict)" % (stock.symbol))
continue

value_area_low  = context.value_area_dict[stock]['low']
value_area_high = context.value_area_dict[stock]['high']

amount = context.portfolio.positions[stock].amount
price = data[stock].price

threshold = (value_area_high - value_area_low) * 0.01

if amount > 0:
#
# We have a long position, sell when we are approachig the valure area high
#
if value_area_high - price < threshold:
order_target(stock, 0)

elif amount < 0:
#
# We have a short position, buy when we are approachig the valure area low
#
if price - value_area_low < threshold:
order_target(stock,  0)

def close_orders(context,data):
"""
loop through the whole universe and close any open positions still open
"""
for stock in context.portfolio.positions:
order_target(stock, 0)

"""
Runs at 10:30 (two consecutives brackets of 30 minutes)
"""

#
# keep track of portfolio exposure
#
context.exposure.update(context, data)

#
# Trade only if we have some cash
#
available_cash = context.exposure.get_available_cash(context)
if available_cash < 500:
return

price_history   = history(60,'1m','price')
volume_history  = history(60,'1m','volume')

#
# Decide what stock to buy or short
#
for stock in data:

if stock not in context.value_area_dict or \
(stock not in context.possible_shorts and stock not in context.possible_longs):
continue

value_area_low = context.value_area_dict[stock]['low']
value_area_high = context.value_area_dict[stock]['high']

# if the value area is too small we don't trade this stock
if (value_area_high - value_area_low) < 0.40:
continue

price = data[stock].price
threshold = (value_area_high - value_area_low) * 0.15

if stock in context.possible_shorts and \
(price_history[stock] < value_area_high).sum() > 50 and \
(value_area_high - price < threshold):
elif stock in context.possible_longs and \
(price_history[stock] > value_area_low).sum() > 50 and \
(price - value_area_low < threshold):

#
# Alternatively decide to enter a position based on comparison of current area value
# with previous day area value
#

#curr_value_area_low, curr_value_area_high = calculate_value_area_low_high(volume_history[stock], price_history[stock])

#threshold = (value_area_high - value_area_low) * 0.20

#if (value_area_high-curr_value_area_low > threshold) or (curr_value_area_high-value_area_low >  threshold):
#    if stock in context.possible_shorts:
#    elif stock in context.possible_longs:

#
# calculate the positions for each stock given the list of stocks to buy or short
#
long_available_cash, short_available_cash = context.exposure.get_available_cash_long_short(context)
position_per_stock = calculate_positions(data, stocks_to_trade, long_available_cash, short_available_cash, volume_limit = 0.15, min_shares_order = 1)

#
# finally enter orders
#
for stock, positions in position_per_stock.iteritems():
order(stock, positions)

def clear_lists(context,data):
context.value_area_dict = {}
context.possible_shorts = []
context.possible_longs = []

def check_open_price(context,data):
"""
Runs at the beginning of the trading day and checks if the open of each stock is above/below
The Value Area, if yes then append to the longs/shorts list
"""
for stock in data:

if stock not in context.value_area_dict:
continue

value_area_low  = context.value_area_dict[stock]['low']
value_area_high = context.value_area_dict[stock]['high']

if data[stock].open_price > value_area_high:
context.possible_shorts.append(stock)

elif data[stock].open_price < value_area_low:
context.possible_longs.append(stock)

def get_value_area(context,data):
"""
Runs at the beginning of the trading day,and gets the value area high/low of the previous day
and store it into the value_area_dict
"""

#
# Get yesterday history (excluding today entry, the last one)
#
volume_history = history(391,'1m','volume')[:-1]
price_history  = history(391,'1m','close_price')[:-1]

for stock in data:

value_area_low, value_area_high = calculate_value_area_low_high(volume_history[stock], price_history[stock])

# record the value area high and low for this specific stock
context.value_area_dict[stock] = {'high':value_area_high, 'low':value_area_low}

def calculate_value_area_low_high(volume_history, price_history):
"""
calculate_value_area_low_high
"""
stock_frame = pd.DataFrame( {'volume':volume_history, 'price':price_history } )
stock_frame = stock_frame.sort(columns='volume', ascending=False)

cumsum = stock_frame['volume'].cumsum()
_70_percent_activity = stock_frame['volume'].sum() * 0.70
_70_percent_activity_prices = stock_frame['price'][cumsum <= _70_percent_activity]

value_area_low  = _70_percent_activity_prices.min()
value_area_high = _70_percent_activity_prices.max()

return (value_area_low, value_area_high)
There was a runtime error.

Luca, what you are undertaking here is a rather enormous task. You’re essentially setting off to create an algorithm based on the principles of auction market theory (AMT). I have traded, researched, and taught auction market theory for the past six years. I continuously blog examples of putting AMT into practice here. I wouldn’t make a trade decision without an analysis of the auction and volume profile.

More importantly, I would suggest that you first truly understand why AMT is robust and why it is absolutely crucial to making trading decisions (as opposed to simply setting off to write script based on some fixed percent “rule” and see what the cumulative returns plot looks like).

The interaction between transaction volume and price is key to understanding how and why prices change. There are two reasons why price increases after a buyer-initiated trade, e.g. First, market-makers upwardly adjust their beliefs about fair value as the buyer’s trade could contain asymmetric information (true edge). Second, market-makers require compensation for allowing their inventory to position itself away from fair market value. Therefore, a risk-averse market-maker will accommodate a subsequent buy order only at a higher price.

In summary, information asymmetry drives volume and quotes change in the direction of transaction volume. Volume begets price movement.

No one has done more to advance the study and practice of auction market theory than Don Jones, Cisco Futures. His website contains his research and short course materials. By the way, the value area is the central 70% and exists only when the distribution of transaction volume is normal (Gaussian). See? It gets complicated.

I’ll be very interested to see what you come up.

Luca, see if you can visually identify the value areas in SPY for the past six months. I uploaded a screen grab of my market profile chart here. You'll immediately see the market doesn't provide us with nice normal distributions of volume and price from which we can exploit the "value area."

Thank you Stephen, your comment is much more interesting than the 80% rule ;) I will dig into AMT and see what I find.

Great! To do this stuff well, we need to become a hybrid of an avant-garde trader and an IT engineer. The more you know about what drives price, the better script writer you’ll be. Two additional comments:

1. You refer to MarketDelta in your earlier post. They are a data provider for volume on the horizontal axis. I don’t believe Quantopian provides that data. Q staff, please chime in.

2. Since I’m a trader first, quant second, I first translate the basic concept onto a chart to see what kind of trade signals it produces. That initial step serves as a “proof of concept.” So, I did exactly that with your value area trading concept that you're embarking on. And I used an intra-day chart since that was the title of your post.

Here’s the proof-of-concept script:

def lr = 6 * ( WMA(OHLC4, 10) - Average(OHLC4, 10) ) / 9;
def upBar = close > open;
def dnBar = close < open;
def short = lr > 0 && upBar && close > VAHigh;
def long = lr < 0 && dnBar && close < VALow;
plot longSignal = if long && !long1 and !long[2] then low else double.nan;
plot shortSignal = if short && !short1 and !short[2] then high else double.nan;

That means we plot a long signal on an upbar, in an uptrend, when price exceeds the value area high. (vv. for a short) This ensures you are fading the herd, buying on weakness/selling on strength, and trading when prices are distorted from fair market value.

Your concept looks like this during the past five days, trading S&P futures, intra-day at 5000 ticks. The results show a highly positive predictive value: 28 signals profitable, 5 false positives.

So, Luca …if you can get volume data on the horizontal axis into Quantopian’s IDE and write a python script equivalent to my thinkscript …you’re going to kick some butt. Good luck! (By the way, I took trades on a number of those signals last week.)

Yes, well put Steven. I am interested in an ES or NQ algo based on AMT. I don't know if futures are supported by Q or will be but it is there that I think an AMT algo can be fun. Won't qualify for the Q game but I do like the idea of an ES algo.

Lucas, to illustrate how arbitrary a fixed percentage level is, I labelled the auction levels in the Market Profile graphic here. (In the profile I labeled, both time and volume are roughly normally distributed.) If you just look at the past four days (S&P futures), the value areas expand and contract around the volume node as the auction proceeds. Rarely do they lie equidistant from the high volume node. If, however, you focus solely on the volume profile …your strategy of selling over-value and buying under-value (as defined by the auction and lying greater than one standard deviation away from fair market value) should still perform quite well.

@Stephen I was trying to use your script in ThinkOrSwim but It is not working could you paste the full script to check it out?

Luca, Adham, Thanks for the implementation. The idea behind this is to give you an insight to define an intraday edge. The strategy as you can see it by itself It wont be profitable. However it can be another variable for more complete strategy. I am lately experimenting with strategies with probabilities and statistics of movements in the indexes. It would be nice to get in into the market first 2 hours and get out of RISk :) and sleep well at night heheh

EG

Stephen pointed to some WSJ Options data at http://online.wsj.com/mdc/public/page/2_3022-mfgppl-moneyflow-20151014.html?mod=mdc_pastcalendar that goes back to May 2007. I'm hoping there is someone out there idealistic enough to want to scrape that and offer up for backtesting via fetcher for folks, seems if tried by anyone, they've been quiet, while surely significant upside potential in options data.