Enhancing Short-Term Mean-Reversion Strategies

For the majority of quant equity hedge funds that have holding periods on the order of a few days to a couple weeks (“medium frequency” funds), by far the most common strategy is some variation of short-term mean reversion. Of course, while no hard data exists to support this claim, in my experience working alongside several dozen quant groups within two multi-strategy hedge funds, and admittedly only seeing aggregate performance or individual performance that was anonymized, I was able to observe a strong correlation between hedge fund performance and the returns to a simple mean-reversion strategy (for example, buying the quintile of stocks in the S&P500 with the lowest five-day returns and shorting the quintile of stocks with the highest five-day returns). When the simple mean-reversion strategy was going through a rough period, the quant groups were almost universally down as well.

Given how common short-term mean-reversion strategies are, and more importantly, how well and consistently these strategies have held up over the years, it’s worthwhile to consider ways to enhance the performance of a simple mean-reversion strategy. Of course, every quant fund has their own way of implementing a mean-reversion strategy, which they often refer to as their “secret sauce”. In this post, I’d like to offer some possible ingredients that the community can use to create their own secret sauce.

I backtested numerous ideas on Quantopian, some worked as expected and many failed. Here is a summary of a few interesting ones:

• Distinguish between liquidity events and news events
o Use news sentiment data from Accern and Sentdex
o Use volume data
o Look for steady stock moves instead of jumps
• Trade on a universe of stocks where there is less uncertainty about fair value
o Use a low volatility universe
o Use a universe of stocks that have a lower dispersion of analyst estimates
• Miscellaneous enhancements
o Separate different types of announcements using EventVestor data
o Trade on a universe of lower liquidity stocks
o Trade on a universe that excludes extreme momentum stocks
o Skip the most recent day when computing mean reversion

As a baseline, I compared all the potential enhancements with a simple mean-reversion strategy: I sorted stocks in the Q500US into quintiles based on five-day returns. For the current day’s return, I used the 3:55 price and rebalanced the equally weighted, unleveraged portfolio daily at the close. Stocks were unwound when they dropped out of the extreme quintile. For such a simple strategy, it performed reasonably well. The cumulative return over the entire backtesting period from 2002 to present was about 100% and the Sharpe Ratio was 0.57.

Distinguishing between liquidity events and news events

There are at least two competing theories about why short-term mean-reversion strategies work (for example, see Subrahmanyam):

• Because of behavioral biases (for example, investors overweight recent information), the market overreacts to both good news and bad news
• Liquidity shocks (for example, a large portfolio rebalancing trade by an uniformed trader) lead to temporary moves that get reversed

There is some evidence that for certain news events, investors actually underreact to news, leading to trending, rather than mean reversion. So if it were possible to identify liquidity trades as opposed to news-related trades, you could avoid potentially money-losing news-related trades and focus on the more profitable liquidity trades.

Ravenpack, a news analytics data provider that competes with Accern AlphaOne and Sentdex, has released a white paper arguing that their data can enhance a mean-reversion strategy (there’s no link, but you can request a copy of their paper “Enhancing Short-term Stock Reversal Strategies With News Analytics“ from their website here). Their first enhancement is to combine mean reversion with news sentiment, using their own “Sentiment Strength Indicator”. They do a double sort on five-day returns and news sentiment and find that if they buy past losers that have strong positive sentiment and short past winners that have strong negative sentiment, they can improve the performance of the straight mean-reversion strategy. They also combine mean reversion with a measure of the number of news stories (regardless of whether they are positive or negative), which is their “Event Volume Indicator”. Here, they buy losers with low event volume but sell winners with high event volume. I would have expected selling on low event volume to work better, given the premise that high event volume represents more news-related trades.

I tried something similar with the daily estimates of news sentiment supplied by Accern and Sentdex (neither dataset has a field for news volume). I made numerous attempts to combine the data with a mean-reversion signal but was unable to enhance the simple mean-reversion strategy or replicate Ravenpack’s results.

But there are other, simpler ways to potentially distinguish liquidity events and news events. I tried using volume information - for example, sorting by the ratio of five day volume over the mean-reversion period to average daily volume over a longer period. I wasn’t that successful using volume, but I found a more fruitful approach was to look at the pattern of returns. My conjecture was that a 10% one-day return is more likely to be news-related whereas five consecutive days of 2% returns each day is more likely to be liquidity related, given that there is some evidence that large liquidity trades take place over consecutive days. In fact, Heston, Korajczyk, and Sakda argue that large trades actually get executed not only on consecutive days but also at the same time each day.

There are many ways to penalize return patterns that are dominated by large one-day moves and reward steadier return patterns that have the same cumulative return. I only tried one simple, but obvious, filter: I sorted stocks by the five-day standard deviation of returns. This worked very well and was robust. The results were nearly monotonic when filtering by the five-day standard deviation. Nonetheless, other techniques may work better and achieve the same goal.

Trading on a universe of stocks where there is less uncertainty about fair value

Another enhancement is to use a universe of lower volatility stocks. This idea was presented at a UBS Quant Conference in 2013. The rationale is that when there is less uncertainty about a stock’s “fair value”, stock prices are more likely to reverse after large price moves that deviate from “fair value”.

The improvement was modest but robust: it worked for different trading frequencies as well as a different universe (UBS looked at a monthly reversal strategy and a universe of 1000 stocks in North America). And although the results were not always strictly monotonic, the higher volatility quantiles consistently performed worse than any other quantile, both in terms of Sharpe Ratio and in terms of returns as well.

Applying this concept to a low analyst dispersion universe performed even better, according to UBS. Their measure of dispersion was the standard deviation of analyst earnings estimates. The rationale is the same, and in fact, the two measures are correlated. Quantopian is in the process of incorporating a dataset of analyst earnings forecasts, and as soon as this data is available, I’ll post the results and the algorithm.

These strategies also highlight the idea of using data, like analysts earnings estimates, not as a signal per se, but in a totally different way – as a means to condition your alpha or modify the universe of stocks.

Miscellaneous Enhancements

One could also try to examine, and then separate, different types of stock-moving events using EventVestor data. For some events, like analysts upgrades and downgrades, announcement of stock buybacks, and unexpected changes in dividends, the market may underreact to news and fail to mean revert. On the other hand, the market may overreact to other events, like earnings announcements. As other posts have pointed out, Post Earnings Announcement Drift no longer seems to work, and indeed, I backtested the simple mean-reversion strategy excluding earnings announcements from the sample, and returns were cut in half (although the Sharpe Ratio was almost the same, because mean-reversion trades following earnings announcements, while positive, are also more volatile).

An academic paper by Avramov, Chordia, and Goyal suggests that stocks that are less liquid have stronger reversals. The argument is that the compensation demanded by liquidity providers is greater for less liquid stocks. I tested this out using a simple measure of liquidity, volume/(shares outstanding), and it worked reasonably well. Avramov et al. suggest a more sophisticated measure of liquidity, which I did not try but might be interesting to look at.

I looked at filtering out the extreme momentum deciles. The idea here is that some momentum stocks seem to move in one direction and when they reverse, also move in one direction. This filter resulted in only a modest improvement.

Finally, the last idea is to skip the most recent day when computing the five-day mean reversion. In other words, compute returns from five days ago up to the previous day’s close. Whether you believe the source of mean-reversion profits is from providing liquidity or from overreaction, it’s plausible that it takes longer than one day for the (non-high frequency trading) liquidity providers to step in with capital, or that the overreaction to the news cycle lasts more than one day. Indeed, I found a one-day mean-reversion strategy did not perform very well. But for whatever reason, skipping the most recent day significantly enhanced performance and was robust to all the variations I tried.

While I believe it makes sense to test alpha generating signals separately, ultimately the goal is to combine them. I didn’t focus on this aspect, but I did notice that when I combined several of these ideas, the results were dominated by two simple enhancements - filtering out return jumps and skipping the last day’s return. These two simple enhancements triple the cumulative returns and raise the Sharpe Ratio to 1.34 (see attached algorithm).

I should point out that I’ve only focused on alpha ideas, as opposed to other performance enhancing techniques that are surely done by practitioners, like coming up with smarter ways to unwind positions, improving upon equal weighting of stocks, sector or industry neutralizing the portfolio or employing portfolio optimization techniques (see this post on Optimize API ), etc.

These enhancements will hopefully spur more ideas. Even just a few of these ideas can increase the Sharpe Ratio of a simple mean-reversion strategy to relatively high levels.

2560
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""

This algorithm enhances a simple five-day mean reversion strategy by:
1. Skipping the last day's return
2. Sorting stocks based on the volatility of the five day return, to get steady moves vs jumpy ones
I also commented out two other filters that I looked at:
1. Six month volatility
2. Liquidity (volume/(shares outstanding))

"""

import numpy as np
import pandas as pd
from quantopian.pipeline import Pipeline
from quantopian.pipeline import CustomFactor
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume
from quantopian.pipeline.data import morningstar
from quantopian.pipeline.filters import Q500US

def initialize(context):

# Set benchmark to short-term Treasury note ETF (SHY) since strategy is dollar neutral
set_benchmark(sid(23911))

# Schedule our rebalance function to run at the end of each day.
schedule_function(my_rebalance, date_rules.every_day(), time_rules.market_close())

# Record variables at the end of each day.
schedule_function(my_record_vars, date_rules.every_day(), time_rules.market_close())

# Get intraday prices today before the close if you are not skipping the most recent data
schedule_function(get_prices,date_rules.every_day(), time_rules.market_close(minutes=5))

# Set commissions and slippage to 0 to determine pure alpha

# Number of quantiles for sorting returns for mean reversion
context.nq=5

# Number of quantiles for sorting volatility over five-day mean reversion period
context.nq_vol=3

# Create our pipeline and attach it to our algorithm.
my_pipe = make_pipeline()
attach_pipeline(my_pipe, 'my_pipeline')

class Volatility(CustomFactor):
inputs = [USEquityPricing.close]
window_length=132

def compute(self, today, assets, out, close):
# I compute 6-month volatility, starting before the five-day mean reversion period
daily_returns = np.log(close[1:-6]) - np.log(close[0:-7])
out[:] = daily_returns.std(axis = 0)

class Liquidity(CustomFactor):
inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding]
window_length = 1

def compute(self, today, assets, out, volume, shares):
out[:] = volume[-1]/shares[-1]

class Sector(CustomFactor):
inputs=[morningstar.asset_classification.morningstar_sector_code]
window_length=1

def compute(self, today, assets, out, sector):
out[:] = sector[-1]

def make_pipeline():
"""
Create our pipeline.
"""

pricing=USEquityPricing.close.latest

# Volatility filter (I made it sector neutral to replicate what UBS did).  Uncomment and
# change the percentile bounds as you would like before adding to 'universe'
# sector=morningstar.asset_classification.morningstar_sector_code.latest
# vol=vol.zscore(groupby=sector)
# vol_filter=vol.percentile_between(0,100)

# Liquidity filter (Uncomment and change the percentile bounds as you would like before
# adding to 'universe'
# I included NaN in liquidity filter because of the large amount of missing data for shares out
# liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan()

universe = (
Q500US()
& (pricing > 5)
# & liquidity_filter
# & volatility_filter
)

return Pipeline(
screen=universe
)

# Gets our pipeline output every day.
context.output = pipeline_output('my_pipeline')

def get_prices(context, data):
# Get the last 6 days of prices for every stock in our universe
Universe500=context.output.index.tolist()
prices = data.history(Universe500,'price',6,'1d')
daily_rets=np.log(prices/prices.shift(1))

rets=(prices.iloc[-2] - prices.iloc[0]) / prices.iloc[0]
# I used data.history instead of Pipeline to get historical prices so you can have the
# option of using the intraday price just before the close to get the most recent return.
# In my post, I argue that you generally get better results when you skip that return.
# If you don't want to skip the most recent return, however, use .iloc[-1] instead of .iloc[-2]:
# rets=(prices.iloc[-1] - prices.iloc[0]) / prices.iloc[0]

stdevs=daily_rets.std(axis=0)

rets_df=pd.DataFrame(rets,columns=['five_day_ret'])
stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret'])

context.output=context.output.join(rets_df,how='outer')
context.output=context.output.join(stdevs_df,how='outer')

context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1
context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1

context.longs=context.output[(context.output['ret_quantile']==1) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()
context.shorts=context.output[(context.output['ret_quantile']==context.nq) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()

def my_rebalance(context, data):
"""
Rebalance daily.
"""
Universe500=context.output.index.tolist()

existing_longs=0
existing_shorts=0
for security in context.portfolio.positions:
# Unwind stocks that have moved out of Q500US
if security not in Universe500 and data.can_trade(security):
order_target_percent(security, 0)
else:
current_quantile=context.output['ret_quantile'].loc[security]
if context.portfolio.positions[security].amount>0:
if (current_quantile==1) and (security not in context.longs):
existing_longs += 1
elif (current_quantile>1) and (security not in context.shorts):
order_target_percent(security, 0)
elif context.portfolio.positions[security].amount<0:
if (current_quantile==context.nq) and (security not in context.shorts):
existing_shorts += 1
elif (current_quantile<context.nq) and (security not in context.longs):
order_target_percent(security, 0)

for security in context.longs:
order_target_percent(security, .5/(len(context.longs)+existing_longs))

for security in context.shorts:
order_target_percent(security, -.5/(len(context.shorts)+existing_shorts))

def my_record_vars(context, data):
"""
Record variables at the end of each day.
"""
longs = shorts = 0
for position in context.portfolio.positions.itervalues():
if position.amount > 0:
longs += 1
elif position.amount < 0:
shorts += 1
# Record our variables.
record(leverage=context.account.leverage, long_count=longs, short_count=shorts)

log.info("Today's shorts: "  +", ".join([short_.symbol for short_ in context.shorts]))
log.info("Today's longs: "  +", ".join([long_.symbol for long_ in context.longs]))
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

76 responses

This strategy appears to be well suited for Quantopian's fund. Fully hedged, consistent, uncorrelated with the market and should have decent capacity.

Why is this algo is shared and not used by Quantopian, or kept for further paper trading? One issue I see is the flat performance in 2016. Is there other reasons?

Ordering at market open instead of market close here approaching 600 percentage points higher returns even though slippage/commissions are turned on. Main difference is order_value(security, context.portfolio.cash / len(context.longs)) to use up more of the unused cash, although, keep an eye on intraday leverage. The lower Sharpe and -3.5 Beta are surprises. Inactive yet sitting there is take_profit() with slope added. A few lines from that toward the end from an earlier run look decent however it might need more magic I haven't discovered yet:

2015-08-06 08:50 take_profit:94 INFO close 3535 HLF  cb 51.30  now 57.36  pnl 21409
2015-08-24 06:50 take_profit:94 INFO close -471 BMY  cb 63.76  now 56.241  pnl 3542
2015-08-24 09:50 take_profit:94 INFO close 2358 MBLY  cb 42.44  now 53.9  pnl 27027
2015-08-24 09:50 take_profit:94 INFO close 5127 YELP  cb 21.06  now 23.74  pnl 13737

315
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
This algorithm enhances a simple five-day mean reversion strategy by:
1. Skipping the last day's return
2. Sorting stocks based on the volatility of the five day return, to get steady moves vs jumpy ones
I also commented out two other filters that I looked at:
1. Six month volatility
2. Liquidity (volume/(shares outstanding))

"""
import numpy as np
import pandas as pd
from quantopian.pipeline import Pipeline
from quantopian.pipeline import CustomFactor
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume
from quantopian.pipeline.data import morningstar
from quantopian.pipeline.filters import Q500US

def initialize(context):

# Set benchmark to short-term Treasury note ETF (SHY) since strategy is dollar neutral
set_benchmark(sid(23911))

# Get intraday prices today before the close if you are not skipping the most recent data
schedule_function(get_prices,date_rules.every_day(), time_rules.market_open())

# Schedule rebalance function to run at the end of each day.
schedule_function(rebalance, date_rules.every_day(), time_rules.market_open())

# Record variables at the end of each day.
schedule_function(record_vars, date_rules.every_day(), time_rules.market_close())

# Set commissions and slippage to 0 to determine pure alpha

# Number of quantiles for sorting returns for mean reversion
context.nq=5

# Number of quantiles for sorting volatility over five-day mean reversion period
context.nq_vol=3

# Create pipeline and attach it to algorithm.
pipe = make_pipeline()
attach_pipeline(pipe, 'pipeline')

context.waits = {}
context.waits_max = 3    # trading days

return

context.pnl_sids_exclude  = [ sid(2561) ]
context.pnl_sids  = [  ]
context.day_count = 0
#schedule_function(record_pnl, date_rules.every_day(), time_rules.market_close())

# Take profit several times a day
context.profit_threshold = .11
context.profit_logging   = 1
for i in range(20, 390, 60):    # (low, high, every i minutes)
#continue    # uncomment to deactivate
schedule_function(take_profit, date_rules.every_day(), time_rules.market_open(minutes=i))

def wait(c, sec=None, action=None):
if sec and action:
if action == 1:
c.waits[sec] = 1    # start wait
elif action == 0:
del c.waits[sec]    # end wait
else:
for sec in c.waits.copy():
if c.waits[sec] > c.waits_max:
del c.waits[sec]
else:
c.waits[sec] += 1   # increment

def take_profit(context, data):    # Close some positions to take profit
pos     = context.portfolio.positions
history = data.history(pos.keys(), 'close', 10, '1m').bfill().ffill()
for s in pos:
if not data.can_trade(s):      continue
if slope(history[s])      > 0: continue
if slope(history[s][-5:]) > 0: continue
if history[s][-1] > history[s][-2]: continue
prc = data.current(s, 'price')
amt = pos[s].amount
if (amt / abs(amt)) * ((prc / pos[s].cost_basis) - 1) > context.profit_threshold:
order_target(s, 0)
wait(context, s, 1)    # start wait
if not context.profit_logging: continue
pnl = (amt * (prc - pos[s].cost_basis))
if pnl < 3000: continue
log.info('close {} {}  cb {}  now {}  pnl {}'.format(
amt, s.symbol, '%.2f' % pos[s].cost_basis, prc, '%.0f' % pnl))

import statsmodels.api as sm
def slope(in_list):     # Return slope of regression line. [Make sure this list contains no nans]
return sm.OLS(in_list, sm.add_constant(range(-len(in_list) + 1, 1))).fit().params[-1]  # slope

def record_pnl(context, data):
def _pnl_value(sec, context, data):
pos = context.portfolio.positions[sec]
return pos.amount * (data.current(sec, 'price') - pos.cost_basis)

context.day_count += 1

for s in context.portfolio.positions:
if not data.can_trade(s): continue
if s in context.pnl_sids_exclude: continue

# periodically log all
if context.day_count % 126 == 0:
log.info('{} {}'.format(s.symbol, int(_pnl_value(s, context, data))))

# add up to 5 securities for record
if len(context.pnl_sids) < 5 and s not in context.pnl_sids:
context.pnl_sids.append(s)
if s not in context.pnl_sids: continue     # limit to only them

# record their profit and loss
who  = s.symbol
what = _pnl_value(s, context, data)
record( **{ who: what } )

class Volatility(CustomFactor):
inputs = [USEquityPricing.close]
window_length=132

def compute(self, today, assets, out, close):
# I compute 6-month volatility, starting before the five-day mean reversion period
daily_returns = np.log(close[1:-6]) - np.log(close[0:-7])
out[:] = daily_returns.std(axis = 0)

class Liquidity(CustomFactor):
inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding]
window_length = 1

def compute(self, today, assets, out, volume, shares):
out[:] = volume[-1]/shares[-1]

class Sector(CustomFactor):
inputs=[morningstar.asset_classification.morningstar_sector_code]
window_length=1

def compute(self, today, assets, out, sector):
out[:] = sector[-1]

def make_pipeline():
"""
Create pipeline.
"""

pricing=USEquityPricing.close.latest

# Volatility filter (I made it sector neutral to replicate what UBS did).  Uncomment and
# change the percentile bounds as you would like before adding to 'universe'
# sector=morningstar.asset_classification.morningstar_sector_code.latest
# vol=vol.zscore(groupby=sector)
# vol_filter=vol.percentile_between(0,100)

# Liquidity filter (Uncomment and change the percentile bounds as you would like before
# adding to 'universe'
# I included NaN in liquidity filter because of the large amount of missing data for shares out
# liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan()

universe = (
Q500US()
& (pricing > 5)
# & liquidity_filter
# & volatility_filter
)
return Pipeline(
screen  = universe
)

# Gets pipeline output every day.
context.output = pipeline_output('pipeline')

wait(context)    # Increment any that are present

def get_prices(context, data):
# Get the last 6 days of prices for every stock in universe
Universe500=context.output.index.tolist()
prices = data.history(Universe500,'price',6,'1d')
daily_rets = np.log(prices/prices.shift(1))

rets=(prices.iloc[-2] - prices.iloc[0]) / prices.iloc[0]
# I used data.history instead of Pipeline to get historical prices so you can have the
# option of using the intraday price just before the close to get the most recent return.
# In my post, I argue that you generally get better results when you skip that return.
# If you don't want to skip the most recent return, however, use .iloc[-1] instead of .iloc[-2]:
# rets=(prices.iloc[-1] - prices.iloc[0]) / prices.iloc[0]

stdevs=daily_rets.std(axis=0)

rets_df=pd.DataFrame(rets,columns=['five_day_ret'])
stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret'])

context.output=context.output.join(rets_df,how='outer')
context.output=context.output.join(stdevs_df,how='outer')

context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1
context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1

context.longs=context.output[(context.output['ret_quantile']==1) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()
context.shorts=context.output[(context.output['ret_quantile']==context.nq) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()

def rebalance(context, data):
"""
Rebalance daily.
"""
Universe500=context.output.index.tolist()

existing_longs=0
existing_shorts=0
for security in context.portfolio.positions:
# Unwind stocks that have moved out of Q500US
if security not in Universe500 and data.can_trade(security):
order_target(security, 0)
else:
current_quantile=context.output['ret_quantile'].loc[security]
if context.portfolio.positions[security].amount>0:
if (current_quantile==1) and (security not in context.longs):
existing_longs += 1
elif (current_quantile>1) and (security not in context.shorts):
order_target(security, 0)
elif context.portfolio.positions[security].amount<0:
if (current_quantile==context.nq) and (security not in context.shorts):
existing_shorts += 1
elif (current_quantile<context.nq) and (security not in context.longs):
order_target(security, 0)

order_sids = get_open_orders().keys()
for security in context.longs:
if security in context.waits: continue
if security in order_sids:    continue
order_value(security, context.portfolio.cash / len(context.longs) )

for security in context.shorts:
if security in context.waits: continue
if security in order_sids:    continue
order_target_percent(security, -.5/(len(context.shorts)+existing_shorts))

def record_vars(context, data):
"""
Record variables at the end of each day.

longs = shorts = 0
for position in context.portfolio.positions.itervalues():
if position.amount > 0:
longs += 1
elif position.amount < 0:
shorts += 1
"""

record(
leverage=context.account.leverage,
long_count  = len(context.longs),
short_count = len(context.shorts),
)

#log.info("Today's shorts: "  +", ".join([short_.symbol for short_ in context.shorts]))
#log.info("Today's longs: "  +", ".join([long_.symbol for long_ in context.longs]))

def pvr(context, data):
''' Custom chart and/or logging of profit_vs_risk returns and related information
'''
# # # # # # # # # #  Options  # # # # # # # # # #
logging         = 0            # Info to logging window with some new maximums

record_pvr      = 1            # Profit vs Risk returns (percentage)
record_pvrp     = 0            # PvR (p)roportional neg cash vs portfolio value
record_cash     = 1            # Cash available
record_max_lvrg = 1            # Maximum leverage encountered
record_risk_hi  = 0            # Highest risk overall
record_shorting = 0            # Total value of any shorts
record_max_shrt = 1            # Max value of shorting total
record_cash_low = 1            # Any new lowest cash level
record_q_return = 0            # Quantopian returns (percentage)
record_pnl      = 0            # Profit-n-Loss
record_risk     = 0            # Risked, max cash spent or shorts beyond longs+cash
record_leverage = 0            # Leverage (context.account.leverage)
record_overshrt = 0            # Shorts beyond longs+cash
if record_pvrp: record_pvr = 0 # if pvrp is active, straight pvr is off

import time
from datetime import datetime
from pytz import timezone      # Python will only do once, makes this portable.
#   Move to top of algo for better efficiency.
c = context  # Brevity is the soul of wit -- Shakespeare [for readability]
if 'pvr' not in c:
date_strt = get_environment('start').date()
date_end  = get_environment('end').date()
cash_low  = c.portfolio.starting_cash
c.cagr    = 0.0
c.pvr     = {
'pvr'        : 0,      # Profit vs Risk returns based on maximum spent
'max_lvrg'   : 0,
'max_shrt'   : 0,
'risk_hi'    : 0,
'days'       : 0.0,
'date_prv'   : '',
'date_end'   : date_end,
'cash_low'   : cash_low,
'cash'       : cash_low,
'start'      : cash_low,
'pstart'     : c.portfolio.portfolio_value, # Used if restart
'begin'      : time.time(),                 # For run time
'log_summary': 126,                         # Summary every x days
'run_str'    : '{} to {}  ${} {} US/Eastern'.format(date_strt, date_end, int(cash_low), datetime.now(timezone('US/Eastern')).strftime("%Y-%m-%d %H:%M")) } log.info(c.pvr['run_str']) def _pvr(c): c.cagr = ((c.portfolio.portfolio_value / c.pvr['start']) ** (1 / (c.pvr['days'] / 252.))) - 1 ptype = 'PvR' if record_pvr else 'PvRp' log.info('{} {} %/day cagr {} Portfolio value {} PnL {}'.format(ptype, '%.4f' % (c.pvr['pvr'] / c.pvr['days']), '%.1f' % c.cagr, '%.0f' % c.portfolio.portfolio_value, '%.0f' % (c.portfolio.portfolio_value - c.pvr['start']))) log.info(' Profited {} on {} activated/transacted for PvR of {}%'.format('%.0f' % (c.portfolio.portfolio_value - c.pvr['start']), '%.0f' % c.pvr['risk_hi'], '%.1f' % c.pvr['pvr'])) log.info(' QRet {} PvR {} CshLw {} MxLv {} RskHi {} MxShrt {}'.format('%.2f' % q_rtrn, '%.2f' % c.pvr['pvr'], '%.0f' % c.pvr['cash_low'], '%.2f' % c.pvr['max_lvrg'], '%.0f' % c.pvr['risk_hi'], '%.0f' % c.pvr['max_shrt'])) def _minut(): dt = get_datetime().astimezone(timezone('US/Eastern')) return str((dt.hour * 60) + dt.minute - 570).rjust(3) # (-570 = 9:31a) date = get_datetime().date() if c.pvr['date_prv'] != date: c.pvr['date_prv'] = date c.pvr['days'] += 1.0 do_summary = 0 if c.pvr['log_summary'] and c.pvr['days'] % c.pvr['log_summary'] == 0 and _minut() == '100': do_summary = 1 # Log summary every x days if do_summary or date == c.pvr['date_end']: c.pvr['cash'] = c.portfolio.cash elif c.pvr['cash'] == c.portfolio.cash and not logging: return # for speed longs = sum([p.amount * p.last_sale_price for s, p in c.portfolio.positions.items() if p.amount > 0]) shorts = sum([p.amount * p.last_sale_price for s, p in c.portfolio.positions.items() if p.amount < 0]) q_rtrn = 100 * (c.portfolio.portfolio_value - c.pvr['start']) / c.pvr['start'] cash = c.portfolio.cash new_risk_hi = 0 new_max_lv = 0 new_max_shrt = 0 new_cash_low = 0 # To trigger logging in cash_low case overshorts = 0 # Shorts value beyond longs plus cash cash_dip = int(max(0, c.pvr['pstart'] - cash)) risk = int(max(cash_dip, -shorts)) if record_pvrp and cash < 0: # Let negative cash ding less when portfolio is up. cash_dip = int(max(0, c.pvr['start'] - cash * c.pvr['start'] / c.portfolio.portfolio_value)) # Imagine: Start with 10, grows to 1000, goes negative to -10, should not be 200% risk. if int(cash) < c.pvr['cash_low']: # New cash low new_cash_low = 1 c.pvr['cash_low'] = int(cash) # Lowest cash level hit if record_cash_low: record(CashLow = c.pvr['cash_low']) if c.account.leverage > c.pvr['max_lvrg']: new_max_lv = 1 c.pvr['max_lvrg'] = c.account.leverage # Maximum intraday leverage if record_max_lvrg: record(MaxLv = c.pvr['max_lvrg']) if shorts < c.pvr['max_shrt']: new_max_shrt = 1 c.pvr['max_shrt'] = shorts # Maximum shorts value if record_max_shrt: record(MxShrt = c.pvr['max_shrt']) if risk > c.pvr['risk_hi']: new_risk_hi = 1 c.pvr['risk_hi'] = risk # Highest risk overall if record_risk_hi: record(RiskHi = c.pvr['risk_hi']) # Profit_vs_Risk returns based on max amount actually spent (risk high) if c.pvr['risk_hi'] != 0: # Avoid zero-divide c.pvr['pvr'] = 100 * (c.portfolio.portfolio_value - c.pvr['start']) / c.pvr['risk_hi'] ptype = 'PvRp' if record_pvrp else 'PvR' if record_pvr or record_pvrp: record(**{ptype: c.pvr['pvr']}) if shorts > longs + cash: overshorts = shorts # Shorts when too high if record_shorting: record(Shorts = shorts) # Shorts value as a positve if record_overshrt: record(OvrShrt = overshorts) # Shorts beyond payable if record_leverage: record(Lvrg = c.account.leverage) # Leverage if record_cash: record(Cash = cash) # Cash if record_risk: record(Risk = risk) # Amount in play, maximum of shorts or cash used if record_q_return: record(QRet = q_rtrn) # Quantopian returns to compare to pvr returns curve if record_pnl: record(PnL = c.portfolio.portfolio_value - c.pvr['start']) # Profit|Loss if logging and (new_risk_hi or new_cash_low or new_max_lv or new_max_shrt): csh = ' Cash ' + '%.0f' % cash risk = ' Risk ' + '%.0f' % risk qret = ' QRet ' + '%.1f' % q_rtrn shrt = ' Shrt ' + '%.0f' % shorts ovrshrt = ' oShrt ' + '%.0f' % overshorts lv = ' Lv ' + '%.1f' % c.account.leverage pvr = ' PvR ' + '%.1f' % c.pvr['pvr'] rsk_hi = ' RskHi ' + '%.0f' % c.pvr['risk_hi'] csh_lw = ' CshLw ' + '%.0f' % c.pvr['cash_low'] mxlv = ' MxLv ' + '%.2f' % c.pvr['max_lvrg'] mxshrt = ' MxShrt ' + '%.0f' % c.pvr['max_shrt'] pnl = ' PnL ' + '%.0f' % (c.portfolio.portfolio_value - c.pvr['start']) log.info('{}{}{}{}{}{}{}{}{}{}{}{}{}'.format(_minut(), lv, mxlv, qret, pvr, pnl, csh, csh_lw, shrt, mxshrt, ovrshrt, risk, rsk_hi)) if do_summary: _pvr(c) if get_datetime() == get_environment('end'): # Summary at end of run _pvr(c) elapsed = (time.time() - c.pvr['begin']) / 60 # minutes log.info( '{}\nRuntime {} hr {} min'.format(c.pvr['run_str'], int(elapsed / 60), '%.1f' % (elapsed % 60))) #def handle_data(context, data): # pvr(context, data)  There was a runtime error. Here's the tear sheet for Rob's algo above, except with the default benchmark. One question is, shouldn't there be a lower-level analysis (e.g. autocorrelation) that might shed some light on things here? For example, given the short-term nature of the effect, could one use minute data, and perform autocorrelations on the time series data, to get a sense for the reversion time scale versus time? It seems fundamentally, mean reversion strategies are relying on 'memory' or 'stickiness' of prices, versus a pure random walk. So in the context of stock prices, are there standard ways of measuring the strength and time scale of the effect, across broad universes, versus time? Perhaps some overall mean reversion figure-of-merit could be devised, that would give some sense for the overall trend in the potential profitability of mean reversion factors. 36 Loading notebook preview... Notebook previews are currently unavailable. A little boost by changing to: profitable = morningstar.valuation_ratios.ev_to_ebitda.latest > 0 universe = ( Q500US() & (pricing > 5) & profitable # & liquidity_filter # & volatility_filter )  259 Loading... Backtest from to with initial capital Total Returns -- Alpha -- Beta -- Sharpe -- Sortino -- Max Drawdown -- Benchmark Returns -- Volatility --  Returns 1 Month 3 Month 6 Month 12 Month  Alpha 1 Month 3 Month 6 Month 12 Month  Beta 1 Month 3 Month 6 Month 12 Month  Sharpe 1 Month 3 Month 6 Month 12 Month  Sortino 1 Month 3 Month 6 Month 12 Month  Volatility 1 Month 3 Month 6 Month 12 Month  Max Drawdown 1 Month 3 Month 6 Month 12 Month """ This algorithm enhances a simple five-day mean reversion strategy by: 1. Skipping the last day's return 2. Sorting stocks based on the volatility of the five day return, to get steady moves vs jumpy ones I also commented out two other filters that I looked at: 1. Six month volatility 2. Liquidity (volume/(shares outstanding)) """ import numpy as np import pandas as pd from quantopian.pipeline import Pipeline from quantopian.pipeline import CustomFactor from quantopian.algorithm import attach_pipeline, pipeline_output from quantopian.pipeline.data.builtin import USEquityPricing from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume from quantopian.pipeline.data import morningstar from quantopian.pipeline.filters import Q500US def initialize(context): # Set benchmark to short-term Treasury note ETF (SHY) since strategy is dollar neutral set_benchmark(sid(23911)) # Schedule our rebalance function to run at the end of each day. schedule_function(my_rebalance, date_rules.every_day(), time_rules.market_close()) # Record variables at the end of each day. schedule_function(my_record_vars, date_rules.every_day(), time_rules.market_close()) # Get intraday prices today before the close if you are not skipping the most recent data schedule_function(get_prices,date_rules.every_day(), time_rules.market_close(minutes=5)) # Set commissions and slippage to 0 to determine pure alpha set_commission(commission.PerShare(cost=0, min_trade_cost=0)) set_slippage(slippage.FixedSlippage(spread=0)) # Number of quantiles for sorting returns for mean reversion context.nq=5 # Number of quantiles for sorting volatility over five-day mean reversion period context.nq_vol=3 # Create our pipeline and attach it to our algorithm. my_pipe = make_pipeline() attach_pipeline(my_pipe, 'my_pipeline') class Volatility(CustomFactor): inputs = [USEquityPricing.close] window_length=132 def compute(self, today, assets, out, close): # I compute 6-month volatility, starting before the five-day mean reversion period daily_returns = np.log(close[1:-6]) - np.log(close[0:-7]) out[:] = daily_returns.std(axis = 0) class Liquidity(CustomFactor): inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding] window_length = 1 def compute(self, today, assets, out, volume, shares): out[:] = volume[-1]/shares[-1] class Sector(CustomFactor): inputs=[morningstar.asset_classification.morningstar_sector_code] window_length=1 def compute(self, today, assets, out, sector): out[:] = sector[-1] def make_pipeline(): """ Create our pipeline. """ pricing=USEquityPricing.close.latest # Volatility filter (I made it sector neutral to replicate what UBS did). Uncomment and # change the percentile bounds as you would like before adding to 'universe' # vol=Volatility(mask=Q500US()) # sector=morningstar.asset_classification.morningstar_sector_code.latest # vol=vol.zscore(groupby=sector) # vol_filter=vol.percentile_between(0,100) # Liquidity filter (Uncomment and change the percentile bounds as you would like before # adding to 'universe' # liquidity=Liquidity(mask=Q500US()) # I included NaN in liquidity filter because of the large amount of missing data for shares out # liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan() profitable = morningstar.valuation_ratios.ev_to_ebitda.latest > 0 universe = ( Q500US() & (pricing > 5) & profitable # & liquidity_filter # & volatility_filter ) return Pipeline( screen=universe ) def before_trading_start(context, data): # Gets our pipeline output every day. context.output = pipeline_output('my_pipeline') def get_prices(context, data): # Get the last 6 days of prices for every stock in our universe Universe500=context.output.index.tolist() prices = data.history(Universe500,'price',6,'1d') daily_rets=np.log(prices/prices.shift(1)) rets=(prices.iloc[-2] - prices.iloc[0]) / prices.iloc[0] # I used data.history instead of Pipeline to get historical prices so you can have the # option of using the intraday price just before the close to get the most recent return. # In my post, I argue that you generally get better results when you skip that return. # If you don't want to skip the most recent return, however, use .iloc[-1] instead of .iloc[-2]: # rets=(prices.iloc[-1] - prices.iloc[0]) / prices.iloc[0] stdevs=daily_rets.std(axis=0) rets_df=pd.DataFrame(rets,columns=['five_day_ret']) stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret']) context.output=context.output.join(rets_df,how='outer') context.output=context.output.join(stdevs_df,how='outer') context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1 context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1 context.longs=context.output[(context.output['ret_quantile']==1) & (context.output['stdev_quantile']<context.nq_vol)].index.tolist() context.shorts=context.output[(context.output['ret_quantile']==context.nq) & (context.output['stdev_quantile']<context.nq_vol)].index.tolist() def my_rebalance(context, data): """ Rebalance daily. """ Universe500=context.output.index.tolist() existing_longs=0 existing_shorts=0 for security in context.portfolio.positions: # Unwind stocks that have moved out of Q500US if security not in Universe500 and data.can_trade(security): order_target_percent(security, 0) else: if data.can_trade(security): current_quantile=context.output['ret_quantile'].loc[security] if context.portfolio.positions[security].amount>0: if (current_quantile==1) and (security not in context.longs): existing_longs += 1 elif (current_quantile>1) and (security not in context.shorts): order_target_percent(security, 0) elif context.portfolio.positions[security].amount<0: if (current_quantile==context.nq) and (security not in context.shorts): existing_shorts += 1 elif (current_quantile<context.nq) and (security not in context.longs): order_target_percent(security, 0) for security in context.longs: if data.can_trade(security): order_target_percent(security, .5/(len(context.longs)+existing_longs)) for security in context.shorts: if data.can_trade(security): order_target_percent(security, -.5/(len(context.shorts)+existing_shorts)) def my_record_vars(context, data): """ Record variables at the end of each day. """ longs = shorts = 0 for position in context.portfolio.positions.itervalues(): if position.amount > 0: longs += 1 elif position.amount < 0: shorts += 1 # Record our variables. record(leverage=context.account.leverage, long_count=longs, short_count=shorts) log.info("Today's shorts: " +", ".join([short_.symbol for short_ in context.shorts])) log.info("Today's longs: " +", ".join([long_.symbol for long_ in context.longs])) There was a runtime error. Wow Rob, this is one of my favourite posts ever. Great work! I wanted to point out to others than this backtest has trading costs and slippage at zero, hence the excellent return distribution. I've has some luck with reducing the trading costs by a) trading only the most liquid stocks and b) trying portfolio rebalancing strategies that achieve something similar but with lower turnover c) going long only and using a SPY hedge. Another good academic paper is "Evaporating Liquidity" by Nagel 2011: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1971476 The returns of short-term reversal strategies in equity markets can be interpreted as a proxy for the returns from liquidity provision. Analysis of reversal strategies shows that the expected return from liquidity provision is strongly time-varying and highly predictable with the VIX index. Expected returns and conditional Sharpe Ratios increase enormously along with the VIX during times of nancial market turmoil, such as the nancial crisis 2007-09. Even reversal strategies formed from industry portfolios (which do not yield high returns unconditionally) produce high rates of return and high Sharpe Ratios during times of high VIX. The results point to withdrawal of liquidity supply, and an associated increase in the expected returns from liquidity provision, as a main driver behind the evaporation of liquidity during times of nancial market turmoil, consistent with theories of liquidity provision by nancially constrained intermediaries Great post! I've used similar strategies, still to be coded for larger datasets. I've focused on the causality of mean reversion and tried to analyse the trend in factors that generate movements in the hedged portfolio's residual. There is also the effect of autocorrelation on each asset, using a VAR model may help include lagged observations rather than just relying on instantaneous flow of information into prices. Blue, you could do really minor changes to improve the picture. Only three numbers, and since you do charge for commissions and slippage... Your net liquidating value holds. 367 Loading... Backtest from to with initial capital Total Returns -- Alpha -- Beta -- Sharpe -- Sortino -- Max Drawdown -- Benchmark Returns -- Volatility --  Returns 1 Month 3 Month 6 Month 12 Month  Alpha 1 Month 3 Month 6 Month 12 Month  Beta 1 Month 3 Month 6 Month 12 Month  Sharpe 1 Month 3 Month 6 Month 12 Month  Sortino 1 Month 3 Month 6 Month 12 Month  Volatility 1 Month 3 Month 6 Month 12 Month  Max Drawdown 1 Month 3 Month 6 Month 12 Month """ Blue's version. This algorithm enhances a simple five-day mean reversion strategy by: 1. Skipping the last day's return 2. Sorting stocks based on the volatility of the five day return, to get steady moves vs jumpy ones I also commented out two other filters that I looked at: 1. Six month volatility 2. Liquidity (volume/(shares outstanding)) """ import numpy as np import pandas as pd from quantopian.pipeline import Pipeline from quantopian.pipeline import CustomFactor from quantopian.algorithm import attach_pipeline, pipeline_output from quantopian.pipeline.data.builtin import USEquityPricing from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume from quantopian.pipeline.data import morningstar from quantopian.pipeline.filters import Q500US def initialize(context): # Set benchmark to short-term Treasury note ETF (SHY) since strategy is dollar neutral set_benchmark(sid(23911)) # Get intraday prices today before the close if you are not skipping the most recent data schedule_function(get_prices,date_rules.every_day(), time_rules.market_open(minutes=5)) # Schedule rebalance function to run at the end of each day. schedule_function(rebalance, date_rules.every_day(), time_rules.market_open(minutes=10)) # Record variables at the end of each day. schedule_function(record_vars, date_rules.every_day(), time_rules.market_close()) # Set commissions and slippage to 0 to determine pure alpha #set_commission(commission.PerShare(cost=0, min_trade_cost=0)) #set_slippage(slippage.FixedSlippage(spread=0)) # Number of quantiles for sorting returns for mean reversion context.nq=5 # Number of quantiles for sorting volatility over five-day mean reversion period context.nq_vol=3 # Create pipeline and attach it to algorithm. pipe = make_pipeline() attach_pipeline(pipe, 'pipeline') context.waits = {} context.waits_max = 3 # trading days return context.pnl_sids_exclude = [ sid(2561) ] context.pnl_sids = [ ] context.day_count = 0 #schedule_function(record_pnl, date_rules.every_day(), time_rules.market_close()) # Take profit several times a day context.profit_threshold = .05 context.profit_logging = 1 for i in range(20, 390, 60): # (low, high, every i minutes) #continue # uncomment to deactivate schedule_function(take_profit, date_rules.every_day(), time_rules.market_open(minutes=i)) def wait(c, sec=None, action=None): if sec and action: if action == 1: c.waits[sec] = 1 # start wait elif action == 0: del c.waits[sec] # end wait else: for sec in c.waits.copy(): if c.waits[sec] > c.waits_max: del c.waits[sec] else: c.waits[sec] += 1 # increment def take_profit(context, data): # Close some positions to take profit pos = context.portfolio.positions history = data.history(pos.keys(), 'close', 10, '1m').bfill().ffill() for s in pos: if not data.can_trade(s): continue if slope(history[s]) > 0: continue if slope(history[s][-5:]) > 0: continue if history[s][-1] > history[s][-2]: continue prc = data.current(s, 'price') amt = pos[s].amount if (amt / abs(amt)) * ((prc / pos[s].cost_basis) - 1) > context.profit_threshold: order_target(s, 0) wait(context, s, 1) # start wait if not context.profit_logging: continue pnl = (amt * (prc - pos[s].cost_basis)) if pnl < 3000: continue log.info('close {} {} cb {} now {} pnl {}'.format( amt, s.symbol, '%.2f' % pos[s].cost_basis, prc, '%.0f' % pnl)) import statsmodels.api as sm def slope(in_list): # Return slope of regression line. [Make sure this list contains no nans] return sm.OLS(in_list, sm.add_constant(range(-len(in_list) + 1, 1))).fit().params[-1] # slope def record_pnl(context, data): def _pnl_value(sec, context, data): pos = context.portfolio.positions[sec] return pos.amount * (data.current(sec, 'price') - pos.cost_basis) context.day_count += 1 for s in context.portfolio.positions: if not data.can_trade(s): continue if s in context.pnl_sids_exclude: continue # periodically log all if context.day_count % 126 == 0: log.info('{} {}'.format(s.symbol, int(_pnl_value(s, context, data)))) # add up to 5 securities for record if len(context.pnl_sids) < 5 and s not in context.pnl_sids: context.pnl_sids.append(s) if s not in context.pnl_sids: continue # limit to only them # record their profit and loss who = s.symbol what = _pnl_value(s, context, data) record( **{ who: what } ) class Volatility(CustomFactor): inputs = [USEquityPricing.close] window_length=132 def compute(self, today, assets, out, close): # I compute 6-month volatility, starting before the five-day mean reversion period daily_returns = np.log(close[1:-6]) - np.log(close[0:-7]) out[:] = daily_returns.std(axis = 0) class Liquidity(CustomFactor): inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding] window_length = 1 def compute(self, today, assets, out, volume, shares): out[:] = volume[-1]/shares[-1] class Sector(CustomFactor): inputs=[morningstar.asset_classification.morningstar_sector_code] window_length=1 def compute(self, today, assets, out, sector): out[:] = sector[-1] def make_pipeline(): """ Create pipeline. """ pricing=USEquityPricing.close.latest # Volatility filter (I made it sector neutral to replicate what UBS did). Uncomment and # change the percentile bounds as you would like before adding to 'universe' # vol=Volatility(mask=Q500US()) # sector=morningstar.asset_classification.morningstar_sector_code.latest # vol=vol.zscore(groupby=sector) # vol_filter=vol.percentile_between(0,100) # Liquidity filter (Uncomment and change the percentile bounds as you would like before # adding to 'universe' # liquidity=Liquidity(mask=Q500US()) # I included NaN in liquidity filter because of the large amount of missing data for shares out # liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan() universe = ( Q500US() & (pricing > 5) # & liquidity_filter # & volatility_filter ) return Pipeline( screen = universe ) def before_trading_start(context, data): # Gets pipeline output every day. context.output = pipeline_output('pipeline') wait(context) # Increment any that are present def get_prices(context, data): # Get the last 6 days of prices for every stock in universe Universe500=context.output.index.tolist() prices = data.history(Universe500,'price',6,'1d') daily_rets = np.log(prices/prices.shift(1)) rets=(prices.iloc[-2] - prices.iloc[0]) / prices.iloc[0] # I used data.history instead of Pipeline to get historical prices so you can have the # option of using the intraday price just before the close to get the most recent return. # In my post, I argue that you generally get better results when you skip that return. # If you don't want to skip the most recent return, however, use .iloc[-1] instead of .iloc[-2]: # rets=(prices.iloc[-1] - prices.iloc[0]) / prices.iloc[0] stdevs=daily_rets.std(axis=0) rets_df=pd.DataFrame(rets,columns=['five_day_ret']) stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret']) context.output=context.output.join(rets_df,how='outer') context.output=context.output.join(stdevs_df,how='outer') context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1 context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1 context.longs=context.output[(context.output['ret_quantile']==1) & (context.output['stdev_quantile']<context.nq_vol)].index.tolist() context.shorts=context.output[(context.output['ret_quantile']==context.nq) & (context.output['stdev_quantile']<context.nq_vol)].index.tolist() def rebalance(context, data): """ Rebalance daily. """ Universe500=context.output.index.tolist() existing_longs=0 existing_shorts=0 for security in context.portfolio.positions: # Unwind stocks that have moved out of Q500US if security not in Universe500 and data.can_trade(security): order_target(security, 0) else: if data.can_trade(security): current_quantile=context.output['ret_quantile'].loc[security] if context.portfolio.positions[security].amount>0: if (current_quantile==1) and (security not in context.longs): existing_longs += 1 elif (current_quantile>1) and (security not in context.shorts): order_target(security, 0) elif context.portfolio.positions[security].amount<0: if (current_quantile==context.nq) and (security not in context.shorts): existing_shorts += 1 elif (current_quantile<context.nq) and (security not in context.longs): order_target(security, 0) order_sids = get_open_orders().keys() for security in context.longs: if security in context.waits: continue if security in order_sids: continue if data.can_trade(security): order_value(security, context.portfolio.cash / len(context.longs) ) for security in context.shorts: if security in context.waits: continue if security in order_sids: continue if data.can_trade(security): order_target_percent(security, -.5/(len(context.shorts)+existing_shorts)) def record_vars(context, data): """ Record variables at the end of each day. longs = shorts = 0 for position in context.portfolio.positions.itervalues(): if position.amount > 0: longs += 1 elif position.amount < 0: shorts += 1 """ record( leverage=context.account.leverage, long_count = len(context.longs), short_count = len(context.shorts), ) #log.info("Today's shorts: " +", ".join([short_.symbol for short_ in context.shorts])) #log.info("Today's longs: " +", ".join([long_.symbol for long_ in context.longs])) def pvr(context, data): ''' Custom chart and/or logging of profit_vs_risk returns and related information ''' # # # # # # # # # # Options # # # # # # # # # # logging = 0 # Info to logging window with some new maximums record_pvr = 1 # Profit vs Risk returns (percentage) record_pvrp = 0 # PvR (p)roportional neg cash vs portfolio value record_cash = 1 # Cash available record_max_lvrg = 1 # Maximum leverage encountered record_risk_hi = 0 # Highest risk overall record_shorting = 0 # Total value of any shorts record_max_shrt = 1 # Max value of shorting total record_cash_low = 1 # Any new lowest cash level record_q_return = 0 # Quantopian returns (percentage) record_pnl = 0 # Profit-n-Loss record_risk = 0 # Risked, max cash spent or shorts beyond longs+cash record_leverage = 0 # Leverage (context.account.leverage) record_overshrt = 0 # Shorts beyond longs+cash if record_pvrp: record_pvr = 0 # if pvrp is active, straight pvr is off import time from datetime import datetime from pytz import timezone # Python will only do once, makes this portable. # Move to top of algo for better efficiency. c = context # Brevity is the soul of wit -- Shakespeare [for readability] if 'pvr' not in c: date_strt = get_environment('start').date() date_end = get_environment('end').date() cash_low = c.portfolio.starting_cash c.cagr = 0.0 c.pvr = { 'pvr' : 0, # Profit vs Risk returns based on maximum spent 'max_lvrg' : 0, 'max_shrt' : 0, 'risk_hi' : 0, 'days' : 0.0, 'date_prv' : '', 'date_end' : date_end, 'cash_low' : cash_low, 'cash' : cash_low, 'start' : cash_low, 'pstart' : c.portfolio.portfolio_value, # Used if restart 'begin' : time.time(), # For run time 'log_summary': 126, # Summary every x days 'run_str' : '{} to {}${}  {} US/Eastern'.format(date_strt, date_end, int(cash_low), datetime.now(timezone('US/Eastern')).strftime("%Y-%m-%d %H:%M"))
}
log.info(c.pvr['run_str'])

def _pvr(c):
c.cagr = ((c.portfolio.portfolio_value / c.pvr['start']) ** (1 / (c.pvr['days'] / 252.))) - 1
ptype = 'PvR' if record_pvr else 'PvRp'
log.info('{} {} %/day   cagr {}   Portfolio value {}   PnL {}'.format(ptype, '%.4f' % (c.pvr['pvr'] / c.pvr['days']), '%.1f' % c.cagr, '%.0f' % c.portfolio.portfolio_value, '%.0f' % (c.portfolio.portfolio_value - c.pvr['start'])))
log.info('  Profited {} on {} activated/transacted for PvR of {}%'.format('%.0f' % (c.portfolio.portfolio_value - c.pvr['start']), '%.0f' % c.pvr['risk_hi'], '%.1f' % c.pvr['pvr']))
log.info('  QRet {} PvR {} CshLw {} MxLv {} RskHi {} MxShrt {}'.format('%.2f' % q_rtrn, '%.2f' % c.pvr['pvr'], '%.0f' % c.pvr['cash_low'], '%.2f' % c.pvr['max_lvrg'], '%.0f' % c.pvr['risk_hi'], '%.0f' % c.pvr['max_shrt']))

def _minut():
dt = get_datetime().astimezone(timezone('US/Eastern'))
return str((dt.hour * 60) + dt.minute - 570).rjust(3)  # (-570 = 9:31a)

date = get_datetime().date()
if c.pvr['date_prv'] != date:
c.pvr['date_prv'] = date
c.pvr['days'] += 1.0
do_summary = 0
if c.pvr['log_summary'] and c.pvr['days'] % c.pvr['log_summary'] == 0 and _minut() == '100':
do_summary = 1              # Log summary every x days
if do_summary or date == c.pvr['date_end']:
c.pvr['cash'] = c.portfolio.cash
elif c.pvr['cash'] == c.portfolio.cash and not logging: return  # for speed

longs  = sum([p.amount * p.last_sale_price for s, p in c.portfolio.positions.items() if p.amount > 0])
shorts = sum([p.amount * p.last_sale_price for s, p in c.portfolio.positions.items() if p.amount < 0])
q_rtrn       = 100 * (c.portfolio.portfolio_value - c.pvr['start']) / c.pvr['start']
cash         = c.portfolio.cash
new_risk_hi  = 0
new_max_lv   = 0
new_max_shrt = 0
new_cash_low = 0               # To trigger logging in cash_low case
overshorts   = 0               # Shorts value beyond longs plus cash
cash_dip     = int(max(0, c.pvr['pstart'] - cash))
risk         = int(max(cash_dip, -shorts))

if record_pvrp and cash < 0:   # Let negative cash ding less when portfolio is up.
cash_dip = int(max(0, c.pvr['start'] - cash * c.pvr['start'] / c.portfolio.portfolio_value))
# Imagine: Start with 10, grows to 1000, goes negative to -10, should not be 200% risk.

if int(cash) < c.pvr['cash_low']:             # New cash low
new_cash_low = 1
c.pvr['cash_low']  = int(cash)            # Lowest cash level hit
if record_cash_low: record(CashLow = c.pvr['cash_low'])

if c.account.leverage > c.pvr['max_lvrg']:
new_max_lv = 1
c.pvr['max_lvrg'] = c.account.leverage    # Maximum intraday leverage
if record_max_lvrg: record(MaxLv   = c.pvr['max_lvrg'])

if shorts < c.pvr['max_shrt']:
new_max_shrt = 1
c.pvr['max_shrt'] = shorts                # Maximum shorts value
if record_max_shrt: record(MxShrt  = c.pvr['max_shrt'])

if risk > c.pvr['risk_hi']:
new_risk_hi = 1
c.pvr['risk_hi'] = risk                   # Highest risk overall
if record_risk_hi:  record(RiskHi  = c.pvr['risk_hi'])

# Profit_vs_Risk returns based on max amount actually spent (risk high)
if c.pvr['risk_hi'] != 0: # Avoid zero-divide
c.pvr['pvr'] = 100 * (c.portfolio.portfolio_value - c.pvr['start']) / c.pvr['risk_hi']
ptype = 'PvRp' if record_pvrp else 'PvR'
if record_pvr or record_pvrp: record(**{ptype: c.pvr['pvr']})

if shorts > longs + cash: overshorts = shorts             # Shorts when too high
if record_shorting: record(Shorts    = shorts)            # Shorts value as a positve
if record_overshrt: record(OvrShrt   = overshorts)        # Shorts beyond payable
if record_leverage: record(Lvrg = c.account.leverage)     # Leverage
if record_cash:     record(Cash = cash)                   # Cash
if record_risk:     record(Risk = risk)   # Amount in play, maximum of shorts or cash used
if record_q_return: record(QRet = q_rtrn) # Quantopian returns to compare to pvr returns curve
if record_pnl:      record(PnL  = c.portfolio.portfolio_value - c.pvr['start']) # Profit|Loss

if logging and (new_risk_hi or new_cash_low or new_max_lv or new_max_shrt):
csh     = ' Cash '   + '%.0f' % cash
risk    = ' Risk '   + '%.0f' % risk
qret    = ' QRet '   + '%.1f' % q_rtrn
shrt    = ' Shrt '   + '%.0f' % shorts
ovrshrt = ' oShrt '  + '%.0f' % overshorts
lv      = ' Lv '     + '%.1f' % c.account.leverage
pvr     = ' PvR '    + '%.1f' % c.pvr['pvr']
rsk_hi  = ' RskHi '  + '%.0f' % c.pvr['risk_hi']
csh_lw  = ' CshLw '  + '%.0f' % c.pvr['cash_low']
mxlv    = ' MxLv '   + '%.2f' % c.pvr['max_lvrg']
mxshrt  = ' MxShrt ' + '%.0f' % c.pvr['max_shrt']
pnl     = ' PnL '    + '%.0f' % (c.portfolio.portfolio_value - c.pvr['start'])
log.info('{}{}{}{}{}{}{}{}{}{}{}{}{}'.format(_minut(), lv, mxlv, qret, pvr, pnl, csh, csh_lw, shrt, mxshrt, ovrshrt, risk, rsk_hi))
if do_summary: _pvr(c)
if get_datetime() == get_environment('end'):    # Summary at end of run
_pvr(c)
elapsed = (time.time() - c.pvr['begin']) / 60  # minutes
log.info( '{}\nRuntime {} hr {} min'.format(c.pvr['run_str'], int(elapsed / 60), '%.1f' % (elapsed % 60)))

#def handle_data(context, data):
#    pvr(context, data)

There was a runtime error.

LTCM is back in town.

Leverage is not the answer to our prayers I suspect. More the architect of our down fall.

LEVERAGE IS A TOOL. Nothing more. Some can design trading strategies that can handle some leverage like this one, while others can't.

Even if you design a trading strategy that can withstand some leverage, it remains for anyone a matter of choice to apply some or not.

You don't like leverage, then don't use it. It is very simple.

When you turn on commissions and trade costs the performance severely lowers. I'm testing with numbers that I could potentially actually invest, which is only around 10k. Some of the backtests with 1,000,000 in capital aren't effected by trade costs, so the performance is higher. Is there a way to tweak the code to make it useful for more small time investors?

Alexander, I don't think that that strategy is designed for a $10k account. Try it out, it should blow the account within a year or two. Trade size will gradually decrease till you only trade 1 or 2 shares at a time. Frictional costs will erode whatever is left. Might as well find another kind of strategy than try to adapt this one. MHO. However, since I have not read all the code, someone might find a way to salvage the strategy even starting with a small account. I know I won't even try. LEVERAGE IS A TOOL Yes but the point is that there are much better ways of taking risk that gearing up to the eyeballs and risking wipeout. The vast majority of traders are fooling themselves if they believe playing the markets in a probabilistic manner is going to make their fortune. The may have read Jesse Livermore but they have not learnt his lessons. Well nor did he did he? He went endlessly bust and committed suicide. There are ways to trade markets where the odds may be better. Market making, arbitrage and so on. These days such operations tend to be rather costly to set up and are by no means foolproof. Nonetheless they have a better chance of long term success than pretending you can apply probability in the same way you can to poker. Hey guys, can you take religious debates to another thread? Amazing! Although not profitable below 500.000$ with slippage and commission turned on.

Hi Rob, Nice work! These type of strategies have been proven working very well in the past. And hence become very crowed in the market. Event/News filter definitely helps, since those information can not be automatically excluded by the mean reversion principal. However, the herding effect is significant in recent years. You may make more money by "timing" these type of strategies if you can figure out the "liquidation factor".

I just wanted to say this is algo is very clean. The only issue I have with it is the >1 leverage as in most accounts only let you borrow 100% of your portfolio value. This can easily be worked around though. Great work.

Same Algorithm: But With: Short= -.3, Long = .7
The Market, Over Time, Tends LONG

180
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""

This algorithm enhances a simple five-day mean reversion strategy by:
1. Skipping the last day's return
2. Sorting stocks based on the volatility of the five day return, to get steady moves vs jumpy ones
I also commented out two other filters that I looked at:
1. Six month volatility
2. Liquidity (volume/(shares outstanding))

"""

import numpy as np
import pandas as pd
from quantopian.pipeline import Pipeline
from quantopian.pipeline import CustomFactor
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume
from quantopian.pipeline.data import morningstar
from quantopian.pipeline.filters import Q500US

def initialize(context):

# Set benchmark to short-term Treasury note ETF (SHY) since strategy is dollar neutral
set_benchmark(sid(8554))

# Schedule our rebalance function to run at the end of each day.
schedule_function(my_rebalance, date_rules.every_day(), time_rules.market_close())

# Record variables at the end of each day.
schedule_function(my_record_vars, date_rules.every_day(), time_rules.market_close())

# Get intraday prices today before the close if you are not skipping the most recent data
schedule_function(get_prices,date_rules.every_day(), time_rules.market_close(minutes=5))

# Set commissions and slippage to 0 to determine pure alpha

# Number of quantiles for sorting returns for mean reversion
context.nq=5

# Number of quantiles for sorting volatility over five-day mean reversion period
context.nq_vol=3

# Create our pipeline and attach it to our algorithm.
my_pipe = make_pipeline()
attach_pipeline(my_pipe, 'my_pipeline')

class Volatility(CustomFactor):
inputs = [USEquityPricing.close]
window_length=132

def compute(self, today, assets, out, close):
# I compute 6-month volatility, starting before the five-day mean reversion period
daily_returns = np.log(close[1:-6]) - np.log(close[0:-7])
out[:] = daily_returns.std(axis = 0)

class Liquidity(CustomFactor):
inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding]
window_length = 1

def compute(self, today, assets, out, volume, shares):
out[:] = volume[-1]/shares[-1]

class Sector(CustomFactor):
inputs=[morningstar.asset_classification.morningstar_sector_code]
window_length=1

def compute(self, today, assets, out, sector):
out[:] = sector[-1]

def make_pipeline():
"""
Create our pipeline.
"""

pricing=USEquityPricing.close.latest

# Volatility filter (I made it sector neutral to replicate what UBS did).  Uncomment and
# change the percentile bounds as you would like before adding to 'universe'
# sector=morningstar.asset_classification.morningstar_sector_code.latest
# vol=vol.zscore(groupby=sector)
# vol_filter=vol.percentile_between(0,100)

# Liquidity filter (Uncomment and change the percentile bounds as you would like before
# adding to 'universe'
# I included NaN in liquidity filter because of the large amount of missing data for shares out
# liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan()

universe = (
Q500US()
& (pricing > 5)
# & liquidity_filter
# & volatility_filter
)

return Pipeline(
screen=universe
)

# Gets our pipeline output every day.
context.output = pipeline_output('my_pipeline')

def get_prices(context, data):
# Get the last 6 days of prices for every stock in our universe
Universe500=context.output.index.tolist()
prices = data.history(Universe500,'price',6,'1d')
daily_rets=np.log(prices/prices.shift(1))

rets=(prices.iloc[-2] - prices.iloc[0]) / prices.iloc[0]
# I used data.history instead of Pipeline to get historical prices so you can have the
# option of using the intraday price just before the close to get the most recent return.
# In my post, I argue that you generally get better results when you skip that return.
# If you don't want to skip the most recent return, however, use .iloc[-1] instead of .iloc[-2]:
# rets=(prices.iloc[-1] - prices.iloc[0]) / prices.iloc[0]

stdevs=daily_rets.std(axis=0)

rets_df=pd.DataFrame(rets,columns=['five_day_ret'])
stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret'])

context.output=context.output.join(rets_df,how='outer')
context.output=context.output.join(stdevs_df,how='outer')

context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1
context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1

context.longs=context.output[(context.output['ret_quantile']==1) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()
context.shorts=context.output[(context.output['ret_quantile']==context.nq) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()

def my_rebalance(context, data):
"""
Rebalance daily.
"""
Universe500=context.output.index.tolist()

existing_longs=0
existing_shorts=0
for security in context.portfolio.positions:
# Unwind stocks that have moved out of Q500US
if security not in Universe500 and data.can_trade(security):
order_target_percent(security, 0)
else:
current_quantile=context.output['ret_quantile'].loc[security]
if context.portfolio.positions[security].amount>0:
if (current_quantile==1) and (security not in context.longs):
existing_longs += 1
elif (current_quantile>1) and (security not in context.shorts):
order_target_percent(security, 0)
elif context.portfolio.positions[security].amount<0:
if (current_quantile==context.nq) and (security not in context.shorts):
existing_shorts += 1
elif (current_quantile<context.nq) and (security not in context.longs):
order_target_percent(security, 0)

for security in context.longs:
order_target_percent(security, .7/(len(context.longs)+existing_longs))

for security in context.shorts:
order_target_percent(security, -.3/(len(context.shorts)+existing_shorts))

def my_record_vars(context, data):
"""
Record variables at the end of each day.
"""
longs = shorts = 0
for position in context.portfolio.positions.itervalues():
if position.amount > 0:
longs += 1
elif position.amount < 0:
shorts += 1
# Record our variables.
record(leverage=context.account.leverage, long_count=longs, short_count=shorts)

log.info("Today's shorts: "  +", ".join([short_.symbol for short_ in context.shorts]))
log.info("Today's longs: "  +", ".join([long_.symbol for long_ in context.longs]))
There was a runtime error.

@Jacob, you might not have noticed but Blue's program version as well as the one I presented used 100/50 leverage settings which also favored the upside.

This is reminiscent of 150/50 or 130/30 hedge funds which also aimed for a market exposure (beta) of 1.00, just as your program version using 70/30 results in a reduced exposure with a 1.00 leverage objective.

One can use leverage when a trading strategy can support it. I presented this formula before, it helps make an estimate of leveraging costs: A(t) = (1 + L)∙k∙A(0)∙(1 + r + α - 0.03)^t.

I wanted to increase k as well as see if a 130/30 scenario as used by some hedge funds would generate more profits. So, I used Blue's version and just change the settings to 130/30 and raised initial cash to $10M. The strategy required too much from Quantopian resources and never completed. I had to compromise, cut the trading interval into 4 sections and reduced initial capital to$5M. I would lose some on the interval crossing but that was not my query. I wanted to know if it was upward scalable and to what extent.

Starting with $5M, the 4-part execution ended with$85,530,823. A 21.87% CAGR over the 14.36 years. Not bad, a 1,710% total return!

The strategy might not be good if scaled down, but it could handle scaling up. However, there would be a cost for the leveraging. One would be left with $5M∙(1+0.2187 – 0.03)^14.36 =$59,803,108. The total leveraging charges over the period: about $25M. Nonetheless, it would still be an 18.87% CAGR which is still above market averages. That turns out to a total return of 1,196% for the period, all expenses paid. @Jacob Equal weighting longs and shorts makes sense to me in the following context. You are a University endowment fund with a giant portfolio of stocks and bonds. You have a lot of market exposure already, and your returns are negatively skewed (generally positive, but occasional crashes). You want to add market neutral "hedge fund" strategies, which are often positively skewed (zero or negative day-to-day, but occasional explosive growth) to make your return distribution more symmetric with a low kurtosis (consistently positive, without crashes or explosive growth). Another strategy which is positively skewed is Private Equity (most investments in small companies don't go anywhere, but occasionally you invest in Google). You allocate say 25% of your capital to hedge funds and private equity. David Swensen's book "Unconventional Success" is a good backgrounder to this investment thesis. @Trader Cat, commissions and slippage WERE enabled. Check Blue's version of the program, I only changed two numbers (130/30). Leveraging fees were also considered. They were over-estimated by about$10M. I like to see what a program will do under adverse conditions.

Blue's program rebalances 10 minutes after the open, and not near the close. I do think it gives ample time for trades to be executed.

For the leveraging part, since the program does not calculate the fees, I used the formula: A(t) = (1 + L)∙k∙A(0)∙(1 + r + α - 0.03)^t. But, it should have been: A(t) = (1 + L)∙k∙A(0)∙(1 + r + α – 0.03*0.60)^t since the leverage would have registered at 1.60 and not 2.00. It would explain the over-charge, and paint an even better picture.

It would raise the total return to 1,381% all expenses paid, commissions, slippage and leveraging fees.

Guy
With your formulae are rather difficult to read. Would you consider defining the variables on each occasion? For me at least that would make them comprehensible.

@Anthony, you start with the usual: A(t) = A(0)∙(1 + r)^t where r is viewed as the average market return which you can have just by buying SPY, DIA, or QQQ, and holding on for the duration.

You want to design a trading strategy that is scalable: A(t) = k∙A(0)∙(1 + r)^t. Note that buying k SPY and holding is scalable 100%. It is also why I design my trading strategies to be scalable. Otherwise, how could they ever reach the level of managing a large portfolio if they were not scalable.

You want to trade, then you have to outperform the average. Bring some alpha to the mix. This will result in: A(t) = k∙A(0)∙(1 + r + α)^t where α represent the premium or contribution your trading skills bring to the problem.

Evidently, your skills needs to have a positive impact (α > 0). Nonetheless, you still want the strategy to be scalable. As previously observed, this particular trading strategy is not that scalable down. If you go small, it will simply destroy the account. It is only partially scalable up. But, still, it can generate some alpha. And, positive alpha translate to outperforming averages, generating more money which was the whole purpose of the game.

IB charges less than 3% for leverage, but leverage is only for the sum exceeding your equity. It is only when leverage exceeds one; L > 1.00 that you will be charged interests on the excess. So you can get an approximation using: A(t) = k∙A(0)∙(1 + r + α – 0.03)^t. Using leverage is the same as increasing the bet size and therefore it will have an impact on the overall score, giving: A(t) = (1 + L)∙k∙A(0)∙(1 + r + α - 0.03)^t.

The strategy was not using 100% leverage, but 60%. It was designed as a 130/30 hedge fund after all. This would have for impact to reduce the leveraging estimate to:
A(t) = (1 + L)∙k∙A(0)∙(1 + r + α – 0.03*0.60)^t.

With the modifications to Blue's program (two numbers) I got: r + α = 21.87%, L = 0.60, A(0) = $1M, and t = 14.36 years. My simulation did put k = 5 since I wanted to see the extent of the strategy's scalability. On that front, the trading strategy did not respond that well. It is understandable, it does not make any compensation for return degradation over time. But, that is a flaw that can be corrected, and doing so will improve performance even more. Thanks I'll take a look when I have a quiet moment. In general it would be very helpful for the non mathematician such as myself to have a formal definition of all variables of a given formula. Let a represent. Etc @Anthony, the formula used is very basic. It considers the end points, not the path to get there. Nonetheless, the end points give the same numbers, an as if. A(t) represents the ongoing net liquidating value of a portfolio (sum of its assets). You could also express the same thing using: A(t) = A(0) + Ʃ ƒ H(t)∙dP which reads the same. A(t) is the sum of the initial capital to which is added the end results of all trades. A(0) is the initial portfolio value (usually the available initial trading capital). r is the compounded annual growth rate (CAGR). Here, r is expressed as the average market return. It can also serve as the most expected portfolio outcome for someone playing for the long term, say more than 10 years. t is for the time in years. In this case t was for 14.36 years. k is just a scaling factor. You put 2 times more in initial capital (2∙A(0)), you should expect 2 times more in total profits. This, if the strategy is scalable. The alpha α is used in the same way as Jensen presented it in the late 60's. It is for the skills brought to the game. A positive alpha meant that what you did in managing a portfolio was beneficial since it was adding to the CAGR. What Jensen observed in his seminal paper was that the average portfolio manager had a negative alpha. Meaning that what they did, on average, was detrimental to their overall long term performance since the majority of them ended up not beating the averages. Your trading script needs to generate a positive alpha (α > 0), otherwise your portfolio might fall in the above cited category. L is for the leverage. There are no leveraging fees for L < 1.00 since it is equivalent to be not fully invested. In Blue's program, if you wanted a leverage of 2.00, you could change one, or two numbers to do the job. As was said, leveraging has the same impact as increasing the bet size. You pay for the ability of doing so. What you want your program to do is have your α > lc, its leveraging charge (lc). I don't find leveraging a trading strategy interesting unless it has an α > 2∙lc, meaning the added alpha is at least twice the leveraging charges. Otherwise you might be trading for the benefit of the broker, not your own. And as you know, the result could be even worse if the trading strategy is not well behaved or well controlled. So, with the presented equation, one can estimate the outcome of different scenarios even before performing them knowing what will be the impact on total return. If you set k = 5, meaning 5 times more initial capital, you should expect 5 times more profits. If you don't get it, then your program is showing weaknesses in the scalability department. This should lead to finding ways to compensate for the performance degradation. Hello Rob & all - Rob's first paragraph above, introducing the topic of short-term mean reversion, states that it has been a popular strategy among hedge funds. I've also heard about the summer of 2007 "quant crisis" which presumably affected hedge funds doing short-term mean reversion of equities. If so, I would think that the crisis would be reflected in backtests, but there don't appear to be any crashes in returns over that time period in the various backtests above. And conversely, nothing dramatic happens on the upside, either. It just appears to be smooth sailing through what reportedly was a rough patch for hedge funds, many of which presumably were applying the kind of short-term mean reversion strategy under discussion here. Perhaps this is just a tangential curiosity, but I'm trying to reconcile the fact of an industry crisis not being reflected in the backtests. Any insights? Thank you Guy that is very helpful. I apologise if my demands sound autistic but I have a genuine desire to understand what you are getting at. And your definition of the variables should enable me to do that. Guy Yes, having read your definitions I am now clear what you are saying. Definitions are a wonderful way to ensure clarity. I am very, very torn indeed on the question of "alpha" and indeed have severe doubts as to its existence. And indeed as to its very definition. "Beating the averages" is of course what is generally meant by alpha. But I am not at all sure that the "averages" is necessarily a helpful definition. The averages is taken to mean the manager benchmark but there are are so many "averages" that the expression has become almost meaningless especially since the modern tendency towards so called "smart beta". Take the S&P 500. If a manager consistently beats the return of the market cap weighted S&P 500 which he quotes as his benchmark, has he created "alpha"? Well, if he had chosen a different benchmark such as the equal weighted S&P 500 over many periods he may find he had NOT beaten the "average" and so his alpha disappears. It's all relative. The market cap weighted S&P 500 is merely one of a number of ways to define the "market". It is certainly not the only way as we have seen with the simple example above. And it is of course myopic to consider the US alone as a definition of the "market" when arguably in the global age the global market would be a better yardstick. With increasing certainty the longer the timescale. I understand very clearly your very simple proposal that leverage increases returns under some circumstances and the arguments as to why this will not be the case in all circumstances are too well rehearsed to repeat. On perhaps a rather different tangent I have come to seriously doubt the wisdom of "trading" as such. A systematic approach to investment (such as an index) yes, no problem. The use of technical or even fundamental indicators to "improve" upon the simple trend following nature of an index- I have my doubts. I am absorbing all I can about machine learning at present and the journey is fascinating. I believe it may help guide asset allocation. I believe it may help to point out short term market anomalies such as arbitrage opportunities. But I am under no illusion that ML can predict the future or make a Midas out of you. What I am saying is that I believe much of what most traders attempt is doomed to failure over the long term unless you engage in front running or market making or arbitrage or inside information (all of which are practiced on a daily basis by the hedge fund community). As to leverage I am not at all sure that in the long term it will help much when history tells us unleveraged investments all too often are subjected to horrendous drawdown. But of course that is merely my opinion and in the short term (however you want to define that) it can work wonders for your bank balance if you have the luck or skill to avoid a Black Swan. Another observation is that Rob's algo above tend to sail past the Great Recession, even showing a jump in returns. Was this born out in practice by hedge funds using such short-term mean reversion strategies? Did they actually make out like bandits, relative to the market (at least the ones who "stuck by their guns")? I'm just wondering to what extent this is grounded in historical reality. Or if some unaccounted for forces set in, just when the strategy, per a backtest, would suggest good profitability? @Grant, Blue changed Rob's program behavior in many ways. Instead of using the market close, he went for the open, allowing more trades to be executed. A simple change and he increased trading activity. He wanted a more realistic view of what the strategy would do, therefore, he put commissions and slippage back in. The point is: Blue made sufficient modifications to make it a different animal. Blue would do a better job at explaining all his modifications. The most important part for his added performance came from using all available cash equity at all times and leveraging to 1.5 making the strategy a 100/50 hedge fund. What I did was change this ratio to 130/30 to have a beta of 1.00. If someone wanted to generate more they could simply go for a 150/50 fund. But, not for small accounts which would blow up more rapidly. This strategy is definitely not designed for small accounts. The program goes through the financial crisis because it only projects 5 days ahead, it does not see a future beyond that. And during the downturn, there were fewer long candidates to trade with which reduced the downside. So, Blue's program is ok. The stock selection is doable in the future, the projections are doable in the future, and the trading logic is also doable. However, just as Rob's program, there is no compensation for performance degradation. A problem that should be addressed which would improve performance even further. Well, Rob says of hedge funds he's worked alongside: by far the most common strategy is some variation of short-term mean reversion So, what are they doing that is not being done here? Or have they all gone out of business? What am I missing? In practice, I think he's saying it may be a promising approach, but then one should be able to cook up a half-decent algo on Quantopian, right? I'm confused. @Anthony, sure everyone can have a definition for alpha. As mentioned, I am still on the Jensen definition of the late 60's. But, nonetheless, here is a perfect example. You buy 1,000 shares of SPY. Your leverage is 1.00. Your beta is 1.00. Your volatility, drawdown, Sharpe or whatever metrics are all 1:1 while you are holding SPY. Your return compared to SPY is 1:1. All you get is SPY's CAGR. You get exactly the same return. Therefore, the r in A(0)∙(1 + r)^t is the same as SPY's CAGR. It is also all you can get, it is all SPY can offer. And you can get it just by sitting it out. No work, but also no alpha, none was available. None to be had since r + α = r has only one solution. A side note, I only trade US stocks, so for me, the rest of the world is irrelevant. I am not designing universal tools but ones adapted to the market I want to trade and that I know best. In other markets I would design different and adapted tools. If you opt to trade SPY over the long term, you will have periods where you are in, and others where you are out of the SPY market. Automatically, you will be affecting r by some factor. Jensen used alpha as a representation of this factor. You could do better or worse than the average depending on this factor: A(0)∙(1 + r ± α)^t. But all it did translate was the impact of all the time slicing and trade dicing over the trading interval t. If alpha was greater than zero, what you did improved the outcome at the CAGR level. It had an impact on the end results. Blue's program is generating some positive alpha. Mr. Buffett has been in the game for over 50 years. The sum of all his holdings have become the equivalent of an index, they have been for some time. You could use BRK/A the same way as an index, say the S&P500. For all the stocks that went in Mr. Buffett's portfolio, they all had this 1:1 relationship with the corresponding listed stock on some exchange. Therefore, the sum of his holdings should have close to this 1:1 relationship with the S&P500 index. His return should tend to be: A(0)∙(1 + r)^t. But, it is not the case. His excess return can be expressed as: A(0)∙(1 + r + α)^t. His skills generated the alpha. And he was subject to the same constraints we have, that is in every stock he got into, there was this 1:1 relationship. So the question becomes, how did he generate his alpha? And, be assured it is not by luck. I am of the opinion that everybody can win at this game. That anyone can design trading strategies that will stand the test of time. But first, they have to program for the long term. Have a vision of where they are going and how they will get there. Along the way, they will see that math is certainly part of the game. It is like asking the question: will Blue's strategy come to a point where it will underperform the averages? The answer is yes. The reason is simple, it is not compensating for performance degradation. However, you have years to find a solution, since it did outperformed for the first 14.36 years. Can you compensate for this degradation and make the strategy outperform for a much longer trading interval? The answer here too is yes. But that too, you have to program it in. A trading strategy, a program, will not develop these skills on its own. @Grant, sorry if it took some reflection time to come up with a partial answer. If you add frictional costs to Rob's strategy, you will see performance degrade to about a 3.88% CAGR for the 14.36 years. The strategy makes about 140 trades a day. This alone translate to over 500,000 trades over the trading interval. It puts the average net profit at about$1.40 per trade. Even if you added more trades in the future, the average net profit per trade would still tend to this limiting value due to the large number of trades. One could say: this trading strategy has, on average, a $1.40 net profit per executed trade. And we would be right with a very high confidence level. Such a statement also makes the strategy a linear proposition which will have for impact that it degrades, performance wise, over time. After the initial boom, its CAGR is going down with time! This, even if the market is going up! Using an equation to express this gives: A(t) = A(0)∙(1 + r + α – fc)^t = A(0)∙(1 + 0.0388)^t, where fc stands for frictional costs as a percent, r the average market return, and α the skills contribution to the game in percent. Since r is about 10.0% (market secular trend), the alpha (α) is negative. A way of saying that the trading strategy has no economic value. A trading strategy can be viewed as a block having a predefined structure composed of its trading procedures triggering trading decisions. You can treat this structure as a block, or micro-manage the procedures. But, doing so, will introduce curve fitting. The more you micro-manage procedures, the more you are making your strategy an outsider to reality. Rob's strategy, as presented, without the frictional costs came in at a 10.26% CAGR. Giving a gross profit of$5.86 per trade. Meaning that, on average, $4.46 per trade was going to cover frictional costs. That is$2.3 million went to cover these costs over the 14.36 years. This says that the frictional cost was about 0.0638. A frictional cost having a drag on its CAGR of 6.38%! If I was a broker, I would invite anyone with such a strategy. Provide red carpet and VIP treatment. The broker here is the one making the money.

Blue's modifications change the very dynamics of the trading strategy. Enabled it to go for a positive alpha in spite of the frictional costs since there were included in his version of the program. The alpha was sufficiently high that he could introduce leverage to the mix and further increase performance. All I did was change two numbers to go for full exposure, for a beta of 1.00 by making it a 130/30 hedge fund. It was a little bit higher, leverage wise, than Blue's 100/50 design, and evidently produce a little bit more. It is a way of saying I had no merit in this.

Also note that Rob's trading strategy is not scalable. It appears as if designed for A(0) = $1M. If you start with$100k, you will be wiped out if you include frictional costs. And at $500k, your net CAGR would be about 1.9%. Whereas at$2M, it will bearly add 0.34%; when it should have doubled the generated profits. Pushing for A(0) = $10M, the CAGR would do about 2.87%. Still lower than the$1M design. However, it does fill the requirements for the contest, but I would not expect to win or even come close.

Thanks Guy -

My interpretation of Rob's post was to encapsulate the main result of a study he did in the form of an illustrative backtest, ignoring the effects of slippage and trading costs. The implication, I think, is that his conclusions about potential ways of enhancing mean reversion would be valid for practical strategies, as well. My question is, in the context of the Quantopian long-short program (see https://www.quantopian.com/allocation and https://blog.quantopian.com/a-professional-quant-equity-workflow/), and the industry, in general, where does this approach fit in, and what are the ways to make it work as an alpha factor (in a long-short, market neutral fashion)? I'd note that Quantopian recently published another study on mean reversion, https://www.quantopian.com/posts/quantpedia-trading-strategy-series-an-analysis-on-cross-sectional-mean-reversion-strategies , which may be of interest.

If it is a very common strategy, then how is it done in practice?

Grant
You need to look through the various web sites which track hedge funds and CTA returns. Most of them are free. Most of them give at least some detail of the strategy a fund follows.

I agree with your train of thought.

In the past I have spoken with or met various managers wearing various of my hats and it is always illuminating. I also attended many manager presentations with Q&A sessions.

As you will by now be fully aware much of the chat on web trading forums tends to be of the "blind leading the blind" variety. The answers to your questions will be found by talking to people who actually trade the strategies which interest you. And in trading those strategies for yourself.

Excellent!

@Anthony, even if you look at hedge fund performance statistics or a general description of what they do, it is always a fuzzy ensemble. They are trying to protect their intellectual properties.

Nothing like a program you can study and dissect will reveal the feasibility of a particular trading strategy. For instance, I studied this one for the simple reason that it was within the Quantopian contest requirements.

I appreciate that Rob did put it in public view. But note that he had tried a lot of things to improve on his design before making it public. I quote: “...while no hard data exists to support this claim...”, “...I backtested numerous ideas on Quantopian...”.

I considered the strategy just as a starting point. No need to program. I could change some of the parameters to modify its trading behavior.

Say you took the original design and made the following modifications.

Put the frictional costs back in, and make it a 130/30 hedge fund. You comment out 2 lines of code, and change the 0.50 to 1.30 and the -0.50 to -0.30. Making it a strategy that favors the upside.

The strategy would see an increase in performance, and raise its CAGR to 15.76% before leveraging costs. Leveraging costs would total about $1.6M. It is not negligible, but the fund would still maintain a 14% CAGR which is a major improvement compared to the 3.88% generated by the original program. And just doing this, meaning slightly leveraging, would have generated some positive alpha. This move did not change the strategy or its trading logic. It only changed how you want to treat it as if using some outside parameter. It is not even an optimization factor, but just a change in how you viewed the strategy. Agreed re fuzzy on multi strategy but many are single: distressed debt, TF, stat arb, m and a arb. And even if multi strategy actual manager meetings and Q and A can be very informative. @Anthony, agree. But, informative is not the same as having access to an actual coded trading strategy based on some of the principles used by some of these hedge funds. We both would say that Rob's original trading strategy, as presented, has no economic value since you could buy SPY and have done better doing nothing more than just holding on. Blue, by leveraging this scenario to 1.5 showed that there can be alpha hidden underneath should one be ready to pay for the added expenses which as was shown were not negligible. Maybe, one should go back to the premises underlying the proposed methodology. Should we not question if in reality there is short term mean-reversion? But maybe most importantly, can we really benefit, meaning extract some alpha from this knowledge? by leveraging this scenario to 1.5 showed that there can be alpha hidden underneath should one be ready to pay for the added expenses which as was shown were not negligible. Yes. Margin could be a very good citizen if not for its superpower of invisibility turning it into a Gollum. Couple of things to try: 1. Instead of closing and opening both long and short all in the same minute, close any matching a condition at market open, then open shorts, say around 60 minutes, then open longs an hour or whatever later. 2. Renew get_prices() at start of each of those (will need some rewrite of some variable names). 3. Grab the wait() function in my version and add wait(context, security, 1) after each order. It was incomplete in that way. 4. As leverage goes up, reduce short allocations proportionally (and visa-versa, to a point). Imagine leverage at 1.1, val below (the value of the stock that will be ordered) is divided by 1.1, thus lowering its value. The higher the leverage the less shorting. Or similar for order_target_percent(). Similar for longs.  lev = context.account.leverage if lev > .9: val /= lev  That's a thumbnail idea for using the leverage figure to dynamically determine ordering. Or even also close some positions based on it if things really get out of hand with leverage at some point, beyond some threshold of acceptability you set. Maybe those with high or low PnL. Leverage is (abs(short_value) + long_value) / portfolio_value. Would like to see a function that says here's my target leverage, returning how much long or short ought to be adjusted. Yes. Margin could be a very good citizen if not for its superpower of invisibility turning it into a Gollum. Ha ha ha. Or even Golem. People will always be wrapped up in impossible dreams. For some things you have no choice but to borrow: infrastructure investment, property and so forth. Even there over-leverage ends in ruin. For financial markets it's inevitable over the long term. Wikipedia. The existence of a golem is sometimes a mixed blessing. Golems are not intelligent, and if commanded to perform a task, they will perform the instructions literally. In many depictions Golems are inherently perfectly obedient. In its earliest known modern form, the Golem of Chełm became enormous and uncooperative. In one version of this story, the rabbi had to resort to trickery to deactivate it, whereupon it crumbled upon its creator and crushed him.[2] There is a similar hubris theme in Frankenstein, The Sorcerer's Apprentice, and some other stories in popular culture, for example: The Terminator. The theme also manifests itself in R.U.R. (Rossum's Universal Robots), Karel Čapek's 1921 play which coined the term robot; the play was written in Prague, and while Čapek denied that he modeled the robot after the Golem, there are many similarities in the plot.[40] One of Rob's comments was "While I believe it makes sense to test alpha generating signals separately, ultimately the goal is to combine them." The context here is the overall workflow/strategy construction formula described in A Professional Quant Equity Workflow. There would be N so-called alpha factors, out of which 1 (or perhaps more) would be the mean reversion type described by Rob. The mean reversion alpha factor would be combined, perhaps in a fancy-dancy ML fashion and then pushed though an optimizer/risk management module, and finally passed off to an order management system, and finally a to prime broker. It would seem that this kind of mean reversion alpha factor should be included in the mix, but there would need to be some way to systematically weight it versus time, relative to other factors. Otherwise, one ends up with lots of trading churn over long periods of time. My thought is that there has to be enough information available in the N alpha factors for the ML combination step to be able to sort out how to weight the mean reversion factor point-in-time. Regarding the whole leverage theme here, I'm not sure it is relevant. First off, the leverage would be applied to the whole multi-dimensional long-short algo, not to an individual alpha factor. Secondly, whatever leverage is applied by Quantopian ends up risk-free to the algo author--he will be paid a percentage of the profit including leverage. Of course, the author can make it a risk by quitting his day job, for example, but the leverage can simply be treated as "gravy" if he doesn't want the personal risk. So, my suggestion is that we move the leverage discussion elsewhere. Regarding what might be done in the real world of hedge fund trading, I suspect that for such relatively short-term strategies, higher frequency data are used (e.g. minutely/tick), and perhaps pre-/post-/foreign market data, as well. The time scale here is pretty short (~5 days), and so using daily bar data would seem to be sub-optimal (low-frequency sampling looking for higher frequency effects). If so, this wouldn't necessarily solve the problem of identifying secular trends of mean reversion working/not working, but perhaps would boost the returns to make the whole job easier--another dimension of the problem to consider. Grant The big takeaway as you say is that it's risk free for the algo provider! Very good point. Same with Quantiacs, Numerai and the rest of the Gang. @Grant, you say: “Regarding the whole leverage theme here, I'm not sure it is relevant.” I differ on that one. Knowing that a Quantopian suitable trading strategy could be leveraged up to six times, would it not provide you some added knowledge, a better opportunity, if you knew that your trading strategy was in fact scalable and leverageable to that level? At least, you could know that your trading strategy did meet all the Quantopian selection criteria. On the other hand, once you will have verified that your trading strategy can in fact operate at that level, I think you might lose interest in putting it in the contest anyway. @ Guy - There are some lower limits on returns, I'd have to figure, but based on what I've seen on Quantopian (and my own experience backtesting), once SR is decent, then returns tend to be reasonable, as well. I could be off-base, but I get the sense that leverage-ability is baked into the long-short equity recipe provided by Quantopian (and reportedly is standard practice in the industry). I gotta think that Quantopian is probably not getting too many algos that meet all of the criteria, but just can't be leveraged. If one writes a nice multi-factor algo trading 100-200 stocks at with$10M using the Q500US/Q1500US, it should be leverage-able. Also, it can be glommed together with other algos, so maybe the leverage-ability of individual algos is even more irrelevant.

I just don't see the relevance to this thread, or even really to writing algos for the Q fund, but maybe I'm missing something.

I'm wondering if a hedging instrument (e.g. SPY) or basket of hedging instruments might help here, to limit trading costs? Intuitively, there are transactions associated with capturing the mean-reversion alpha, and transactions to maintain market neutrality. So, perhaps the latter set of transactions could be encapsulated in a single hedging instrument. Since the trading costs are on a per share transacted basis, maybe this would help. I don't think this approach fits so well within the Quantopian long-short equity framework, though, since the guidance seems to be not to use hedging instruments, but rather to have equal long-short positions, or some such thing.

@Grant, I think we simply have a different viewpoint.

For me, an automated trading strategy needs to be upward scalable. If not, its value is considerably reduced.

If scaling up for whatever trading methods used gets to produce over the long haul less than market average, then a strategy becomes literally worthless since the alternative of buying an index fund would have generated more for a lot less work.

No one will put 1,000 $10k strategies to work ($10M). Especially if you have 10 times or 100 times more capital to invest. You need scalable strategies that will not break down under AUM pressure. You also need machines that can handle the data and execute the trades. You need not only a doable, but also a feasible scenario that can handle A(t) for the entire trading/investing interval. This trading interval can not be just for a few years, otherwise you are just taking a timing bet where the period you activate your account becomes the most important.

Also, as you must have seen numerous times here, having a $10k strategy is by no means a guaranteed viable scenario at$100k, $1M, or$10M for that matter. The problems encountered at each level are quite different, and this just based on the initial capital put to work.

I've given this formula elsewhere: A(t) = (1+L)∙k∙A(0)∙(1 + r + α - lc% - fc%)^t, and what I see as dreamland is expecting some positive alpha for a 50/50 long short portfolio with a near zero beta. The examples that have been presented on Q failed that acid test: α > 0. And it usually gets worse when you scale them up: k∙A(0), increase the trading interval, account for frictional costs fc%, and leveraging fees lc%.

There are 3 ways to improve the Sharpe ratio. One is to really increase your portfolio average return (generate some alpha). Another is to reduce volatility. And the other is to do both at the same time. However, anytime you want to reduce volatility, it entails on average, lower risks but also lower returns.

So, for someone wishing some Q allocation or even just a viable trading strategy, they should understand the implications of the formula: A(t) = (1+v∙L)∙k∙A(0)∙(1 + r + α – v∙lc% – v∙fc%)^t where r, the market average, will be reduced by a negative alpha α. And where leveraging fees and frictional costs will be v times the leveraging boost. Evidently, if v = 1 (no boost), k = 1 (no scaling), lc% = 0 (no leverage), you are left with: A(t) = A(0)∙(1 + r + α – fc%)^t since commissions will still have to be paid. Then, you better have some trading skills to outperform the average ( α > fc%).

I don't see that as a very pretty picture. The only remedy is to have a positive alpha of sufficient size to compensate for all the drawbacks. Even the one rarely discussed here which is long term return degradation.

I am from the outside, and I would never put anybody's strategy to work without testing the above formula on a sufficiently large portfolio over an extended period of time which is what I do using Q, explore their limits, see how far they can go, where will they break down?

It is like if you can not show me that your trading strategy is scalable upwards, what is its value? If you can show that it can generate some alpha, again what could be its value going forward? And if it is not leverageable, how can we push it further to do even more?

For me, it does not matter which trading methods you use, my interest is only on A(t). And the score for A(t) is kept on the bottom line of the trading account.

However you want to look at it. If you want more, you will have to do more than the other guy.

It's like if we have the exact same car and take a long race to the finish line. Our respective driving skills will matter. But regardless, I could get a definite positive edge just by changing the tires. Then driving skills might not matter much.

just messing around with it

111
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
import numpy as np
import pandas as pd
from quantopian.pipeline import Pipeline
from quantopian.pipeline import CustomFactor
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume
from quantopian.pipeline.data import morningstar
from quantopian.pipeline.filters import Q500US
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.ensemble import ExtraTreesClassifier

class Volatility(CustomFactor):
inputs = [USEquityPricing.close]
window_length=132

def compute(self, today, assets, out, close):
daily_returns = np.log(close[1:-6]) - np.log(close[0:-7])
out[:] = daily_returns.std(axis = 0)

class Liquidity(CustomFactor):
inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding]
window_length = 1

def compute(self, today, assets, out, volume, shares):
out[:] = volume[-1]/shares[-1]

class Sector(CustomFactor):
inputs=[morningstar.asset_classification.morningstar_sector_code]
window_length=1
def compute(self, today, assets, out, sector):
out[:] = sector[-1]

#not necessary
class Price(object):
var = "VarPrice"
def _init_ (self):
pass
iPrice = Price()
iPrice.var = 20

def initialize(context):

context.Sell = []

context.Universe500 = []

set_benchmark(sid(8554))

schedule_function(get_prices,date_rules.every_day(), time_rules.market_close(minutes=5))

schedule_function(my_rebalance, date_rules.every_day(), time_rules.market_close(minutes = 1))

schedule_function(close_, date_rules.every_day(), time_rules.market_close(minutes = 1))

schedule_function(open_, date_rules.every_day(), time_rules.market_close())

context.Rmodel = RandomForestRegressor()
context.model = ExtraTreesRegressor()
#Number Of Estimators
# 100-150 Seems To Be A Good Place To Start. Anything ABOVE 250 ~= 250 (But Way More Compute Time)
#context.model.n_estimators = 25

#context.model.min_samples_leaf = 5

context.lookback = 2
context.history_range =  50

context.nq=5
context.nq_vol=3

context.existing_longs=0
context.existing_shorts=0

context.mean_predict = []

attach_pipeline( make_pipeline(context), 'my_pipeline')

def make_pipeline(context):

pricing=USEquityPricing.close.latest

profitable = morningstar.valuation_ratios.ev_to_ebitda.latest > 0

#sector = morningstar.asset_classification.morningstar_sector_code.latest

#vol=vol.zscore(groupby=sector)
#vol_filter=vol.percentile_between(0,100)

#liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan()

universe = (
Q500US()
# & volatility_filter
#& ~sector.eq(311)
#& ~sector.eq(102)
#& ~sector.eq(101)
& profitable
#& vol_filter
)
return Pipeline(
screen=universe
)

context.output = pipeline_output('my_pipeline')

def get_prices(context, data):
while True:
pricing_data = []
sorted_data = []
output = pipeline_output('my_pipeline')
context.Universe500=context.output.index.tolist()
prices = data.history(context.Universe500,'price',1,'1d')
for sec in prices:
price = prices[sec][-1]
pricing_data.append(price)
pricing_data = [int(x) for x in pricing_data]
pricing_data.sort()
sorted_data.append(pricing_data[1:100])
iPrice.var = np.mean(sorted_data)
break

# ORIGINAL MEAN REVERSION ALGORITHM AUTHOR ARGUED FOR 6 : NOTEBOOKS HAVE SHOWN THAT 9 WORKS BETTER FOR THIS SITUATION
prices = data.history(context.Universe500,'price',1,'1d')
for sec in prices:
price = prices[sec][-1]
if price > iPrice.var:
context.Universe500.remove(sec)
print(context.Universe500)

prices = data.history(context.Universe500,'price',9,'1d')
daily_rets=np.log(prices/prices.shift(1))

rets=(prices.iloc[-3] - prices.iloc[0]) / prices.iloc[0]

stdevs=daily_rets.std(axis=0)

rets_df=pd.DataFrame(rets,columns=['five_day_ret'])
stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret'])

context.output=context.output.join(rets_df,how='outer')
context.output=context.output.join(stdevs_df,how='outer')

context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1
context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1

context.longs=context.output[(context.output['ret_quantile']==1) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()
context.shorts=context.output[(context.output['ret_quantile']==context.nq) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()

def my_rebalance(context, data):
context.existing_longs=0
context.existing_shorts=0
for security in context.portfolio.positions:
if security not in context.Universe500 and data.can_trade(security):
order_target_percent(security, 0)
else:
current_quantile=context.output['ret_quantile'].loc[security]
if context.portfolio.positions[security].amount>0:
if (current_quantile==1) and (security not in context.longs):
context.existing_longs += 1
elif (current_quantile>1) and (security not in context.shorts):
order_target_percent(security, 0)
elif context.portfolio.positions[security].amount<0:
if (current_quantile==context.nq) and (security not in context.shorts):
context.existing_shorts += 1
elif (current_quantile<context.nq) and (security not in context.longs):
order_target_percent(security, 0)
def close_(context, data):
results = context.shorts
log.info(results)
securities_in_results = results.index
passed = []
context.mean_predict = []
for sec in context.shorts:
"""
#Stock, Price, lookback 5, Days
recent_prices =  data.history(sec, 'price', context.history_range, '1m').values
#Stock, Volume, Lookback 5, Days
recent_volumes = data.history(sec, 'volume', context.history_range, '1m').values
price_changes = np.diff(recent_prices).tolist()
volume_changes = np.diff(recent_volumes).tolist()

#X = Input, Y= Output
X,Y = [],[]

for i in range(context.history_range-context.lookback-1):
X.append(price_changes[i:i+context.lookback] + volume_changes[i:i+context.lookback])
Y.append(price_changes[i+context.lookback])

context.model.fit(X, Y)
context.Rmodel.fit(X, Y)

#Check Model Accuracy:
recent_prices = data.history(sec, 'price', context.lookback+1, '1d').values
recent_volumes = data.history(sec, 'volume', context.lookback+1, '1d').values
price_changes =  np.diff(recent_prices).tolist()
volume_changes = np.diff(recent_volumes).tolist()

#Make Prediction
prediction = context.model.predict(price_changes + volume_changes)
context.mean_predict.append(prediction)
if prediction < 0.01:
passed.append(sec)
"""
for sec in context.shorts:
order_target_percent(sec, -.95/(len(context.shorts)+context.existing_shorts))
"""
elif data.can_trade(sec) and sec not in passed:
order_target_percent(sec, 0.0/(len(context.shorts)+context.existing_shorts))
else:
print("WTF")
"""
def open_(context, data):
results = context.longs
log.info(results)
securities_in_results = results.index
passed = []
for sec in context.longs:
"""
#Stock, Price, lookback 5, Days
recent_prices =  data.history(sec, 'price', context.history_range, '1d').values
#Stock, Volume, Lookback 5, Days
recent_volumes = data.history(sec, 'volume', context.history_range, '1d').values
price_changes = np.diff(recent_prices).tolist()
volume_changes = np.diff(recent_volumes).tolist()

#X = Input, Y= Output
X,Y = [],[]

for i in range(context.history_range-context.lookback-1):
X.append(price_changes[i:i+context.lookback] + volume_changes[i:i+context.lookback])
Y.append(price_changes[i+context.lookback])

context.model.fit(X, Y)
context.Rmodel.fit(X, Y)

#Check Model Accuracy:
recent_prices = data.history(sec, 'price', context.lookback+1, '1d').values
recent_volumes = data.history(sec, 'volume', context.lookback+1, '1d').values
price_changes =  np.diff(recent_prices).tolist()
volume_changes = np.diff(recent_volumes).tolist()

prediction = context.model.predict(price_changes + volume_changes)
Rprediction = context.Rmodel.predict(price_changes + volume_changes)
context.mean_predict.append(prediction)
context.mean_predict.append(prediction)
if 0.03 > prediction > 0.01:
passed.append(sec)
"""
for sec in context.longs:
order_target_percent(sec, 1.0/(len(context.longs)+context.existing_longs) )
# If you dont care how high your beta is. uncomment this line and play around. you will see higher returns but with much higher drawdowns
#   order_target_percent(sec, 1.0/(len(context.longs)+context.existing_longs) )

def handle_data(context, data):
while True:
longs = shorts = 0
for position in context.portfolio.positions.itervalues():
if position.amount > 0:
longs += 1
elif position.amount < 0:
shorts += 1
# Record our variables.
try:
Mpred = np.amax(context.mean_predict)
except:
Mpred = 0.0001
pass

record(long_count=longs, short_count=shorts)
record(leverage=context.account.leverage, P_value = iPrice.var,MaxPrediction = Mpred)
break
"""
log.info("Today's shorts: "  +", ".join([short_.symbol for short_ in context.shorts]))
log.info("Today's longs: "  +", ".join([long_.symbol for long_ in context.longs]))
"""


There was a runtime error.

@Dan
Thanks for pointing out that interesting paper on the strong correlation between the VIX and mean-reversion profits. Timing a mean-reversion strategy might be an interesting enhancement.

@Grant
Regarding your question earlier in this thread about the performance during the “Quant Crash” of Aug 2007, I believe the strategies that were impacted the most were lower frequency strategies that used fundamental factors, and the medium and higher frequency strategies did not fare as badly. Also, strategies that included mid cap and small cap stocks in their universe (typically not high-turnover mean-reversion strategies) did worse. If you sorted stocks by one of the popular value factors, say cash flow-to-price, the low cash flow decile outperformed the high cash flow decile by over 7% over the three-day period of August 7-9, 2007. In contrast, if you sorted stocks based on past five-day returns as of August 6 and held over the next three days, the returns were only slightly negative. And for the funds that did not choose to (or were not forced to) cut back or completely liquidate their positions during the Quant Crash, things reversed for most part pretty quickly, which masked some of the losses earlier in the week. I’ve attached a short notebook demonstrating what happened during the Quant Crash to fundamental factor strategies and mean-reversion strategies. Also, the paper by Khandani and Lo (see here) has a nice discussion and analysis about what happened to the quants during this period. They do indeed find that mean-reversion strategies had significant losses on Aug 7-9 before rebounding on Aug 10, but keep in mind that they looked at a one-day mean-reversion strategy and they also find the losses were more muted for the highest market cap decile.

@Alexander Souvall, @Jens Kristian Skovgaard, @Trader Cat
There were also a few comments earlier in the thread about transaction costs with this algorithm, which is certainly a big issue for any strategy like this that has a higher turnover (from the Pyfolio tearsheet, the holding period (1/turnover) is about 2.5 days). Let me make few remarks about transaction costs. First, for large institutions, I think the default transaction costs are probably a little too high, particularly the commissions. It’s instructive to look at each component of transaction costs incrementally.

                                                              Cumulative Returns        Sharpe Ratio
No slippage, No commission                                          306%                   1.34
Default slippage, No commission                                     250%                   1.29
Default slippage, min_trade_cost=1, No cost(per share)              154%                   0.95
Default slippage, No min_trade_cost, cost(per share)=0.0075          94%                   0.69
Default slippage, Default commission                                 73%                   0.57


When trading a universe of Q500 stocks and not trading huge size, my experience is that the default slippage (capped at 0.625 b.p) is reasonable for institutional investors (maybe even a little low). But most of the degradation in performance comes from the default commissions, not the slippage. Even the $1/trade minimum cost has a huge effect on performance, partly because the algorithm was not designed to consider this cost. For simplicity, I rebalanced every day to maintain equal weights, so if I had positions in 50 stocks and added one stock, I would sell tiny amounts of the other 50 stocks to maintain equal weights, which is certainly not optimal with fixed transaction costs. And institutional investors don't typically pay a minimum transaction cost per trade. Also, the default cost (per share) of$0.0075 is much higher than large, quant funds typically pay. The actual commissions are usually a small fraction of those costs. And, there are probably other ways to reduce the turnover and consequently lower the transaction costs. One could look at using other rules for unwinding, rather than immediately unwinding a position as soon as the stock moves out of an extreme quantile.

25
Notebook previews are currently unavailable.

I'm guessing this isn't a long enough time slice to be representative of anything, but it is at least concerning. Any thoughts?

edit: Apologies, finally got caught up with this thread. My understanding is that the volume of trades and trading fees makes this cost prohibitive with a smaller initial capital under the default slippage/commission.

8
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
This algorithm enhances a simple five-day mean reversion strategy by:
1. Skipping the last day's return
2. Sorting stocks based on the volatility of the five day return, to get steady moves vs jumpy ones
I also commented out two other filters that I looked at:
1. Six month volatility
2. Liquidity (volume/(shares outstanding))

"""
import numpy as np
import pandas as pd
from quantopian.pipeline import Pipeline
from quantopian.pipeline import CustomFactor
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume
from quantopian.pipeline.data import morningstar
from quantopian.pipeline.filters import Q500US

def initialize(context):

# Set benchmark to short-term Treasury note ETF (SHY) since strategy is dollar neutral
set_benchmark(sid(23911))

# Get intraday prices today before the close if you are not skipping the most recent data
schedule_function(get_prices,date_rules.every_day(), time_rules.market_open())

# Schedule rebalance function to run at the end of each day.
schedule_function(rebalance, date_rules.every_day(), time_rules.market_open())

# Record variables at the end of each day.
schedule_function(record_vars, date_rules.every_day(), time_rules.market_close())

# Set commissions and slippage to 0 to determine pure alpha

# Number of quantiles for sorting returns for mean reversion
context.nq=5

# Number of quantiles for sorting volatility over five-day mean reversion period
context.nq_vol=3

# Create pipeline and attach it to algorithm.
pipe = make_pipeline()
attach_pipeline(pipe, 'pipeline')

context.waits = {}
context.waits_max = 3    # trading days

return

context.pnl_sids_exclude  = [ sid(2561) ]
context.pnl_sids  = [  ]
context.day_count = 0
#schedule_function(record_pnl, date_rules.every_day(), time_rules.market_close())

# Take profit several times a day
context.profit_threshold = .11
context.profit_logging   = 1
for i in range(20, 390, 60):    # (low, high, every i minutes)
#continue    # uncomment to deactivate
schedule_function(take_profit, date_rules.every_day(), time_rules.market_open(minutes=i))

def wait(c, sec=None, action=None):
if sec and action:
if action == 1:
c.waits[sec] = 1    # start wait
elif action == 0:
del c.waits[sec]    # end wait
else:
for sec in c.waits.copy():
if c.waits[sec] > c.waits_max:
del c.waits[sec]
else:
c.waits[sec] += 1   # increment

def take_profit(context, data):    # Close some positions to take profit
pos     = context.portfolio.positions
history = data.history(pos.keys(), 'close', 10, '1m').bfill().ffill()
for s in pos:
if not data.can_trade(s):      continue
if slope(history[s])      > 0: continue
if slope(history[s][-5:]) > 0: continue
if history[s][-1] > history[s][-2]: continue
prc = data.current(s, 'price')
amt = pos[s].amount
if (amt / abs(amt)) * ((prc / pos[s].cost_basis) - 1) > context.profit_threshold:
order_target(s, 0)
wait(context, s, 1)    # start wait
if not context.profit_logging: continue
pnl = (amt * (prc - pos[s].cost_basis))
if pnl < 3000: continue
log.info('close {} {}  cb {}  now {}  pnl {}'.format(
amt, s.symbol, '%.2f' % pos[s].cost_basis, prc, '%.0f' % pnl))

import statsmodels.api as sm
def slope(in_list):     # Return slope of regression line. [Make sure this list contains no nans]
return sm.OLS(in_list, sm.add_constant(range(-len(in_list) + 1, 1))).fit().params[-1]  # slope

def record_pnl(context, data):
def _pnl_value(sec, context, data):
pos = context.portfolio.positions[sec]
return pos.amount * (data.current(sec, 'price') - pos.cost_basis)

context.day_count += 1

for s in context.portfolio.positions:
if not data.can_trade(s): continue
if s in context.pnl_sids_exclude: continue

# periodically log all
if context.day_count % 126 == 0:
log.info('{} {}'.format(s.symbol, int(_pnl_value(s, context, data))))

# add up to 5 securities for record
if len(context.pnl_sids) < 5 and s not in context.pnl_sids:
context.pnl_sids.append(s)
if s not in context.pnl_sids: continue     # limit to only them

# record their profit and loss
who  = s.symbol
what = _pnl_value(s, context, data)
record( **{ who: what } )

class Volatility(CustomFactor):
inputs = [USEquityPricing.close]
window_length=132

def compute(self, today, assets, out, close):
# I compute 6-month volatility, starting before the five-day mean reversion period
daily_returns = np.log(close[1:-6]) - np.log(close[0:-7])
out[:] = daily_returns.std(axis = 0)

class Liquidity(CustomFactor):
inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding]
window_length = 1

def compute(self, today, assets, out, volume, shares):
out[:] = volume[-1]/shares[-1]

class Sector(CustomFactor):
inputs=[morningstar.asset_classification.morningstar_sector_code]
window_length=1

def compute(self, today, assets, out, sector):
out[:] = sector[-1]

def make_pipeline():
"""
Create pipeline.
"""

pricing=USEquityPricing.close.latest

# Volatility filter (I made it sector neutral to replicate what UBS did).  Uncomment and
# change the percentile bounds as you would like before adding to 'universe'
# sector=morningstar.asset_classification.morningstar_sector_code.latest
# vol=vol.zscore(groupby=sector)
# vol_filter=vol.percentile_between(0,100)

# Liquidity filter (Uncomment and change the percentile bounds as you would like before
# adding to 'universe'
# I included NaN in liquidity filter because of the large amount of missing data for shares out
# liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan()

universe = (
Q500US()
& (pricing > 5)
# & liquidity_filter
# & volatility_filter
)
return Pipeline(
screen  = universe
)

# Gets pipeline output every day.
context.output = pipeline_output('pipeline')

wait(context)    # Increment any that are present

def get_prices(context, data):
# Get the last 6 days of prices for every stock in universe
Universe500=context.output.index.tolist()
prices = data.history(Universe500,'price',6,'1d')
daily_rets = np.log(prices/prices.shift(1))

rets=(prices.iloc[-2] - prices.iloc[0]) / prices.iloc[0]
# I used data.history instead of Pipeline to get historical prices so you can have the
# option of using the intraday price just before the close to get the most recent return.
# In my post, I argue that you generally get better results when you skip that return.
# If you don't want to skip the most recent return, however, use .iloc[-1] instead of .iloc[-2]:
# rets=(prices.iloc[-1] - prices.iloc[0]) / prices.iloc[0]

stdevs=daily_rets.std(axis=0)

rets_df=pd.DataFrame(rets,columns=['five_day_ret'])
stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret'])

context.output=context.output.join(rets_df,how='outer')
context.output=context.output.join(stdevs_df,how='outer')

context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1
context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1

context.longs=context.output[(context.output['ret_quantile']==1) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()
context.shorts=context.output[(context.output['ret_quantile']==context.nq) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()

def rebalance(context, data):
"""
Rebalance daily.
"""
Universe500=context.output.index.tolist()

existing_longs=0
existing_shorts=0
for security in context.portfolio.positions:
# Unwind stocks that have moved out of Q500US
if security not in Universe500 and data.can_trade(security):
order_target(security, 0)
else:
current_quantile=context.output['ret_quantile'].loc[security]
if context.portfolio.positions[security].amount>0:
if (current_quantile==1) and (security not in context.longs):
existing_longs += 1
elif (current_quantile>1) and (security not in context.shorts):
order_target(security, 0)
elif context.portfolio.positions[security].amount<0:
if (current_quantile==context.nq) and (security not in context.shorts):
existing_shorts += 1
elif (current_quantile<context.nq) and (security not in context.longs):
order_target(security, 0)

order_sids = get_open_orders().keys()
for security in context.longs:
if security in context.waits: continue
if security in order_sids:    continue
order_value(security, context.portfolio.cash / len(context.longs) )

for security in context.shorts:
if security in context.waits: continue
if security in order_sids:    continue
order_target_percent(security, -.5/(len(context.shorts)+existing_shorts))

def record_vars(context, data):
"""
Record variables at the end of each day.

longs = shorts = 0
for position in context.portfolio.positions.itervalues():
if position.amount > 0:
longs += 1
elif position.amount < 0:
shorts += 1
"""

record(
leverage=context.account.leverage,
long_count  = len(context.longs),
short_count = len(context.shorts),
)

#log.info("Today's shorts: "  +", ".join([short_.symbol for short_ in context.shorts]))
#log.info("Today's longs: "  +", ".join([long_.symbol for long_ in context.longs]))

def pvr(context, data):
''' Custom chart and/or logging of profit_vs_risk returns and related information
'''
# # # # # # # # # #  Options  # # # # # # # # # #
logging         = 0            # Info to logging window with some new maximums

record_pvr      = 1            # Profit vs Risk returns (percentage)
record_pvrp     = 0            # PvR (p)roportional neg cash vs portfolio value
record_cash     = 1            # Cash available
record_max_lvrg = 1            # Maximum leverage encountered
record_risk_hi  = 0            # Highest risk overall
record_shorting = 0            # Total value of any shorts
record_max_shrt = 1            # Max value of shorting total
record_cash_low = 1            # Any new lowest cash level
record_q_return = 0            # Quantopian returns (percentage)
record_pnl      = 0            # Profit-n-Loss
record_risk     = 0            # Risked, max cash spent or shorts beyond longs+cash
record_leverage = 0            # Leverage (context.account.leverage)
record_overshrt = 0            # Shorts beyond longs+cash
if record_pvrp: record_pvr = 0 # if pvrp is active, straight pvr is off

import time
from datetime import datetime
from pytz import timezone      # Python will only do once, makes this portable.
#   Move to top of algo for better efficiency.
c = context  # Brevity is the soul of wit -- Shakespeare [for readability]
if 'pvr' not in c:
date_strt = get_environment('start').date()
date_end  = get_environment('end').date()
cash_low  = c.portfolio.starting_cash
c.cagr    = 0.0
c.pvr     = {
'pvr'        : 0,      # Profit vs Risk returns based on maximum spent
'max_lvrg'   : 0,
'max_shrt'   : 0,
'risk_hi'    : 0,
'days'       : 0.0,
'date_prv'   : '',
'date_end'   : date_end,
'cash_low'   : cash_low,
'cash'       : cash_low,
'start'      : cash_low,
'pstart'     : c.portfolio.portfolio_value, # Used if restart
'begin'      : time.time(),                 # For run time
'log_summary': 126,                         # Summary every x days
'run_str'    : '{} to {}  ${} {} US/Eastern'.format(date_strt, date_end, int(cash_low), datetime.now(timezone('US/Eastern')).strftime("%Y-%m-%d %H:%M")) } log.info(c.pvr['run_str']) def _pvr(c): c.cagr = ((c.portfolio.portfolio_value / c.pvr['start']) ** (1 / (c.pvr['days'] / 252.))) - 1 ptype = 'PvR' if record_pvr else 'PvRp' log.info('{} {} %/day cagr {} Portfolio value {} PnL {}'.format(ptype, '%.4f' % (c.pvr['pvr'] / c.pvr['days']), '%.1f' % c.cagr, '%.0f' % c.portfolio.portfolio_value, '%.0f' % (c.portfolio.portfolio_value - c.pvr['start']))) log.info(' Profited {} on {} activated/transacted for PvR of {}%'.format('%.0f' % (c.portfolio.portfolio_value - c.pvr['start']), '%.0f' % c.pvr['risk_hi'], '%.1f' % c.pvr['pvr'])) log.info(' QRet {} PvR {} CshLw {} MxLv {} RskHi {} MxShrt {}'.format('%.2f' % q_rtrn, '%.2f' % c.pvr['pvr'], '%.0f' % c.pvr['cash_low'], '%.2f' % c.pvr['max_lvrg'], '%.0f' % c.pvr['risk_hi'], '%.0f' % c.pvr['max_shrt'])) def _minut(): dt = get_datetime().astimezone(timezone('US/Eastern')) return str((dt.hour * 60) + dt.minute - 570).rjust(3) # (-570 = 9:31a) date = get_datetime().date() if c.pvr['date_prv'] != date: c.pvr['date_prv'] = date c.pvr['days'] += 1.0 do_summary = 0 if c.pvr['log_summary'] and c.pvr['days'] % c.pvr['log_summary'] == 0 and _minut() == '100': do_summary = 1 # Log summary every x days if do_summary or date == c.pvr['date_end']: c.pvr['cash'] = c.portfolio.cash elif c.pvr['cash'] == c.portfolio.cash and not logging: return # for speed longs = sum([p.amount * p.last_sale_price for s, p in c.portfolio.positions.items() if p.amount > 0]) shorts = sum([p.amount * p.last_sale_price for s, p in c.portfolio.positions.items() if p.amount < 0]) q_rtrn = 100 * (c.portfolio.portfolio_value - c.pvr['start']) / c.pvr['start'] cash = c.portfolio.cash new_risk_hi = 0 new_max_lv = 0 new_max_shrt = 0 new_cash_low = 0 # To trigger logging in cash_low case overshorts = 0 # Shorts value beyond longs plus cash cash_dip = int(max(0, c.pvr['pstart'] - cash)) risk = int(max(cash_dip, -shorts)) if record_pvrp and cash < 0: # Let negative cash ding less when portfolio is up. cash_dip = int(max(0, c.pvr['start'] - cash * c.pvr['start'] / c.portfolio.portfolio_value)) # Imagine: Start with 10, grows to 1000, goes negative to -10, should not be 200% risk. if int(cash) < c.pvr['cash_low']: # New cash low new_cash_low = 1 c.pvr['cash_low'] = int(cash) # Lowest cash level hit if record_cash_low: record(CashLow = c.pvr['cash_low']) if c.account.leverage > c.pvr['max_lvrg']: new_max_lv = 1 c.pvr['max_lvrg'] = c.account.leverage # Maximum intraday leverage if record_max_lvrg: record(MaxLv = c.pvr['max_lvrg']) if shorts < c.pvr['max_shrt']: new_max_shrt = 1 c.pvr['max_shrt'] = shorts # Maximum shorts value if record_max_shrt: record(MxShrt = c.pvr['max_shrt']) if risk > c.pvr['risk_hi']: new_risk_hi = 1 c.pvr['risk_hi'] = risk # Highest risk overall if record_risk_hi: record(RiskHi = c.pvr['risk_hi']) # Profit_vs_Risk returns based on max amount actually spent (risk high) if c.pvr['risk_hi'] != 0: # Avoid zero-divide c.pvr['pvr'] = 100 * (c.portfolio.portfolio_value - c.pvr['start']) / c.pvr['risk_hi'] ptype = 'PvRp' if record_pvrp else 'PvR' if record_pvr or record_pvrp: record(**{ptype: c.pvr['pvr']}) if shorts > longs + cash: overshorts = shorts # Shorts when too high if record_shorting: record(Shorts = shorts) # Shorts value as a positve if record_overshrt: record(OvrShrt = overshorts) # Shorts beyond payable if record_leverage: record(Lvrg = c.account.leverage) # Leverage if record_cash: record(Cash = cash) # Cash if record_risk: record(Risk = risk) # Amount in play, maximum of shorts or cash used if record_q_return: record(QRet = q_rtrn) # Quantopian returns to compare to pvr returns curve if record_pnl: record(PnL = c.portfolio.portfolio_value - c.pvr['start']) # Profit|Loss if logging and (new_risk_hi or new_cash_low or new_max_lv or new_max_shrt): csh = ' Cash ' + '%.0f' % cash risk = ' Risk ' + '%.0f' % risk qret = ' QRet ' + '%.1f' % q_rtrn shrt = ' Shrt ' + '%.0f' % shorts ovrshrt = ' oShrt ' + '%.0f' % overshorts lv = ' Lv ' + '%.1f' % c.account.leverage pvr = ' PvR ' + '%.1f' % c.pvr['pvr'] rsk_hi = ' RskHi ' + '%.0f' % c.pvr['risk_hi'] csh_lw = ' CshLw ' + '%.0f' % c.pvr['cash_low'] mxlv = ' MxLv ' + '%.2f' % c.pvr['max_lvrg'] mxshrt = ' MxShrt ' + '%.0f' % c.pvr['max_shrt'] pnl = ' PnL ' + '%.0f' % (c.portfolio.portfolio_value - c.pvr['start']) log.info('{}{}{}{}{}{}{}{}{}{}{}{}{}'.format(_minut(), lv, mxlv, qret, pvr, pnl, csh, csh_lw, shrt, mxshrt, ovrshrt, risk, rsk_hi)) if do_summary: _pvr(c) if get_datetime() == get_environment('end'): # Summary at end of run _pvr(c) elapsed = (time.time() - c.pvr['begin']) / 60 # minutes log.info( '{}\nRuntime {} hr {} min'.format(c.pvr['run_str'], int(elapsed / 60), '%.1f' % (elapsed % 60))) #def handle_data(context, data): # pvr(context, data)  There was a runtime error. Disclaimer: I am quite new in Quantopian this is my first algo post, so forgive me if there is something too wrong! I have adapted the original algorithm in order to be able to use it automatically with Robinhood Gold once the API is ready for it. The changes are: a) RH is long_only, so I have made the long/short ratio 1.0/0.0 b) I used SPY as benchmark c) I have added standard slippage and commissions similar to FINRA's and SEC's (I put 0.00019$ per share, see here)
d) I have also added Grant's "profitable" filter (thanks!)

Here are the results I get for the longest possible backtest. They are pretty decent even when starting with 25,000 I have some questions about my results: 1) The results are good, but I am concerned about the returns since 2014. For example, 2y returns are almost always worse than the benchmark since January 2016; before then the average was 43% above the benchmark and never negative at all, almost always above +20%. Should I be worried about this? Maybe the algorithm no longer works that well? 2) Is there anything I may be missing for implementing it in RH or it is basically "ready to go" automatically when adding the RH specific lines? I guess I would just need to size my orders depending on the buying power. Thanks a lot for your attention! 93 Loading... Backtest from to with initial capital Total Returns -- Alpha -- Beta -- Sharpe -- Sortino -- Max Drawdown -- Benchmark Returns -- Volatility --  Returns 1 Month 3 Month 6 Month 12 Month  Alpha 1 Month 3 Month 6 Month 12 Month  Beta 1 Month 3 Month 6 Month 12 Month  Sharpe 1 Month 3 Month 6 Month 12 Month  Sortino 1 Month 3 Month 6 Month 12 Month  Volatility 1 Month 3 Month 6 Month 12 Month  Max Drawdown 1 Month 3 Month 6 Month 12 Month """ This algorithm enhances a simple five-day mean reversion strategy by: 1. Skipping the last day's return 2. Sorting stocks based on the volatility of the five day return, to get steady moves vs jumpy ones I also commented out two other filters that I looked at: 1. Six month volatility 2. Liquidity (volume/(shares outstanding)) """ import numpy as np import pandas as pd from quantopian.pipeline import Pipeline from quantopian.pipeline import CustomFactor from quantopian.algorithm import attach_pipeline, pipeline_output from quantopian.pipeline.data.builtin import USEquityPricing from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume from quantopian.pipeline.data import morningstar from quantopian.pipeline.filters import Q500US def initialize(context): # Set benchmark to short-term Treasury note ETF (SHY) since strategy is dollar neutral set_benchmark(sid(8554)) # Schedule our rebalance function to run at the end of each day. schedule_function(my_rebalance, date_rules.every_day(), time_rules.market_close()) # schedule_function(my_rebalance, date_rules.every_day(), time_rules.market_open()) # Record variables at the end of each day. schedule_function(my_record_vars, date_rules.every_day(), time_rules.market_close()) # schedule_function(my_record_vars, date_rules.every_day(), time_rules.market_open()) # Get intraday prices today before the close if you are not skipping the most recent data schedule_function(get_prices,date_rules.every_day(), time_rules.market_close(minutes=5)) # schedule_function(get_prices,date_rules.every_day(), time_rules.market_close()) # Set commissions and slippage to 0 to determine pure alpha set_commission(commission.PerShare(cost=0.000119, min_trade_cost=0.01)) #set_slippage(slippage.FixedSlippage(spread=0)) # Number of quantiles for sorting returns for mean reversion context.nq=5 # Number of quantiles for sorting volatility over five-day mean reversion period context.nq_vol=3 #context.account.leverage = 1.0 context.longs=() context.shorts=() # Create our pipeline and attach it to our algorithm. my_pipe = make_pipeline() attach_pipeline(my_pipe, 'my_pipeline') class Volatility(CustomFactor): inputs = [USEquityPricing.close] window_length=132 def compute(self, today, assets, out, close): # I compute 6-month volatility, starting before the five-day mean reversion period daily_returns = np.log(close[1:-6]) - np.log(close[0:-7]) out[:] = daily_returns.std(axis = 0) class Liquidity(CustomFactor): inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding] window_length = 1 def compute(self, today, assets, out, volume, shares): out[:] = volume[-1]/shares[-1] class Sector(CustomFactor): inputs=[morningstar.asset_classification.morningstar_sector_code] window_length=1 def compute(self, today, assets, out, sector): out[:] = sector[-1] def make_pipeline(): """ Create our pipeline. """ pricing=USEquityPricing.close.latest # Volatility filter (I made it sector neutral to replicate what UBS did). Uncomment and # change the percentile bounds as you would like before adding to 'universe' # vol=Volatility(mask=Q500US()) # sector=morningstar.asset_classification.morningstar_sector_code.latest # vol=vol.zscore(groupby=sector) # vol_filter=vol.percentile_between(0,100) # Liquidity filter (Uncomment and change the percentile bounds as you would like before # adding to 'universe' # liquidity=Liquidity(mask=Q500US()) # I included NaN in liquidity filter because of the large amount of missing data for shares out # liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan() profitable = morningstar.valuation_ratios.ev_to_ebitda.latest > 0 universe = ( Q500US() & (pricing > 5) & profitable # & liquidity_filter # & volatility_filter ) return Pipeline( screen=universe ) def before_trading_start(context, data): # Gets our pipeline output every day. context.output = pipeline_output('my_pipeline') def get_prices(context, data): # Get the last 6 days of prices for every stock in our universe Universe500=context.output.index.tolist() prices = data.history(Universe500,'price',6,'1d') daily_rets=np.log(prices/prices.shift(1)) rets=(prices.iloc[-2] - prices.iloc[0]) / prices.iloc[0] # I used data.history instead of Pipeline to get historical prices so you can have the # option of using the intraday price just before the close to get the most recent return. # In my post, I argue that you generally get better results when you skip that return. # If you don't want to skip the most recent return, however, use .iloc[-1] instead of .iloc[-2]: # rets=(prices.iloc[-1] - prices.iloc[0]) / prices.iloc[0] stdevs=daily_rets.std(axis=0) rets_df=pd.DataFrame(rets,columns=['five_day_ret']) stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret']) context.output=context.output.join(rets_df,how='outer') context.output=context.output.join(stdevs_df,how='outer') context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1 context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1 context.longs=context.output[(context.output['ret_quantile']==1) & (context.output['stdev_quantile']<context.nq_vol)].index.tolist() context.shorts=context.output[(context.output['ret_quantile']==context.nq) & (context.output['stdev_quantile']<context.nq_vol)].index.tolist() def my_rebalance(context, data): """ Rebalance daily. """ Universe500=context.output.index.tolist() existing_longs=0 existing_shorts=0 for security in context.portfolio.positions: # Unwind stocks that have moved out of Q500US if security not in Universe500 and data.can_trade(security): order_target_percent(security, 0) else: if data.can_trade(security): current_quantile=context.output['ret_quantile'].loc[security] if context.portfolio.positions[security].amount>0: if (current_quantile==1) and (security not in context.longs): existing_longs += 1 elif (current_quantile>1) and (security not in context.shorts): order_target_percent(security, 0) elif context.portfolio.positions[security].amount<0: if (current_quantile==context.nq) and (security not in context.shorts): existing_shorts += 1 elif (current_quantile<context.nq) and (security not in context.longs): order_target_percent(security, 0) # if len(context.longs)>0: for security in context.longs: if data.can_trade(security): order_target_percent(security, 1.0/(len(context.longs)+existing_longs)) # if len(context.shorts)>0: for security in context.shorts: if data.can_trade(security): order_target_percent(security, 0.0/(len(context.shorts)+existing_shorts)) def my_record_vars(context, data): """ Record variables at the end of each day. """ longs = shorts = 0 for position in context.portfolio.positions.itervalues(): if position.amount > 0: longs += 1 elif position.amount < 0: shorts += 1 # Record our variables. record(leverage=context.account.leverage, long_count=longs, short_count=shorts) log.info("Today's shorts: " +", ".join([short_.symbol for short_ in context.shorts])) log.info("Today's longs: " +", ".join([long_.symbol for long_ in context.longs])) There was a runtime error. @Jaime, you have a high beta (1.13), which means your results are very closely aligned to the market (and therefore at its mercy). You'd want beta between -0.3 and 0.3. Also, it looks like you're not doing any shorts, which may be why you have such a high beta. Just putting this out there as a tenacious backtester and crappy coder, looking for that bucket of gold: One of my 'go to' backtests is to see if any of the algos are viable, one is starting the backtest in a shitty post 2008 environment , i start in august 1st 2014 as I see a lot of them plummet during this period and want to see if they can recover and still beat the SPY, and then I use reasonable capital, 10K tops, because ideally a winning strategy truly 'levels' the playing field for the rest of us who don't have 25K plus of expendable cash. Try the ones above. Figured I'd post this here since I've developed well past this. I've gotten better 2015 performance out of newer algorithms and a few old ones that finish at around 15000-18000. The algorithm is long unless SPY has a low enough velocity. 280 Loading... Backtest from to with initial capital Total Returns -- Alpha -- Beta -- Sharpe -- Sortino -- Max Drawdown -- Benchmark Returns -- Volatility --  Returns 1 Month 3 Month 6 Month 12 Month  Alpha 1 Month 3 Month 6 Month 12 Month  Beta 1 Month 3 Month 6 Month 12 Month  Sharpe 1 Month 3 Month 6 Month 12 Month  Sortino 1 Month 3 Month 6 Month 12 Month  Volatility 1 Month 3 Month 6 Month 12 Month  Max Drawdown 1 Month 3 Month 6 Month 12 Month import numpy as np import pandas as pd from quantopian.pipeline import Pipeline from quantopian.pipeline import CustomFactor from quantopian.algorithm import attach_pipeline, pipeline_output from quantopian.pipeline.data.builtin import USEquityPricing from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume from quantopian.pipeline.data import morningstar from quantopian.pipeline.filters import Q500US def initialize(context): set_benchmark(sid(8554)) schedule_function(my_rebalance, date_rules.every_day(), time_rules.market_close(hours = 5, minutes = 1)) schedule_function(close_, date_rules.every_day(), time_rules.market_close(hours = 5, minutes = 1)) schedule_function(open_, date_rules.every_day(), time_rules.market_close(hours = 5)) schedule_function(my_record_vars, date_rules.every_day(), time_rules.market_close()) schedule_function(get_prices,date_rules.every_day(), time_rules.market_close(hours=5, minutes = 2 )) set_commission(commission.PerShare(cost=0, min_trade_cost=0)) set_slippage(slippage.FixedSlippage(spread=0)) context.nq=5 context.nq_vol=3 my_pipe = make_pipeline() attach_pipeline(my_pipe, 'my_pipeline') context.max_leverage = [0] class Volatility(CustomFactor): inputs = [USEquityPricing.close] window_length=132 def compute(self, today, assets, out, close): daily_returns = np.log(close[1:-6]) - np.log(close[0:-7]) out[:] = daily_returns.std(axis = 0) class Liquidity(CustomFactor): inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding] window_length = 1 def compute(self, today, assets, out, volume, shares): out[:] = volume[-1]/shares[-1] class Sector(CustomFactor): inputs=[morningstar.asset_classification.morningstar_sector_code] window_length=1 def compute(self, today, assets, out, sector): out[:] = sector[-1] def make_pipeline(): pricing=USEquityPricing.close.latest # vol=Volatility(mask=Q500US()) # sector=morningstar.asset_classification.morningstar_sector_code.latest # vol=vol.zscore(groupby=sector) # vol_filter=vol.percentile_between(0,100) # liquidity=Liquidity(mask=Q500US()) # liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan() universe = ( Q500US() # & liquidity_filter # & volatility_filter ) return Pipeline( screen=universe) def before_trading_start(context, data): context.output = pipeline_output('my_pipeline') def get_prices(context, data): Universe500=context.output.index.tolist() prices = data.history(Universe500,'price',6,'1d') daily_rets=np.log(prices/prices.shift(1)) rets=(prices.iloc[-2] - prices.iloc[0]) / prices.iloc[0] # If you don't want to skip the most recent return, however, use .iloc[-1] instead of .iloc[-2]: # rets=(prices.iloc[-1] - prices.iloc[0]) / prices.iloc[0] stdevs=daily_rets.std(axis=0) rets_df=pd.DataFrame(rets,columns=['five_day_ret']) stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret']) context.output=context.output.join(rets_df,how='outer') context.output=context.output.join(stdevs_df,how='outer') context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1 context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1 context.longs=context.output[(context.output['ret_quantile']==1) & (context.output['stdev_quantile']<context.nq_vol)].index.tolist() context.shorts=context.output[(context.output['ret_quantile']==context.nq) & (context.output['stdev_quantile']<context.nq_vol)].index.tolist() def my_rebalance(context, data): Universe500=context.output.index.tolist() context.existing_longs=0 context.existing_shorts=0 for security in context.portfolio.positions: if security not in Universe500 and data.can_trade(security): order_target_percent(security, 0) else: if data.can_trade(security): current_quantile=context.output['ret_quantile'].loc[security] if context.portfolio.positions[security].amount>0: if (current_quantile==1) and (security not in context.longs): context.existing_longs += 1 elif (current_quantile>1) and (security not in context.shorts): order_target_percent(security, 0) elif context.portfolio.positions[security].amount<0: if (current_quantile==context.nq) and (security not in context.shorts): context.existing_shorts += 1 elif (current_quantile<context.nq) and (security not in context.longs): order_target_percent(security, 0) def close_(context, data): SPY = [sid(8554)] SPY_Velocity = 0 long_leverage = 0 short_leverage = 0 for security in SPY: pri = data.history(SPY, "price",200, "1d") pos = 'pri[security]' pos_one = (pri[security][-1]) pos_six = (pri[security][-75:].mean()) #VELOCITY velocity_stop = (pos_one - pos_six)/100.0 SPY_Velocity = velocity_stop if SPY_Velocity > 0.0: long_leverage = 1.8 short_leverage = -0.0 else: long_leverage = 1.1 short_leverage = -0.7 for security in context.shorts: if data.can_trade(security): order_target_percent(security, short_leverage/(len(context.shorts)+context.existing_shorts)) def open_(context, data): SPY = [sid(8554)] SPY_Velocity = 0 long_leverage = 0 short_leverage = 0 for security in SPY: pri = data.history(SPY, "price",200, "1d") pos = 'pri[security]' pos_one = (pri[security][-1]) pos_six = (pri[security][-75:].mean()) #VELOCITY velocity_stop = (pos_one - pos_six)/100.0 SPY_Velocity = velocity_stop if SPY_Velocity > 0.0: long_leverage = 1.8 short_leverage = -0.0 else: long_leverage = 1.1 short_leverage = -0.7 for security in context.longs: if data.can_trade(security): order_target_percent(security, long_leverage/(len(context.longs)+context.existing_longs)) def my_record_vars(context, data): longs = shorts = 0 for position in context.portfolio.positions.itervalues(): if position.amount > 0: longs += 1 elif position.amount < 0: shorts += 1 leverage = context.account.leverage for num in context.max_leverage: if leverage > num: context.max_leverage.remove(num) context.max_leverage.append(leverage) record(Max_Leverage = context.max_leverage[-1]) record(leverage=context.account.leverage, long_count=longs, short_count=shorts) log.info("Today's shorts: " +", ".join([short_.symbol for short_ in context.shorts])) log.info("Today's longs: " +", ".join([long_.symbol for long_ in context.longs])) There was a runtime error. Oh and this one just makes me giggle when I look at it - maybe you will too. 280 Loading... Backtest from to with initial capital Total Returns -- Alpha -- Beta -- Sharpe -- Sortino -- Max Drawdown -- Benchmark Returns -- Volatility --  Returns 1 Month 3 Month 6 Month 12 Month  Alpha 1 Month 3 Month 6 Month 12 Month  Beta 1 Month 3 Month 6 Month 12 Month  Sharpe 1 Month 3 Month 6 Month 12 Month  Sortino 1 Month 3 Month 6 Month 12 Month  Volatility 1 Month 3 Month 6 Month 12 Month  Max Drawdown 1 Month 3 Month 6 Month 12 Month import numpy as np import pandas as pd from quantopian.pipeline import Pipeline from quantopian.pipeline import CustomFactor from quantopian.algorithm import attach_pipeline, pipeline_output from quantopian.pipeline.data.builtin import USEquityPricing from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume from quantopian.pipeline.data import morningstar from quantopian.pipeline.filters import Q500US def initialize(context): set_benchmark(sid(8554)) schedule_function(my_rebalance, date_rules.every_day(), time_rules.market_close(hours = 5, minutes = 1)) schedule_function(close_, date_rules.every_day(), time_rules.market_close(hours = 5, minutes = 1)) schedule_function(open_, date_rules.every_day(), time_rules.market_close(hours = 5)) schedule_function(my_record_vars, date_rules.every_day(), time_rules.market_close()) schedule_function(get_prices,date_rules.every_day(), time_rules.market_close(hours=5, minutes = 2 )) set_commission(commission.PerShare(cost=0, min_trade_cost=0)) set_slippage(slippage.FixedSlippage(spread=0)) context.nq=5 context.nq_vol=3 my_pipe = make_pipeline() attach_pipeline(my_pipe, 'my_pipeline') context.max_leverage = [0] class Volatility(CustomFactor): inputs = [USEquityPricing.close] window_length=132 def compute(self, today, assets, out, close): daily_returns = np.log(close[1:-6]) - np.log(close[0:-7]) out[:] = daily_returns.std(axis = 0) class Liquidity(CustomFactor): inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding] window_length = 1 def compute(self, today, assets, out, volume, shares): out[:] = volume[-1]/shares[-1] class Sector(CustomFactor): inputs=[morningstar.asset_classification.morningstar_sector_code] window_length=1 def compute(self, today, assets, out, sector): out[:] = sector[-1] def make_pipeline(): pricing=USEquityPricing.close.latest # vol=Volatility(mask=Q500US()) # sector=morningstar.asset_classification.morningstar_sector_code.latest # vol=vol.zscore(groupby=sector) # vol_filter=vol.percentile_between(0,100) # liquidity=Liquidity(mask=Q500US()) # liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan() universe = ( Q500US() & (pricing < 10) # & liquidity_filter # & volatility_filter ) return Pipeline( screen=universe) def before_trading_start(context, data): context.output = pipeline_output('my_pipeline') def get_prices(context, data): Universe500=context.output.index.tolist() prices = data.history(Universe500,'price',6,'1d') daily_rets=np.log(prices/prices.shift(1)) rets=(prices.iloc[-2] - prices.iloc[0]) / prices.iloc[0] # If you don't want to skip the most recent return, however, use .iloc[-1] instead of .iloc[-2]: # rets=(prices.iloc[-1] - prices.iloc[0]) / prices.iloc[0] stdevs=daily_rets.std(axis=0) rets_df=pd.DataFrame(rets,columns=['five_day_ret']) stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret']) context.output=context.output.join(rets_df,how='outer') context.output=context.output.join(stdevs_df,how='outer') context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1 context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1 context.longs=context.output[(context.output['ret_quantile']==1) & (context.output['stdev_quantile']<context.nq_vol)].index.tolist() context.shorts=context.output[(context.output['ret_quantile']==context.nq) & (context.output['stdev_quantile']<context.nq_vol)].index.tolist() def my_rebalance(context, data): Universe500=context.output.index.tolist() context.existing_longs=0 context.existing_shorts=0 for security in context.portfolio.positions: if security not in Universe500 and data.can_trade(security): order_target_percent(security, 0) else: if data.can_trade(security): current_quantile=context.output['ret_quantile'].loc[security] if context.portfolio.positions[security].amount>0: if (current_quantile==1) and (security not in context.longs): context.existing_longs += 1 elif (current_quantile>1) and (security not in context.shorts): order_target_percent(security, 0) elif context.portfolio.positions[security].amount<0: if (current_quantile==context.nq) and (security not in context.shorts): context.existing_shorts += 1 elif (current_quantile<context.nq) and (security not in context.longs): order_target_percent(security, 0) def close_(context, data): SPY = [sid(8554)] SPY_Velocity = 0 long_leverage = 0 short_leverage = 0 for security in SPY: pri = data.history(SPY, "price",200, "1d") pos = 'pri[security]' pos_one = (pri[security][-1]) pos_six = (pri[security][-75:].mean()) #VELOCITY velocity_stop = (pos_one - pos_six)/100.0 SPY_Velocity = velocity_stop if SPY_Velocity > 0.0: long_leverage = 1.8 short_leverage = -0.0 else: long_leverage = 1.1 short_leverage = -0.7 for security in context.shorts: if data.can_trade(security): order_target_percent(security, short_leverage/(len(context.shorts)+context.existing_shorts)) def open_(context, data): SPY = [sid(8554)] SPY_Velocity = 0 long_leverage = 0 short_leverage = 0 for security in SPY: pri = data.history(SPY, "price",200, "1d") pos = 'pri[security]' pos_one = (pri[security][-1]) pos_six = (pri[security][-75:].mean()) #VELOCITY velocity_stop = (pos_one - pos_six)/100.0 SPY_Velocity = velocity_stop if SPY_Velocity > 0.0: long_leverage = 1.8 short_leverage = -0.0 else: long_leverage = 1.1 short_leverage = -0.7 for security in context.longs: if data.can_trade(security): order_target_percent(security, long_leverage/(len(context.longs)+context.existing_longs)) def my_record_vars(context, data): longs = shorts = 0 for position in context.portfolio.positions.itervalues(): if position.amount > 0: longs += 1 elif position.amount < 0: shorts += 1 leverage = context.account.leverage for num in context.max_leverage: if leverage > num: context.max_leverage.remove(num) context.max_leverage.append(leverage) record(Max_Leverage = context.max_leverage[-1]) record(leverage=context.account.leverage, long_count=longs, short_count=shorts) log.info("Today's shorts: " +", ".join([short_.symbol for short_ in context.shorts])) log.info("Today's longs: " +", ".join([long_.symbol for long_ in context.longs])) There was a runtime error. Would like to ask whether the code up two could trade. It started with 10k and was sometimes well into positive cash and yet borrowing 300k by 2010. Can it be done? Maybe someone could relate it to broker rules? I had a tough time understanding when reading them. @Blue, by 2010, the returns were 5000%, which leaves us at500,000. borrowing 300,000 would leave us at 1.6 leverage, which is definitely allowed. (assuming we are within our margin maintenance)

Hi,
How is Q500US dealing withe the survival bias? Are stocks of bankrupted companies included or it just includes stocks which were never delisted?

This is the crucial element to determine if the long positions profits are realistic. For shorts it should have the opposite effect so it would actually help. I am surprised nobody has discussed this yet. The Universe for any strategy should include delisted stocks or the results will be completely meaningless.

Quantopian is survivorship-bias free.

On the topic of "Enhancing Short-Term Mean Reversion Strategies" - if anyone has some interesting thoughts on dealing with stop losses, I would like to hear them. Typically as losses build up, mean reversion signals grow stronger. How then should losses be handled?

Hello, I added a filter to skip big drawdowns of SPY, got from another thread i can't find the link! It gets much better for risk, max drawdown of 16%, beta 0.45. Again it is a strategy to implement in RobinHood, only longs and no leverage. What do you guys think? Thanks in advance!

190
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""

This algorithm enhances a simple five-day mean reversion strategy by:
1. Skipping the last day's return
2. Sorting stocks based on the volatility of the five day return, to get steady moves vs jumpy ones
I also commented out two other filters that I looked at:
1. Six month volatility
2. Liquidity (volume/(shares outstanding))

"""

import numpy as np
import pandas as pd
from quantopian.pipeline import Pipeline
from quantopian.pipeline import CustomFactor
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume
from quantopian.pipeline.data import morningstar
from quantopian.pipeline.filters import Q500US

def initialize(context):

# Set benchmark to short-term Treasury note ETF (SHY) since strategy is dollar neutral
set_benchmark(sid(8554))

# Schedule our rebalance function to run at the end of each day.
schedule_function(my_rebalance, date_rules.every_day(), time_rules.market_close())
#    schedule_function(my_rebalance, date_rules.every_day(), time_rules.market_open())

# Record variables at the end of each day.
schedule_function(my_record_vars, date_rules.every_day(), time_rules.market_close())
#    schedule_function(my_record_vars, date_rules.every_day(), time_rules.market_open())

# Get intraday prices today before the close if you are not skipping the most recent data
schedule_function(get_prices,date_rules.every_day(), time_rules.market_close(minutes=30))
#    schedule_function(get_prices,date_rules.every_day(), time_rules.market_close())

# Set commissions and slippage to 0 to determine pure alpha

# Number of quantiles for sorting returns for mean reversion
context.nq=5

# Number of quantiles for sorting volatility over five-day mean reversion period
context.nq_vol=3

#context.account.leverage = 1.0
context.longs=()
context.shorts=()
context.stock = symbol("SPY")
context.stop_price_bull = 0
context.stop_price_bear = 0
context.stop_pct_bull = 0.90
context.stop_pct_bear = 0.90
context.flag = 0.0

# Create our pipeline and attach it to our algorithm.
my_pipe = make_pipeline()
attach_pipeline(my_pipe, 'my_pipeline')

class Volatility(CustomFactor):
inputs = [USEquityPricing.close]
window_length=132

def compute(self, today, assets, out, close):
# I compute 6-month volatility, starting before the five-day mean reversion period
daily_returns = np.log(close[1:-6]) - np.log(close[0:-7])
out[:] = daily_returns.std(axis = 0)

class Liquidity(CustomFactor):
inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding]
window_length = 1

def compute(self, today, assets, out, volume, shares):
out[:] = volume[-1]/shares[-1]

class Sector(CustomFactor):
inputs=[morningstar.asset_classification.morningstar_sector_code]
window_length=1

def compute(self, today, assets, out, sector):
out[:] = sector[-1]

def make_pipeline():
"""
Create our pipeline.
"""

pricing=USEquityPricing.close.latest

# Volatility filter (I made it sector neutral to replicate what UBS did).  Uncomment and
# change the percentile bounds as you would like before adding to 'universe'
sector=morningstar.asset_classification.morningstar_sector_code.latest
vol=vol.zscore(groupby=sector)
vol_filter=vol.percentile_between(0,100)
volatility_filter=vol_filter

# Liquidity filter (Uncomment and change the percentile bounds as you would like before
# adding to 'universe'
# I included NaN in liquidity filter because of the large amount of missing data for shares out
liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan()

profitable = morningstar.valuation_ratios.ev_to_ebitda.latest > 0

universe = (
Q500US()
& (pricing > 5) & profitable
& liquidity_filter
& volatility_filter
)

return Pipeline(
screen=universe
)

# Gets our pipeline output every day.
context.output = pipeline_output('my_pipeline')

def get_prices(context, data):
# Get the last 6 days of prices for every stock in our universe
Universe500=context.output.index.tolist()
prices = data.history(Universe500,'price',6,'1d')
daily_rets=np.log(prices/prices.shift(1))

rets=(prices.iloc[-2] - prices.iloc[0]) / prices.iloc[0]
# I used data.history instead of Pipeline to get historical prices so you can have the
# option of using the intraday price just before the close to get the most recent return.
# In my post, I argue that you generally get better results when you skip that return.
# If you don't want to skip the most recent return, however, use .iloc[-1] instead of .iloc[-2]:
# rets=(prices.iloc[-1] - prices.iloc[0]) / prices.iloc[0]

stdevs=daily_rets.std(axis=0)

rets_df=pd.DataFrame(rets,columns=['five_day_ret'])
stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret'])

context.output=context.output.join(rets_df,how='outer')
context.output=context.output.join(stdevs_df,how='outer')

context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1
context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1

context.longs=context.output[(context.output['ret_quantile']==1) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()
context.shorts=context.output[(context.output['ret_quantile']==context.nq) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()

def my_rebalance(context, data):
"""
Rebalance daily.
"""
#Moving averaged to determine trend (size of stop-loss and entry)
ma_trend1 = data[context.stock].mavg(200)
ma_trend2 = data[context.stock].mavg(400)
#Moving averages for entry
ma_slow_bull = data[context.stock].mavg(50)
ma_slow_bear = data[context.stock].mavg(100)
ma_fast = data[context.stock].mavg(3)

set_trailing_stop(context, data)
if ma_trend1>ma_trend2:
if data[context.stock].price < context.stop_price_bull:
context.flag = 0
context.stop_price_bull = 0
# Buys when stock price comes above slow mavg
elif ma_fast > ma_slow_bull:
context.flag = 1
elif ma_trend1<ma_trend2:
if data[context.stock].price < context.stop_price_bear:
context.flag = 0
context.stop_price_bear = 0
# Buys when stock price comes above slow mavg
elif ma_fast > ma_slow_bear:
context.flag = 1

print context.flag

if context.flag == 0:
for stock in context.portfolio.positions:
#if data.can_trade(stock):# and stock.amount > 0:
order_target_percent(stock,0)
else:
Universe500=context.output.index.tolist()

existing_longs=0
existing_shorts=0
for security in context.portfolio.positions:
# Unwind stocks that have moved out of Q500US
if security not in Universe500 and data.can_trade(security):
order_target_percent(security, 0)
else:
current_quantile=context.output['ret_quantile'].loc[security]
if context.portfolio.positions[security].amount>0:
if (current_quantile==1) and (security not in context.longs):
existing_longs += 1
elif (current_quantile>1) and (security not in context.shorts):
order_target_percent(security, 0)
elif context.portfolio.positions[security].amount<0:
if (current_quantile==context.nq) and (security not in context.shorts):
existing_shorts += 1
elif (current_quantile<context.nq) and (security not in context.longs):
order_target_percent(security, 0)

for security in context.longs:
order_target_percent(security, 0.925/(len(context.longs)+existing_longs))
for security in context.shorts:
order_target_percent(security, 0.0/(len(context.shorts)+existing_shorts))

def set_trailing_stop(context, data):
price = data.current(context.stock, 'price')
context.stop_price_bull = max(context.stop_price_bull, context.stop_pct_bull * price)
context.stop_price_bear = max(context.stop_price_bear, context.stop_pct_bear * price)

def my_record_vars(context, data):
"""
Record variables at the end of each day.
"""
longs = shorts = 0
for position in context.portfolio.positions.itervalues():
if position.amount > 0:
longs += 1
elif position.amount < 0:
shorts += 1
# Record our variables.
record(leverage=context.account.leverage, long_count=longs, short_count=shorts)

#log.info("Today's shorts: "  +", ".join([short_.symbol for short_ in context.shorts]))
#log.info("Today's longs: "  +", ".join([long_.symbol for long_ in context.longs]))
log.info("Today's longs: "  +", ".join([long_.symbol for long_ in context.portfolio.positions]))

There was a runtime error.

@Jamie the latest algorithm has multiple deprecation issues that must be cleaned up before it can be live traded. A full list of issues are identified in the Log Output or at the end of a full back test.

Anyone try this on newly available futures data such as ES ?

@George Bilan
As rightly pointed out placing a stop loss in mean reversion goes against the principle of it as we expect the stock to change trend. However, you can calculate half life of the trade, i.e. a time it will take for stock to mean revert. If the stock doesn't mean revert in the given time frame then you can simply exit the trade. A more discussion about this can be found in a course by Dr. Ernest Chan on Mean Reversion Strategies.

Guy - I have tried to run your algo (https://www.quantopian.com/posts/enhancing-short-term-mean-reversion-strategies-1#58c864773821d40beeb57c0b) with the exact same settings, but the returns seem to be much different - would you know what could be causing this?
The only change I did was to set the benchmark to SPY

37
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
"""
Blue's version.  This algorithm enhances a simple five-day mean reversion strategy by:
1. Skipping the last day's return
2. Sorting stocks based on the volatility of the five day return, to get steady moves vs jumpy ones
I also commented out two other filters that I looked at:
1. Six month volatility
2. Liquidity (volume/(shares outstanding))

"""
import numpy as np
import pandas as pd
from quantopian.pipeline import Pipeline
from quantopian.pipeline import CustomFactor
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume
from quantopian.pipeline.data import morningstar
from quantopian.pipeline.filters import Q500US

def initialize(context):

# Set benchmark to short-term Treasury note ETF (SHY) since strategy is dollar neutral
set_benchmark(sid(8554))

# Get intraday prices today before the close if you are not skipping the most recent data
schedule_function(get_prices,date_rules.every_day(), time_rules.market_open(minutes=5))

# Schedule rebalance function to run at the end of each day.
schedule_function(rebalance, date_rules.every_day(), time_rules.market_open(minutes=10))

# Record variables at the end of each day.
schedule_function(record_vars, date_rules.every_day(), time_rules.market_close())

# Set commissions and slippage to 0 to determine pure alpha

# Number of quantiles for sorting returns for mean reversion
context.nq=5

# Number of quantiles for sorting volatility over five-day mean reversion period
context.nq_vol=3

# Create pipeline and attach it to algorithm.
pipe = make_pipeline()
attach_pipeline(pipe, 'pipeline')

context.waits = {}
context.waits_max = 3    # trading days

return

context.pnl_sids_exclude  = [ sid(2561) ]
context.pnl_sids  = [  ]
context.day_count = 0
#schedule_function(record_pnl, date_rules.every_day(), time_rules.market_close())

# Take profit several times a day
context.profit_threshold = .05
context.profit_logging   = 1
for i in range(20, 390, 60):    # (low, high, every i minutes)
#continue    # uncomment to deactivate
schedule_function(take_profit, date_rules.every_day(), time_rules.market_open(minutes=i))

def wait(c, sec=None, action=None):
if sec and action:
if action == 1:
c.waits[sec] = 1    # start wait
elif action == 0:
del c.waits[sec]    # end wait
else:
for sec in c.waits.copy():
if c.waits[sec] > c.waits_max:
del c.waits[sec]
else:
c.waits[sec] += 1   # increment

def take_profit(context, data):    # Close some positions to take profit
pos     = context.portfolio.positions
history = data.history(pos.keys(), 'close', 10, '1m').bfill().ffill()
for s in pos:
if not data.can_trade(s):      continue
if slope(history[s])      > 0: continue
if slope(history[s][-5:]) > 0: continue
if history[s][-1] > history[s][-2]: continue
prc = data.current(s, 'price')
amt = pos[s].amount
if (amt / abs(amt)) * ((prc / pos[s].cost_basis) - 1) > context.profit_threshold:
order_target(s, 0)
wait(context, s, 1)    # start wait
if not context.profit_logging: continue
pnl = (amt * (prc - pos[s].cost_basis))
if pnl < 3000: continue
log.info('close {} {}  cb {}  now {}  pnl {}'.format(
amt, s.symbol, '%.2f' % pos[s].cost_basis, prc, '%.0f' % pnl))

import statsmodels.api as sm
def slope(in_list):     # Return slope of regression line. [Make sure this list contains no nans]
return sm.OLS(in_list, sm.add_constant(range(-len(in_list) + 1, 1))).fit().params[-1]  # slope

def record_pnl(context, data):
def _pnl_value(sec, context, data):
pos = context.portfolio.positions[sec]
return pos.amount * (data.current(sec, 'price') - pos.cost_basis)

context.day_count += 1

for s in context.portfolio.positions:
if not data.can_trade(s): continue
if s in context.pnl_sids_exclude: continue

# periodically log all
if context.day_count % 126 == 0:
log.info('{} {}'.format(s.symbol, int(_pnl_value(s, context, data))))

# add up to 5 securities for record
if len(context.pnl_sids) < 5 and s not in context.pnl_sids:
context.pnl_sids.append(s)
if s not in context.pnl_sids: continue     # limit to only them

# record their profit and loss
who  = s.symbol
what = _pnl_value(s, context, data)
record( **{ who: what } )

class Volatility(CustomFactor):
inputs = [USEquityPricing.close]
window_length=132

def compute(self, today, assets, out, close):
# I compute 6-month volatility, starting before the five-day mean reversion period
daily_returns = np.log(close[1:-6]) - np.log(close[0:-7])
out[:] = daily_returns.std(axis = 0)

class Liquidity(CustomFactor):
inputs = [USEquityPricing.volume, morningstar.valuation.shares_outstanding]
window_length = 1

def compute(self, today, assets, out, volume, shares):
out[:] = volume[-1]/shares[-1]

class Sector(CustomFactor):
inputs=[morningstar.asset_classification.morningstar_sector_code]
window_length=1

def compute(self, today, assets, out, sector):
out[:] = sector[-1]

def make_pipeline():
"""
Create pipeline.
"""

pricing=USEquityPricing.close.latest

# Volatility filter (I made it sector neutral to replicate what UBS did).  Uncomment and
# change the percentile bounds as you would like before adding to 'universe'
# sector=morningstar.asset_classification.morningstar_sector_code.latest
# vol=vol.zscore(groupby=sector)
# vol_filter=vol.percentile_between(0,100)

# Liquidity filter (Uncomment and change the percentile bounds as you would like before
# adding to 'universe'
# I included NaN in liquidity filter because of the large amount of missing data for shares out
# liquidity_filter=liquidity.percentile_between(0,75) | liquidity.isnan()

universe = (
Q500US()
& (pricing > 5)
# & liquidity_filter
# & volatility_filter
)
return Pipeline(
screen  = universe
)

# Gets pipeline output every day.
context.output = pipeline_output('pipeline')

wait(context)    # Increment any that are present

def get_prices(context, data):
# Get the last 6 days of prices for every stock in universe
Universe500=context.output.index.tolist()
prices = data.history(Universe500,'price',6,'1d')
daily_rets = np.log(prices/prices.shift(1))

rets=(prices.iloc[-2] - prices.iloc[0]) / prices.iloc[0]
# I used data.history instead of Pipeline to get historical prices so you can have the
# option of using the intraday price just before the close to get the most recent return.
# In my post, I argue that you generally get better results when you skip that return.
# If you don't want to skip the most recent return, however, use .iloc[-1] instead of .iloc[-2]:
# rets=(prices.iloc[-1] - prices.iloc[0]) / prices.iloc[0]

stdevs=daily_rets.std(axis=0)

rets_df=pd.DataFrame(rets,columns=['five_day_ret'])
stdevs_df=pd.DataFrame(stdevs,columns=['stdev_ret'])

context.output=context.output.join(rets_df,how='outer')
context.output=context.output.join(stdevs_df,how='outer')

context.output['ret_quantile']=pd.qcut(context.output['five_day_ret'],context.nq,labels=False)+1
context.output['stdev_quantile']=pd.qcut(context.output['stdev_ret'],3,labels=False)+1

context.longs=context.output[(context.output['ret_quantile']==1) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()
context.shorts=context.output[(context.output['ret_quantile']==context.nq) &
(context.output['stdev_quantile']<context.nq_vol)].index.tolist()

def rebalance(context, data):
"""
Rebalance daily.
"""
Universe500=context.output.index.tolist()

existing_longs=0
existing_shorts=0
for security in context.portfolio.positions:
# Unwind stocks that have moved out of Q500US
if security not in Universe500 and data.can_trade(security):
order_target(security, 0)
else:
current_quantile=context.output['ret_quantile'].loc[security]
if context.portfolio.positions[security].amount>0:
if (current_quantile==1) and (security not in context.longs):
existing_longs += 1
elif (current_quantile>1) and (security not in context.shorts):
order_target(security, 0)
elif context.portfolio.positions[security].amount<0:
if (current_quantile==context.nq) and (security not in context.shorts):
existing_shorts += 1
elif (current_quantile<context.nq) and (security not in context.longs):
order_target(security, 0)

order_sids = get_open_orders().keys()
for security in context.longs:
if security in context.waits: continue
if security in order_sids:    continue
order_value(security, context.portfolio.cash / len(context.longs) )

for security in context.shorts:
if security in context.waits: continue
if security in order_sids:    continue
order_target_percent(security, -.5/(len(context.shorts)+existing_shorts))

def record_vars(context, data):
"""
Record variables at the end of each day.

longs = shorts = 0
for position in context.portfolio.positions.itervalues():
if position.amount > 0:
longs += 1
elif position.amount < 0:
shorts += 1
"""

record(
leverage=context.account.leverage,
long_count  = len(context.longs),
short_count = len(context.shorts),
)

#log.info("Today's shorts: "  +", ".join([short_.symbol for short_ in context.shorts]))
#log.info("Today's longs: "  +", ".join([long_.symbol for long_ in context.longs]))

def pvr(context, data):
''' Custom chart and/or logging of profit_vs_risk returns and related information
'''
# # # # # # # # # #  Options  # # # # # # # # # #
logging         = 0            # Info to logging window with some new maximums

record_pvr      = 1            # Profit vs Risk returns (percentage)
record_pvrp     = 0            # PvR (p)roportional neg cash vs portfolio value
record_cash     = 1            # Cash available
record_max_lvrg = 1            # Maximum leverage encountered
record_risk_hi  = 0            # Highest risk overall
record_shorting = 0            # Total value of any shorts
record_max_shrt = 1            # Max value of shorting total
record_cash_low = 1            # Any new lowest cash level
record_q_return = 0            # Quantopian returns (percentage)
record_pnl      = 0            # Profit-n-Loss
record_risk     = 0            # Risked, max cash spent or shorts beyond longs+cash
record_leverage = 0            # Leverage (context.account.leverage)
record_overshrt = 0            # Shorts beyond longs+cash
if record_pvrp: record_pvr = 0 # if pvrp is active, straight pvr is off

import time
from datetime import datetime
from pytz import timezone      # Python will only do once, makes this portable.
#   Move to top of algo for better efficiency.
c = context  # Brevity is the soul of wit -- Shakespeare [for readability]
if 'pvr' not in c:
date_strt = get_environment('start').date()
date_end  = get_environment('end').date()
cash_low  = c.portfolio.starting_cash
c.cagr    = 0.0
c.pvr     = {
'pvr'        : 0,      # Profit vs Risk returns based on maximum spent
'max_lvrg'   : 0,
'max_shrt'   : 0,
'risk_hi'    : 0,
'days'       : 0.0,
'date_prv'   : '',
'date_end'   : date_end,
'cash_low'   : cash_low,
'cash'       : cash_low,
'start'      : cash_low,
'pstart'     : c.portfolio.portfolio_value, # Used if restart
'begin'      : time.time(),                 # For run time
'log_summary': 126,                         # Summary every x days
'run_str'    : '{} to {}  ${} {} US/Eastern'.format(date_strt, date_end, int(cash_low), datetime.now(timezone('US/Eastern')).strftime("%Y-%m-%d %H:%M")) } log.info(c.pvr['run_str']) def _pvr(c): c.cagr = ((c.portfolio.portfolio_value / c.pvr['start']) ** (1 / (c.pvr['days'] / 252.))) - 1 ptype = 'PvR' if record_pvr else 'PvRp' log.info('{} {} %/day cagr {} Portfolio value {} PnL {}'.format(ptype, '%.4f' % (c.pvr['pvr'] / c.pvr['days']), '%.1f' % c.cagr, '%.0f' % c.portfolio.portfolio_value, '%.0f' % (c.portfolio.portfolio_value - c.pvr['start']))) log.info(' Profited {} on {} activated/transacted for PvR of {}%'.format('%.0f' % (c.portfolio.portfolio_value - c.pvr['start']), '%.0f' % c.pvr['risk_hi'], '%.1f' % c.pvr['pvr'])) log.info(' QRet {} PvR {} CshLw {} MxLv {} RskHi {} MxShrt {}'.format('%.2f' % q_rtrn, '%.2f' % c.pvr['pvr'], '%.0f' % c.pvr['cash_low'], '%.2f' % c.pvr['max_lvrg'], '%.0f' % c.pvr['risk_hi'], '%.0f' % c.pvr['max_shrt'])) def _minut(): dt = get_datetime().astimezone(timezone('US/Eastern')) return str((dt.hour * 60) + dt.minute - 570).rjust(3) # (-570 = 9:31a) date = get_datetime().date() if c.pvr['date_prv'] != date: c.pvr['date_prv'] = date c.pvr['days'] += 1.0 do_summary = 0 if c.pvr['log_summary'] and c.pvr['days'] % c.pvr['log_summary'] == 0 and _minut() == '100': do_summary = 1 # Log summary every x days if do_summary or date == c.pvr['date_end']: c.pvr['cash'] = c.portfolio.cash elif c.pvr['cash'] == c.portfolio.cash and not logging: return # for speed longs = sum([p.amount * p.last_sale_price for s, p in c.portfolio.positions.items() if p.amount > 0]) shorts = sum([p.amount * p.last_sale_price for s, p in c.portfolio.positions.items() if p.amount < 0]) q_rtrn = 100 * (c.portfolio.portfolio_value - c.pvr['start']) / c.pvr['start'] cash = c.portfolio.cash new_risk_hi = 0 new_max_lv = 0 new_max_shrt = 0 new_cash_low = 0 # To trigger logging in cash_low case overshorts = 0 # Shorts value beyond longs plus cash cash_dip = int(max(0, c.pvr['pstart'] - cash)) risk = int(max(cash_dip, -shorts)) if record_pvrp and cash < 0: # Let negative cash ding less when portfolio is up. cash_dip = int(max(0, c.pvr['start'] - cash * c.pvr['start'] / c.portfolio.portfolio_value)) # Imagine: Start with 10, grows to 1000, goes negative to -10, should not be 200% risk. if int(cash) < c.pvr['cash_low']: # New cash low new_cash_low = 1 c.pvr['cash_low'] = int(cash) # Lowest cash level hit if record_cash_low: record(CashLow = c.pvr['cash_low']) if c.account.leverage > c.pvr['max_lvrg']: new_max_lv = 1 c.pvr['max_lvrg'] = c.account.leverage # Maximum intraday leverage if record_max_lvrg: record(MaxLv = c.pvr['max_lvrg']) if shorts < c.pvr['max_shrt']: new_max_shrt = 1 c.pvr['max_shrt'] = shorts # Maximum shorts value if record_max_shrt: record(MxShrt = c.pvr['max_shrt']) if risk > c.pvr['risk_hi']: new_risk_hi = 1 c.pvr['risk_hi'] = risk # Highest risk overall if record_risk_hi: record(RiskHi = c.pvr['risk_hi']) # Profit_vs_Risk returns based on max amount actually spent (risk high) if c.pvr['risk_hi'] != 0: # Avoid zero-divide c.pvr['pvr'] = 100 * (c.portfolio.portfolio_value - c.pvr['start']) / c.pvr['risk_hi'] ptype = 'PvRp' if record_pvrp else 'PvR' if record_pvr or record_pvrp: record(**{ptype: c.pvr['pvr']}) if shorts > longs + cash: overshorts = shorts # Shorts when too high if record_shorting: record(Shorts = shorts) # Shorts value as a positve if record_overshrt: record(OvrShrt = overshorts) # Shorts beyond payable if record_leverage: record(Lvrg = c.account.leverage) # Leverage if record_cash: record(Cash = cash) # Cash if record_risk: record(Risk = risk) # Amount in play, maximum of shorts or cash used if record_q_return: record(QRet = q_rtrn) # Quantopian returns to compare to pvr returns curve if record_pnl: record(PnL = c.portfolio.portfolio_value - c.pvr['start']) # Profit|Loss if logging and (new_risk_hi or new_cash_low or new_max_lv or new_max_shrt): csh = ' Cash ' + '%.0f' % cash risk = ' Risk ' + '%.0f' % risk qret = ' QRet ' + '%.1f' % q_rtrn shrt = ' Shrt ' + '%.0f' % shorts ovrshrt = ' oShrt ' + '%.0f' % overshorts lv = ' Lv ' + '%.1f' % c.account.leverage pvr = ' PvR ' + '%.1f' % c.pvr['pvr'] rsk_hi = ' RskHi ' + '%.0f' % c.pvr['risk_hi'] csh_lw = ' CshLw ' + '%.0f' % c.pvr['cash_low'] mxlv = ' MxLv ' + '%.2f' % c.pvr['max_lvrg'] mxshrt = ' MxShrt ' + '%.0f' % c.pvr['max_shrt'] pnl = ' PnL ' + '%.0f' % (c.portfolio.portfolio_value - c.pvr['start']) log.info('{}{}{}{}{}{}{}{}{}{}{}{}{}'.format(_minut(), lv, mxlv, qret, pvr, pnl, csh, csh_lw, shrt, mxshrt, ovrshrt, risk, rsk_hi)) if do_summary: _pvr(c) if get_datetime() == get_environment('end'): # Summary at end of run _pvr(c) elapsed = (time.time() - c.pvr['begin']) / 60 # minutes log.info( '{}\nRuntime {} hr {} min'.format(c.pvr['run_str'], int(elapsed / 60), '%.1f' % (elapsed % 60))) #def handle_data(context, data): # pvr(context, data)  There was a runtime error. @Matteo, you are right. The change of benchmark had a very small impact on the outcome. It should not be considered as the reason for the performance difference. What I now suspect is that somehow the Q500US dataset has changed since the March 14 test. Note, that I do not know what has changed in it. But, it changed the trading dynamics. I redid the March 14 test, under the same conditions as then, and got about the same results you did which could only happen if the dataset was different. Now, this raises questions as to the validity of all tests made under the old Q500US. Any strategy developed with the old Q500US will have to be redone to see if the alpha is still there. This might have other consequences. Thanks Guy. If it is indeed the dataset, then it really has an impressive effect on the results I am wondering if the algo could access the historic Q500US dataset and use the most recent one at each rebalancing - this would probably be a more realistic way of backtesting it What are your thoughts? Hi Rob, and other Quantopians, I have some questions on 2 different sub-themes here that I hope may be usefully thought-provoking. 1) FUNDAMENTALS: In your initial post, Rob, you mention the idea of "... Trading on a universe of stocks where there is less uncertainty about fair value". Is there really such a thing? [I mean "less uncertainty" rather than "fair value"]. How exactly should one determine the genuine degree of uncertainty? If we are considering "fair value", then the UBS concept of "low dispersion of analyst's earnings estimates" surely seems a very flawed methodology indeed for a number of reasons. Firstly, earnings are only one component of fair value, and it can be argued that they are really not the most important one anyway (despite the fixation of the financial press on earnings). Secondly, reported earnings are very much a function of the culture of the company reporting them. There is sufficient latitude in generally accepted accounting practices to allow companies to present widely varying earnings numbers depending on their preference at the time, not to mention of course downright fraud & deliberate complete misrepresentations (think Enron). Thirdly, analysts not only tend to use only a very limited subset of the spectrum of possible valuation methods, but also tend to follow each other, with no-one wanting to be the odd-person-out. I well remember back at the height of the tech bubble in 2000, when CSCO was trading above$60. I kept repeating (and doubting) my own set of more than ten different valuation calculations and kept coming up with a median value of $20. I discussed with a colleague and he told me I must obviously be completely wrong because, as he put it: "Cisco is definitely a sixty dollar stock, and all the analysts say so". Not long afterwards the bubble burst, all the tech stocks started down, and CSCO went down to around$20 +/- $1 or 2, from which it took a very long time to recover. So all the analysts were clustering together and timidly presenting results that were all not too far away from the current (at that time) market price. I contend that the same thing still happens now, almost twenty years later. My suggestion to anyone interested is to do your own fundamental analysis using all the financial statement data available, and as many different valuation methods as possible. We have access to quite a bit of financial data at Quantopian already, at least on a 1-year reporting basis, and I hope more multi-year data might become available in future [hint, hint]. Use as many different valuation methods as possible, and don't put too much emphasis on reported earnings relative to other methods. If anyone hasn't done this and then tries it, they will almost certainly be quite amazed at the huge spread of results obtained for each company, a range of values this is almost always many times larger than the sheep-like analysts' estimates of "fair value". 2) TECHNICALS: The simplistic concept of some instruments being trending / trend-following (TF) in nature while other are mean-reverting (MR) in nature...... is simply.... wrong. This myth is perpetuated by people who just follow what everyone else says, presumably because they are too lazy to do the work to check it out for themselves. The reality is that financial instruments, whether shares or futures or FX, exhibit both TF and MR characteristics in varying degrees and at various times. Some instruments generally tend to be a bit more TF while others tend to be a bit more MR in character, but those general differences between instruments are dwarfed by the huge swings of behavior between periods of TF character and periods of MR character even for a single instrument viewed in isolation. Let's ask why do some trades succeed and other trades fail? In most instances it is likely to be because the trader (or his/her algo) misjudged the trending vs mean-reverting character of the instrument at the time of the trade. Personally I believe that THE single most important question to ask in trading is always: "Is this instrument currently trending, or mean reverting, or possibly in some indeterminate state in between?" The consequences of the answer to that question are obvious. If the instrument is, at the time, predominantly in trending mode, then a TF strategy will succeed and a MR strategy will fail, so use a TF strategy. Conversely, if the instrument is in predominantly mean-reverting mode, then use a MR strategy. (Sort of obvious isn't it?) Finally, if it is not clear which of the two modes the instrument is currently in, then better to just stand aside and not expose the account to the risk of what amounts to a dangerous gamble. What always surprises me is how little effort most people put in to determining the character of the specific instrument as either trending or mean-reverting at the time of the trade. I'm not sure why they don't because, at least to me, this is the single most important question to ask , and is therefore worth putting a lot of effort into. These two questions, one "fundamental" and the other "technical", are where I have directed most of my own trading-related thinking time & effort over the last 30 years or so, and I have enjoyed and continue to enjoy almost every minute of it! . Best regards & best wishes, Tony. @Matteo, I have not seen any notes as to if the old Q500US has been saved in some fashion, or even made available. However, it does, as you said, raise the question of how critical the stock selection process is since only what could be considered as minor changes might have been made to that dataset. And it was enough to drastically change the outcome. It should be up to Quantopian to say how much change and what type of change has been made and what to expect from those changes. For now, all it says is that there is, or there was, survivorship and/or selection biases built in. @Tony, I agree. All the points you raised are valid considerations. What do you see as alternative to all that flawed data from whatever source? I remember Enron, I was in during its downfall and I can say that all the available market data was far from accurate or even reliable. The first market analyst recorded on Yahoo Finance lowering Enron's rating to sell, out of the 20+ having expressed their “opinions” on the matter, did it at 69 cents only days before Enron's bankruptcy. All the way down, Enron's rating retained a buy the dip or hold your position recommendation. People forget these things, but some do remember. When looking at peoples' “opinions”, they should not easily replace your own. Sometimes, mean-reversal expectations does not work! There is a cost to learn about everything. Guy, Re Enron and your comment that "... all the available market data was far from accurate or even reliable". Yes, absolutely. For the younger people who may not have been trading at that time, and who may not have read about this (in-)famous case, here's a quick summary. Enron was a US energy company and back in year 2000 it was not only one of the darlings of the stockmarket (bubble) but it was also widely believed to be one of the best companies in America to work for. The fundamental data coming from the company looked amazingly good and the stock was going up and up. But then the price started to go down. An obvious pullback buy opportunity in a great stock in a bull market, or so it appeared whatever way you looked at it, either technical or fundamental. (Guy, you weren't the only one). Then something strange started happening. The stock price just kept on going down & down, while the fundamentals kept looking better & better ... especially the earnings & earnings growth. What only became apparent later was that the insiders in the know were bailing out as fast as they could, and the management, accountants and auditors were all lying and completely "cooking the books". So much for the mighty Enron. By the end of year 2001 it was bankrupt. At least some of the management went to prison, and the once-famous accounting firm of Arthur Anderson had completely destroyed its reputation. The biggest blow was to former employees. Not only was their supposedly-wonderful employer gone, but so was their pension fund along with it. [For other relevant info, if interested, you could look up "Sarbanes-Oxley Act"]. That's all long-past history, but there are some good lessons here. What I learned personally was the following: 1) Sometimes you cannot trust reported financial data at all, especially the "earnings" numbers that are so beloved of our friends "the analysts". 2) Unless you are Warren Buffett, it is not viable to trade based on financial data alone. In fact actually Buffett doesn't. He puts a lot of store by the quality of management. This is not so easy to quantify, but it can be done. Especially watch out for potential fraud by dishonest management. There is a whole field of forensic accounting that does just that. 3) The "technicals" tell a story and it pays to listen to it as carefully as you can. As they say: "Be careful of catching a falling knife. Better to wait for it to hit the ground first and then bounce" (if it does). To the lovers of mean reversion strategies: caveat emptor! 4) Although fundamentals may sometimes be simply wrong (possibly quite deliberately) there are also lots of honest companies around with good ethical and accounting practices, so don't just dismiss fundamentals because sometimes they are fraudulent. Mostly they will be OK. 5) Another argument that many people have against using fundamental data is that it is a) inherently "stale" by the time it becomes available, and b) has totally the wrong granularity for trading i.e. it is usually Quarterly or Annual, as opposed to whatever timeframe of price data you use for trading (EOD bars, 5 minute bars, tick data or whatever). Personally I reject the argument about fundamental data not being useful just because it is inherently stale. In the "real business world", as opposed to the markets, things change relatively slowly, especially in the case of large industrial & resources companies. It takes time (sometimes many years) to develop a mine or an oilfield or build factories, and also companies don't usually change their management philosophy or financial structure all that often. So, if the financial / fundamental data are a few months old, it should not have changed too much and "a little bit stale" definitely does not mean useless. 6) What about when there are either no fundamental data available, or the fundamentals & technicals tell opposite stories? Then be careful. Maybe its OK, but just be careful. The stock performance (up) may just be all because of hot air with no long-term underlying driver at all. To finish up with your question, Guy: "What do you see as alternative to all that flawed data from whatever source?", here are my own opinions: a) With the possible exception of HFT (which is something that I know nothing about, so I won't say any more), just accept that ALL data are somewhat flawed, and in some cases maybe even outright lies. But don't ignore something just because it might be faulty. Even that is actually useful info. b) Within the context of however you choose to trade, figure out how to mitigate the adverse impact of bad / wrong data, because there WILL be some. c) Even better, with regard to fundamentals, learn how to actually LOOK for data that seems "a bit fishy". If something is suspicious then its a warning flag, because maybe there is "more than just one cockroach" in that company. For me, if the warning flag is up, then it says: Avoid or possibly Short, but be VERY cautious about going Long this one! All the best, Tony. Enron? How about all of the hidden risk across all of finance leading up to the Great Recession? Was there anything on the books at Lehman Brothers that screamed "Uh-oog-gah! Uh-oog-gah! Uh-oog-gah! Abort! Abort! Abort!"? Hi Grant, Yep, definitely not kidding! The following is extracted from an article entitled "Could Lehman be Ernst & Young's Enron?", Thomson Reuters, March 13, 2010 http://www.reuters.com/article/us-lehman-ernstyoung-analysis/could-lehman-be-ernst-youngs-enron-idUSTRE62C05220100313 At issue is a repurchase and sale program called Repo 105, which Lehman used without telling investors or regulators, and the examiner concluded was used for the sole purpose of manipulating Lehman’s books. In the examiner’s report Lehman executives described the Repo 105 as everything from “window dressing” and an “accounting gimmick” to a “drug.” Ernst & Young said in a statement: “Our last audit of the company was for the fiscal year ending November 30, 2007. Our opinion indicated that Lehman’s financial statements for that year were fairly presented in accordance with Generally Accepted Accounting Principles (GAAP), and we remain of that view.” Yeah, right, well so much for GAAP! Of course some people argue that, because of cases like this, all financial reports are inherently worthless lies, but that isn't true. Some are, some aren't. Rather than saying fundamentals are not even worth bothering about, I think a better approach is to say that good financials / fundamentals are a NECESSARY BUT NOT SUFFICIENT condition for a potentially good (Long) investment / trade. Actually even what appear to be good but are fraudulent financial reports by dishonest companies can be caught out, see for example "Benford's Law" (Not that I'm suggesting anyone go as far as that in looking at "alternative data"! .... oh but then again ....... maybe ...? ;-)) Best wishes, Tony. @Tony, yes, agree on all counts. The only recourse is to play an overly diversified statistical expectancy game where any black swan event (Enron, Lehman and the likes) could only represent a minor fraction of one's portfolio at any one time. We could make it difficult for any such stock to even be part of our stock selection process long before they could have an impact. A simple criterion like trading big cap stocks with a long history of returns staying above their benchmark could probably be sufficient to eliminate 99.9% of bankruptcies. No matter what we do trading, we won't be able to escape what it basically is to our trading accounts. We buy some shares (q) in some company (p) for whatever reason at an agreed upon price (p(in) to hold or resell later at a different price (hopefully higher, sometimes lower, and occasionally at the same price p(out)). The account does not ask questions, nor seeks any reasons. It just tallies the numbers and keeps the score. A trade can be done in milliseconds or years. You could write all the trades done as a very simple equation: F(t) = F(0) + Σ(q ∙Δp) = F(0) + n∙APPT, where APPT is the average net profit per trade, and n the number of trades. Whatever amount of profit you want, it will have to respect: F(t) - F(0) = n∙APPT, where the number of trades might get a major role to play. Shortening the trade interval, which is a basic consequence to trading, has a tendency to reduce the expected average net profit per trade (APPT). Therefore, it will require a larger n the more you reduce the trading interval just to maintain the same level of profit. In trading stocks, one should not necessarily only search to trade better by increasing his/her APPT, but also, within their resource constraints, increase n as much as they can. Oftentimes, this could be done by simply doing more of what one's strategy was already doing. This, for instance, is the heart of HFT. It becomes what reasons are you going to use to generate trades? And, how often, on average, can you do it profitably. The above formula does not care how you did it, only that it was done. Hi Guy. I will pick up on just 2 specific points in your post, and now share those with you & everyone else, as follows..... Although the "diversified statistical expectancy game" to mitigate the risk of black swan events is indeed what most professionals play, actually its not the only way. Three of my favorite investment heroes are George Soros, Warren Buffett and Charley Munger. Certainly Soros plays it the way you describe, and I always enjoy looking at his very diverse portfolio holdings as disclosed in form 13F. However Buffett & Munger at Berkshire Hathaway do it differently. Rather than the conventional manager's "shotgun" approach of taking small bites of lots of securities, Buffett & Munger take the "precision sniper rifle" approach with relatively few large bites of VERY carefully selected securities. The key is in the care that goes into the selection and Buffett + Munger have that down to a fine art in a way that I don't think is well understood, despite all the different "Buffett" books on sale. All the stuff about how Buffett built on Benjamin Graham's ideas and included the importance of growth, and how Buffett is so careful about considering management, and how Buffett considers ROE as a key metric, that's all true. But Buffett & Munger do something else that most of the popular books don't describe, and that is to use the conventional numbers in "unconventional" ways. You have to dig a bit to find this, but the ROE that Buffett uses in not the conventional ROE as we find on the Morningstar database. He does some neat "pre-processing" and then after that uses the numbers in a way that is different to conventional valuation. The idea of careful pre-processing and then using conventional tools in unconventional ways is really worth thinking about. I think this is Buffett & Munger's "secret sauce" ... and it works. As to the more philosophical aspects of the question of just how diverse a portfolio "should" be, I won't pursue that here, other than to state the obvious, that diversification also diversifies away the benefit of good stocks and, in reality doesn't diversify away the risk in times of stress when everyone wants to bail out at once and all those nicely (under normal conditions) "uncorrelated"securities suddenly get horribly correlated. So diversification is great in theory, but I think the case of "diversified portfolio Soros" on one hand and "Few but carefully selected Buffett & Munger" on the other hand shows that the "... overly diversified statistical expectancy game" is not necessarily the only way to play. The other point that is interesting in Guy's post is this "fundamental equation of trading", which I notice you (Guy) have written in the same form in a number of posts. What is says, in descriptive rather than mathematical terms, is that account growth depends on the expected average net profit per trade, and the number of trades. True. Actually, if one is continually returning all profits to the account to enjoy the benefits of compound growth, then Guy's equation can be written in a different way that I am more familiar with, and in which the "n" becomes an exponent rather than a multiplier, but the main point I would like to make is with regard to the meaning [pun intended] of the term "average net profit per trade". Generally most traders try to design their trading to maximize the ARITHMETIC MEAN (average) per-trade profit. It can be shown that actually it is preferable to maximize the GEOMETRIC MEAN per-trade profit and, over time, the difference becomes profound. Putting this another way, and again without needing any maths, the typical arithmetic-mean-maximizing amateur trader likes to see some nice BIG wins. (And that applies to me too, as a "little trader" of my own small account, mostly doing trend following). The geomean-maximizing trader on the other hand is not so concerned about big wins. He or she wants to see SIMILAR wins, preferably as similar as possible, on all trades, even if they are not so big. And so, when I put on my other hat of "systems design" rather than "little trader", I need to think differently. Personally I really enjoy this switching between the two different perspectives. Anyone interested in following up further on this topic of arithmetic mean vs geometric mean trade expectancy and its profound impact both on trade design and on account equity growth vs drawdown, I thoroughly recommend all of the books by Ralph Vince that you can find. All the best, Tony. @Tony, there is a tremendous difference between Mr. Buffett's long-term investment strategy and trading almost every day ready to rebalance one's entire portfolio. Nonetheless, the same formula applies. For Mr. Buffett, n would be relatively small (probably less than 1,000 over his 50-year career) and with a large APPT. Note that the major part of his personal assets are in one stock. A large APPT requires either a lot of time and/or very large bets. In fact, you will find that the 5 largest positions in Berkshire Hathaway account for over 60% of its holdings. This implies that for n=5, those positions have an APPT of$55B. There was some trading to get there. You either play big, real big, or you wait a long time, what might be a lifetime.

Whereas, an HFT trading firm like Virtu, will have a large n (could reach a million trades per day) with a correspondingly small APPT, and having to make small bets all the time. Virtu has a winning rate of 51-52% on its trades. It could be viewed as a few pennies per share per trade for its APPT. It is sufficient to make it profitable.

As we can imagine, those are totally different worlds. And yet, both can make money participating in the market. Just as might a lot of other trading methods in between.

When one trades and/or invests, the holding interval, as was said, could be milliseconds or years. However, I have not seen a minute trader being referred to as an investor. Nor have I seen any comment on Mr. Buffett's firm referenced as an HFT.

But, as said, the same formula still apply. No matter what the trading methodology, the bottom line can be expressed as:

E[F(t)] = F(0)∙(1+ E[r_m] + α)^t = F(0) + Σ(H.∙ΔP) = F(0) + n∙APPT

The first part saying that without some added skills (α), the expectancy is r_m, the market average. And if alpha is zero, then it does not matter much what is in the rest of the equality, it will simply compute to say the same thing as r_m.

The second part says that what really matters is the trading strategy itself. The how you will slice and dice price series for a profit. The expectancy of all that slicing and dicing is E[r_m] if there is no added skills. And, less than r_m should the alpha be negative due to the added frictional costs, or lack of skills (α < 0). The payoff matrix may contain millions of trades, but nonetheless, the last part is able to resume all of it to two numbers!

You want to improve on your trading strategy, then take care of those two numbers.

Find ways to increase APPT and at the same time increase n. This will echo back to part one and generate your added alpha, on the condition evidently that APPT > 0, large enough to cover frictional costs, and large enough to exceed E[r_m]. Another way of saying you will need to show some added skills in order to outperform the market.

My point is that we can all win. That it be from Virtu to Berkshire, and anything in between. It is the trading methodology that you will apply that will make it happen. It is the execution of all those trades that in the end will print your account statement.

Yes, Guy, i think we are both saying very much the same thing, albeit in slightly different ways.
To refer to what Buffett does as "trading" or HFT as "investing" would be a semantic nonsense and, for any person or orgnization, their timeframe / style can be anywhere in between, and in the end the result is determined by the balance between individual trade gains and the number of trades, just as you say.
In addition to those general principles which we know, I was also trying to suggest some quite specific items that I think are also worth looking at.