Back to Community
Backtesting and Discussion of "Driven to Distraction" Paper

In the paper, "Driven to Distraction: Extraneous Events and Underreaction to Earnings News" (Hirshleifer, Lim, Teoh, 2009, Journal of Finance 64, 2289-2325), the authors compare Post Earnings Announcement Drift for stocks that announce earnings during peak earnings season and stocks that announce earnings when there are fewer other competing announcements. Post Earnings Announcement Drift (PEAD) refers to the anomaly that when a company beats earnings, not only does the stock rise on the earnings announcement (which isn’t surprising but is difficult to predict), but also drifts higher for weeks after the announcement. Similarly, when a company misses earnings, the stock price drifts lower for weeks after the announcement (for one of the early papers documenting this phenomenon, see Bernard and Thomas here). The authors argue that for companies that report earnings when there are fewer other companies reporting at the same time, there is a strong price reaction on the earnings announcement, but only a small PEAD. But for companies that announce during periods when there are many other simultaneous announcements, the earnings announcement effect is muted and there is a much stronger PEAD. Their argument is that investors have limited attention and get distracted by the competing earnings announcements, which would explain both the muted announcement effect and the strong post announcement effect. Of course, because it's an academic paper, they give many references to show that people get distracted, including a reference to a study showing that British drivers get distracted when seeing semi-naked models on billboards (and the paper's title, "Driven to Distraction:..." is a play on this).

Discussion and Digressions

I don't find the explanation of limited attention particularly appealing (for example, it seems implausible that limited attention to an earnings release would last for weeks), but I do believe the results could be driven by limited capital, not limited attention. For example, if you are a trader in the utilities sector at a hedge fund (and most portfolio managers specialize in a particular sector) and one day during peak earnings season, four utilities that you follow all report earnings on the same day, you may not have the capital to buy two or three of the stocks.

Limited capital may be an explanation for other anomalies too. For example, there is a cottage industry in the hedge fund world of trading stocks in front of expected large index fund flows. In the early 2000's, when S&P announced that a company was added to the S&P500 index, the stock would, on average, go up 5% on the announcement, and go up another 3% between the announcement and when it was added (usually, the announcement precedes the "effective date" by a week). See, for example, Chen, Noronha, Singal, "Index Changes and Losses to Index Fund Investors". But this anomaly of one-off adds to the S&P500 has largely gone away - there is a much smaller announcement effect nowadays and the movement on the effective day is close to zero. However, for large index rebalances like the annual Russell rebalance, the anomaly, although much smaller than it used to be, still seems to exist. One explanation is that because of limited capital by hedge funds, they can take the other side of index fund flows for a one-off S&P500 add for several billion dollars, but in aggregate don't have the capital to take the other side when index funds are trading hundreds of billions of dollars on a single day.

To digress a little further, another anomaly that doesn't seem to work anymore, but maybe can be revived with this idea of limited capital, is the reversal from extreme one-day price drops. A much older study (see Bremer and Sweeney "The Reversal of Large Stock-Price Decreases"), finds that following a stock price drop of 10% or more, stocks reversed by about 2% the next day. This anomaly has gone away, and in fact, a very simple Quantopian backtest shows that it not only has gone away but performs horribly since 2007 (this has interesting implications, incidentally, for how to trade a typical five-day mean reversion strategy). But if you look at only days where there are numerous stocks dropping more than 10%, the strategy works better (unfortunately, almost all the trades are in one time period, during the financial crisis, so it's not very robust).

Algorithm

For the "Driven to Distraction" paper, I backtested their strategy out of sample from 2007-2016. For stocks that announced earnings on days when many other companies announced, I went long PEAD by buying stocks that had large earnings beats and shorting stocks that had large earnings misses. On low earnings announcement days, I do the opposite PEAD trade: I shorted large earnings beaters and went long large earnings missers. I used the Q500US Universe (and also screening, of course, by companies that have estimates of earnings), and used the quintile of earnings beats and misses, and the quintile of high and low announcement companies. The authors define beats and misses in what I think is a very sensible way: (actual earnings-expected earnings)/Price (Zacks has a measure that is normalized by earnings instead of price: (actual earnings-expected earnings)/(expected earnings), which I don't think makes as much sense). I also put together a short notebook that explains how I got the quintile cutoffs that I used in the algo. I’ll post that shortly under the same thread.

Data Mining

I would love to start a tradition on Quantopian where people post at the end of their strategies a section that addresses data mining issues, including things they tried that may not have worked. Also, with any strategy, there are many other choices that must be made: start date, universe, beta neutral vs. dollar neutral, decile vs. quintile sorts, holding periods, equal vs. non-equal weighting, sector, industry, or other factor limits, time of day for execution, ... These are all essentially parameters that can lead to data mining, so a discussion about how these choices were made would be useful also. In academia, the goal is to get something published, which often involves data mining, but in trading, the goal is to find strategies that will make money out of sample. As a Portfolio Manager at my previous hedge funds, I would certainly ask these same questions to anyone that worked for me and backtested a trading strategy.

Along those lines, in the case of this model, the biggest area of data mining, and deviation from the original paper, was in the holding period for PEAD. The paper argues that a 60 day PEAD holding period was optimal, and I tried one other holding period, 20 days, and got significantly better results (the Sharpe Ratio was about a third lower when I used a 60 day PEAD holding period). I tried other cutoffs, including decile cutoffs – the results were consistently positive but can vary considerably, even for small changes. Since the algo holds a relatively small number of positions, small changes in the threshold for earnings beats or the threshold for high announcement days can change the composition of the portfolio considerably. The strategy I tested was dollar neutral rather than beta neutral, and the beta turned out to be negative, so making it beta neutral (in a rising market) might slightly improve the results. One thing that disappointed me with the results is that, although you make money on both the long PEAD side on high announcement days and the short PEAD side on low announcement days, the results are much more driven by the short PEAD side.

Clone Algorithm
189
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5840e0fed160855e6535cb06
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

12 responses

Hi Rob,

Regarding your comments on data mining, in my opinion, Quantopian makes it very challenging to do the kind of response surface and sensitivity analysis you presumably did manually by adjusting parameters and re-running backtests. The backtesting engine supports running a large number of backtests in parallel, but the support for varying parameters programmatically and then post-processing the results needs to be improved. For example, say I want to carry out a study of the effect of a single parameter. I would run N backtests, noting the backtest_id for each, and the corresponding parameter value, which I would have set manually. I would then enter these data into the research platform to retrieve the backtest results for analysis. I would then need to sort out how to analyze the N backtests (presumably I could leverage the existing pyfolio tool, applying it iteratively, and then collating the results).

One solution would be to provide support from within the research platform to be able to run backtests in parallel, with programmatic control over backtest parameters. I think making it a lot easier to "slice and dice" backtest results systematically over a space of M parameters would go a long way toward addressing the data mining issue you raise. If the platform readily supported a workflow that included running N backtests over the M parameter space and some plug-and-play analysis tools, then the "point solution" approach could be avoided. There's an operational cost in terms of supporting the running the N backtests in parallel (users might tend to run more backtests), but I figure it must be close to the cost of electricity to power the servers, at scale (i.e. we are using commodity cloud computing).

Regarding your comments about the hedge fund industry capital limitations, from http://www.hedgeweek.com/2016/10/20/244960/hedge-fund-industry-capital-rose-third-quarter , we have:

Hedge fund assets rose to USD2.972 trillion, an increase of USD73.5 billion from the prior quarter. The 3Q16 hedge fund asset level eclipses the previous record of USD2.969 trillion set in 2Q15.

It would seem that if a true zero-risk arbitrage opportunity is out there, they will come up with the capital to profit from it, one way or the other (I'd read that prior to the mortgage crisis debacle, home mortgages were effectively leveraged up something like 50X, so Wall Street knows how to print money when they need it). It is kinda hard to believe that they wouldn't come up with the money, in the limit of zero risk.

However, for large index rebalances like the annual Russell rebalance, the anomaly, although much smaller than it used to be, still seems to exist.

Perhaps the reward is not high enough to justify the risk. In other words, the cost of coming up with the capital doesn't justify the expected reward, with risk taken into account.

Grant, instead of trying to optimize parameters for a backtest, you should be evaluating your strategy in pieces. For example, you could easily vary parameters of a pipeline programmatically in research, and then run the results through alphalens to evaluate the predictive ability of your alpha signal.

You can't run a parallel search, but it's at least automated, and you can simply copy and paste your pipeline from research to an algorithm. I've attached a simple example notebook of what that might look like.

The notebook performs a parameter search on an alpha factor in the last cell, which should solve the problem that you're running into. I didn't do the optimization step, I'll leave that up to you.

P.S. I used the alphalens tearsheet definition to figure out which functions to call.

Loading notebook preview...
Notebook previews are currently unavailable.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Here is a short notebook that accompanies the "Driven to Distraction" backtesting post.

Loading notebook preview...
Notebook previews are currently unavailable.

Bit of a rudimentary question if you don't mind - Can we think of long PEAD as a momentum bet and shorting PEAD as a mean reversion bet? Then your algo is choosing momentum on crowded earnings dates and mean reversion on sparse earnings dates.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thanks for sharing the strategy Rob. Here is a quick effort to enhance:

Clone Algorithm
58
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 58426e69d4cb3b64f7ad940f
There was a runtime error.

@ Jamie -

Yes, the fundamental way to avoid data mining is to know that if I do X I'll get Y (at least on a statistical basis). So, I'd throw the question back to you. It sounds like Rob used backtesting to explore the parameter space. You seem to be suggesting that he could have used alphalens. Is this the case? Do you have an example for his particular strategy? Or does one need to get all the way to the backtest stage?

My point was that if it is difficult/impossible to map out the complete response surface of a strategy, then there will be a tendency to overfit/mine data. One solution is to make it really easy to get the big picture. Define M parameters, run N backtests, and presto, one can get a sense if an overfit "point solution" has been selected, or if there is actually no dope-smoking going on.

Anyway, no need to hijack the discussion here, to point out the obvious. There's a cost-benefit calculus on your end, but unless those data are brought into the discussion, it is a one-sided dialog.

Response surface and backtest 1001 different parameters. Yes, I worked with a proprietary backtesting engine for years which made this very easy to do and graphically very easy to see.

Even if you avoid an obvious peak of profitability, MAR or whatever your chosen risk metric I fear it made little difference to forecasting the future profitability if the strategy. You whole "response surface" can change unrecognizably after a few years.

No solution there I fear although it is certainly a helpful process in forcing you to realise what absurd difference can result from minute parameter change.

Hi Rob,

A couple more quick comments:

  • I suggest as a baseline, normalizing to a leverage of 1.0. My understanding is that when evaluating algos for potential investment, Quantopian does this. Having a leverage of 2 (or whatever) is arbitary (Quantopian is talking about using 6X leverage for the hedge fund anyway).
  • I just noticed your use of a "hedging instrument" (SPY). I've gathered that this is discouraged for algos that would be candidates for the hedge fund. What happens if you don't hedge? Or hedge in some other fashion, without using a hedging instrument?
  • What if you have a mix of longs and shorts? Your code doesn't appear to handle this case:
If only long or short postions, hedge other side with SPY  
    if(len(shorts)==0 and len(longs)>0):  
        order_target_percent(context.SPY,-1)  
    elif(len(longs)==0 and len(shorts)>0):  
        order_target_percent(context.SPY,1)  
    else:  
        order_target_percent(context.SPY,0)  
  • It would be interesting to put this strategy into the context of the type of algos Quantopian might be wanting to add to their portfolio. There must be something wrong with it, or it wouldn't have been posted (in a private communication, Dan Dunn made it clear to me that Quantopian would never post anything that would work). What's wrong with the strategy, vis-a-vis the Q hedge fund?

Grant,

Thanks for your thoughtful comments.

  • The point you make (in your first reply) is valid that when hedge funds see opportunities like the annual Russell reconstitution, they do have the ability to move capital. One possible counterargument, though, is that the index rebalancing business requires some specialized knowledge (for example, merely predicting the float adjusted shares in the index is quite complex) so a fund that specializes in stat arb may be reluctant to redeploy large amounts of capital in the annual Russell rebalance without having the expertise. And because the amount of capital required for these types of trades shoots up at discrete periods of time, the funds may run into leverage constraints. Also, I didn't intend to imply that this trade is risk free.

  • I should probably have used a target leverage of 1.0 instead of 2.0 so it's consistent with Quantopian's preferred leverage ratio. It's a very simple change to make in this algo, which equally weights all longs and all shorts. For example, change order_target_percent(security,1/len(longs)) to order_target_percent(security,0.5/len(longs)) and do the same for shorts and the possible SPY hedge.

  • I should explain the section of the code that you refer to that checks whether the algorithm holds only longs or only shorts, and if so, hedges with SPY. It is theoretically possible that the algorithm finds PEAD stocks that cross the buy threshold but finds no stocks that cross the sell threshold (or the other way around). In this case, to be truly dollar neutral, I would sell an equal dollar amount of SPY against the long-only positions or buy SPY against short-only positions. With the parameters I use, there are only a handful of days where this actually happens. You can see those days by inserting a log.info() message inside that portion of the code. If I had left this section of the code out, it would have had very little effect on the performance of the algorithm. However, if you run the algo for other sets of parameters, like more extreme quantile cutoffs or shorter PEAD holding periods, there will be more days where the algorithm wants to hold only long positions or only short positions. So I wanted to generalize the code to handle this possibility.

Fawce,

That is correct that the long PEAD trades (on crowded earnings dates) are momentum trades and the short PEAD trades (on sparse earnings dates) are mean reversion trades. The one caveat I would make is that there are cases where a stock declines despite beating earnings expectations (because of weak forward guidance, a revenue miss, analyst expectations not representing the market consensus, and a variety of other possible reasons) as well as cases where a stock advances despite missing earnings expectations. An interesting variation on this algorithm would be to sort stocks into quantiles based on the stock price reaction to earnings rather than on the degree to which it beats Zacks's consensus estimates. Another variation is to use Estimize's consensus estimates rather than Zacks's.

I just want to point out (as I suggested should be pointed out on every backtest that does this), the results are ZERO commisson and slippage.

https://www.quantopian.com/posts/request-add-disclaimer-to-backtest-graphs-without-commissions

    #: Set commissions and slippage to 0 to determine pure alpha  
    set_commission(commission.PerShare(cost=0, min_trade_cost=0))  
    set_slippage(slippage.FixedSlippage(spread=0))  

Here is the original algo with slippage/commission on.

Clone Algorithm
22
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 58454086aae3215e6e4fc1a4
There was a runtime error.

Hi Rob:

Thanks for sharing insight on incorporating price action rather than earning beat/miss alone. As you said, there might be other nuances on the earning release.

One question, you argument about limited capital. Does that apply to both long and short PEAD? If you run a long/short hedge fund, wouldn't you be able to apply a higher leverage than a long only fund, i.e. more access to capital?

A side note regarding achieving dollar/beta neutral by using SPY. An alternative is using sector ETF which can hedge out sector performance issues.