Simulation of (non-marketable) limit orders

Hi -

First, thanks to everyone working on Quantopian and Zipline, I think they have great potential and cannot believe how far they have come. Despite the inclusion of advanced features like pipeline and fundamentals, I still have some concerns about the very basic underlying trading simulation.

I think the attached backtest demonstrates an issue with inaccurate simulation of limit orders.

In the backtest there are only two days with trades:
- on one day we buy by putting in a (marketable) limit order, with a limit far above the market. As this is marketable when we insert it, it should fill just as a market order would, and it does.
- on every subsequent day, at start of day we insert a limit order to sell at a fixed price of 111. When the market eventually reaches this level, our order should fill at 111 (since we have inserted a passive limit order to sell at this level). However, all the fills come in level better than our limit (as in the first case). However, this case is different! If the market "goes through" the level of a limit order which is already in the market, we still only get the limit price.

I think this is important to address: it improves the reported backtest results of any strategy using passive orders, which makes strategies look more viable for real trading than may be the case.

To be clear: this has nothing at all to do with commissions and slippage. The above issue is purely with the price of market fills, before any commission or slippage is taken into account.

I hope the example and explanation is clear. The key point is that non-marketable limit orders "resting in the market" should get filled at the limit price, or not at all. The current simulation seems to treat them as if they were re-inserted at each bar, gaining the benefit of favourable price moves and biasing the trading results upwards. This gives incorrect simulation results, compared to real trading with the same orders.

11
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 57a0748335749f123ba2d11d
There was a runtime error.
19 responses

This is an interesting point but the slippage model has never been a priority for Q anyway.

I can see the zipline code does exactly what you say. It doesn't check if the price "goes through" the level of a limit or not.
https://github.com/quantopian/zipline/blob/master/zipline/finance/slippage.py

class VolumeShareSlippage(SlippageModel):
[...omitted....]
def process_order(self, data, order):
volume = data.current(order.asset, "volume")

max_volume = self.volume_limit * volume

# price impact accounts for the total volume of transactions
# created against the current minute bar
remaining_volume = max_volume - self.volume_for_bar
if remaining_volume < 1:
# we can't fill any more transactions
raise LiquidityExceeded()

# the current order amount will be the min of the
# volume available in the bar or the open amount.
cur_volume = int(min(remaining_volume, abs(order.open_amount)))

if cur_volume < 1:
return None, None

# tally the current amount into our total amount ordered.
# total amount will be used to calculate price impact
total_volume = self.volume_for_bar + cur_volume

volume_share = min(total_volume / volume,
self.volume_limit)

price = data.current(order.asset, "close")

simulated_impact = volume_share ** 2 \
* math.copysign(self.price_impact, order.direction) \
* price
impacted_price = price + simulated_impact

if order.limit:
# this is tricky! if an order with a limit price has reached
# the limit price, we will try to fill the order. do not fill
# these shares if the impacted price is worse than the limit
# price. return early to avoid creating the transaction.

# buy order is worse if the impacted price is greater than
# the limit price. sell order is worse if the impacted price
# is less than the limit price
if (order.direction > 0 and impacted_price > order.limit) or \
(order.direction < 0 and impacted_price < order.limit):
return None, None

return (
impacted_price,
math.copysign(cur_volume, order.direction)
)


I would really like to get some response from the Quantopian / Zipline team on this. It is pretty fundamental stuff.

This is an interesting point but the slippage model has never been a priority for Q anyway.

Thanks Luca, I appreciate you tracking down what seems to be the relevant part of the Zipline code for this. However, as I stressed before, this is nothing to do with the slippage model (despite apparently being in Slippage.py? I guess that means "executions including potential slippage"?). With slippage set to zero always, this discrepancy still exists. It is not a subtle market-impact model type of thing, it is just the bare bones logic of how orders get filled in the market.

The issue is: if I enter an order to sell 100 shares at 111.00, and the market is currently at 110.50, and during the day the market rises up through my level to close at 111.50, how should Zipline report I got filled (assuming no commissions, adequate volume, just the very simplest case)?

If it is simulating correctly, it should report I got filled at 111.00. That is exactly what would happen in live trading. There is no slippage or impact model needed.

But it does not, it reports that I sell at a higher level.

I would like to understand: do Quantopian agree this is a bug or an oversight? If I submit a pull request to Zipline will it be merged? Is there any way the current behaviour can be justified (especially since it improves fills, rather than making them worse?)

Bill, few things :

1 - add the tag "seeking help", this might higher the chances Q will look at this post
2 - or send a private message to "Learn & Support" -> "Contact Support". They have always replied to my messages and they are always very helpful
3 - the current behaviour cannot be considered a bug, it's just too optimistic: you get a better price than you probably would have in real trading but it is consistent with limit order definition (buy/sell a set number of shares at a specified price or better). Still I agree with you that it would be better to have a more realistic model
4- I understand the limit order fill price is not related to slippage model but in zipline the logic is in Slippage.py and it makes sense from the code point of view
5 - if you like to contribute to zipline project you probably have to open an issue before submitting patches. In the issue tracker you can discuss this problem. If the project maintainers agree on changing the current behaviour you can ask to be assigned the task and eventually get you code pulled in.

My opinion is that Q is focused on its hedge fund, they are interested in modelling correctly the kind of algorithms they intend to run in their hedge fund. The little details that impact certain algorithm types (intraday, low volume stocks, hard to borrow, shorting costs, delisting, etc. etc.) are low priority for them (even though, as retail traders, those are high priority for us. But this is where Q current business model is not ideal for a retail trader).

Bill,

I suspect the problem is that the Quantopian simulator is using prices from historical trades (minutely closing prices). It doesn't model the market with a separate "engine" that attempts to simulate precisely what would have occurred.

You might be able to fix the problem with your own custom slippage model, but I'm not sure you can force trades for minutes when there were none.

Grant

Hi Luca,

Thank you for the response: I gave up on trying to give feedback to Q/Z once before because of the lack of response, so although you are also "just" a user, I thank you for at least having the conversation!

I have added the tag you suggest, and will probably contact the Q/Z teams directly if there is still no response.

I think you are right that Q is focused on the hedge fund, and what they currently think is most relevant to that (like large universe equity long/short portfolio construction!). However, I am not sure their judgement on what is important is necessarily infallible. Having bells and whistles like pipeline, when you cannot simulate a limit order getting filled is an odd situation to say the least. In any "real-life" trading, especially automated trading, you have to use limit orders only. You cannot have an algo blindly aggress the market. Although the limit orders will be at least partly marketable a lot of the time, trading with price limits is not an esoteric requirement, or one that concerns retail traders mainly. Arguably it is safer for a retail trader to send a market order then an institutional-volume algo!

If we want the backtest to correspond to reality, the current behaviour is wrong. So I just wanted to raise this issue, but maybe the Zipline project will be a better forum for it.

@Bill - I agree - this is a bug that needs to be addressed. I have created an issue on github and submitted a PR with a patch to fix this.

@Mohammad - Thanks for getting the ball rolling on the Z side: we will see what the response is.

Above, I was deliberately avoiding questions of slippage, to try to keep things as clear as possible. In your patch, it seems that the slippage model is applied as usual. In the case of a non-marketable limit order, even if there is a slippage model specified, it should not change the fill price in the case we are discussing: it will only effect market orders and marketable limit orders.

@Bill - I see what you are saying. In my mind, all limit orders should be filled at the limit price unless there is major slippage - but I guess this is a problem of dealing with minute data and not tick data.

I'm going to remove my PR as it doesn't solve this issue.

Hey everyone,

Great insight on this bug, I agree that this is a artifact from earlier versions of Zipline that has yet to be fixed.
A lot of questions have been asked/answered in this thread, so if I miss one feel free to post it again.

do Quantopian agree this is a bug or an oversight?

Yep, this is a bug in Zipline.

If I submit a pull request to Zipline will it be merged?

Once a PR is submitted to Zipline, its continuous integration systems will check if merging will break other systems.
Further, you'll get feedback from the maintainers about the nature of the PR, and it's rationale.

In general, if there is a PR with adequate unit tests that is useful to Zipline, it will be merged.

Thanks,
Lotanna Ezenwa

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

I think this is quite a major problem that skews results quite much especially with higher volatility stocks and longer term backtests.

Has anyone tried to create a custom slippage model that would address this? It should be possible to apply custom slippage model that would always fill non-marketable limits at limit price and marketable limit prices without negative slippage from limit price.

I created a custom slippage that fills non-marketable limits at limit price and leaves the marketable limit ones as before. Let me know what you think .

The patch is actually three lines of code (plus ten of comments ;) but I had to copy/paste the full VolumeShareSlippage class to test the patch.

@Mikko, I didn't add the feature "fill marketable limit prices without negative slippage from limit price" can you elaborate a bit on that?

EDIT:
To disinguish between marketable and non-marketable limit orders I used the following approximation:
if both open and close price are below/above (buy/sell) the limit price the order is markettable
if open price is above (or below for sell) limit price and close price is below (or above for sell) limit price, then the order is non-marketable
If the price is non-marketable I force the price to be limit price

I said approximation because this code doesn't cover the scenario where the crossing of the limit price happens between the close price of the previous minute and the open price of the next minute. But in my understanding the close price of any minute must be almost identical to the open price of the following minute, so I don't believe this is a real issue.

4
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 57a454e2209b011005407ee8
There was a runtime error.

@Mikko, I didn't add the feature "fill marketable limit prices without negative slippage from limit price" can you elaborate a bit on that?

Thank you for the code!

I think (I'm not sure about this but I think this has been discussed at some other thread?) that current slippage model makes it possible for prices for marketable limits to fill at prices worse than limit price because of the slippage. That should not be possible.

@Mikko, I don't know about that actually. If you find the post that talks about that it might help. I can see this bug fix though, it seems related to what you are saying but it is from long ago.

Luca,

Yes from the code an the bugfix it seems that this has been addressed. Thanks for finding the fix and the code!

For my education, perhaps someone could fill me in on this issue. It sounds like, absent of the slippage model, simulated order fills are done using actual historical prices from the OHLCV minute bar data base.

A few questions:

1. Is it correct that for data.can_trade(stock_A) True regardless of the value of data.is_stale(stock_A) an order will be accepted by the simulator? In other words, so long as stock_A has had its initial historical trade (first OHLCV minute bar) at the simulation time, then an order can be submitted.
2. Submitted orders won't be filled until historical trades are encountered, correct? For example, it is possible that an order submitted at 10:30 would not be evaluated for filling until 11:00, assuming that the next historical trade occurred then.
3. Only minutely closing prices are used by the order execution simulator? Or are OHL values also considered?
4. In the example provided above by Bill Lyons, the claim is that a limit order to sell at $111.00 should be filled at exactly$111.00, but this seems tricky. Say my buy order is filled at 10 am at $110.50, and then there are no historical trades until 10:30, at which point the price has risen to$111.50. So, the limit of $111 was crossed, but when? Should the order be filled at$111 at 10:30? Or should one assume a linear price versus time model, to estimate the fill time (and then post the transaction retroactively)? Basically, one has to simulate what might have happened had the limit order been with the broker after 10:30.
5. It seems that a slippage model still needs to be applied to limit orders, no? Generally, one can't just have the order filled completely and instantaneously at a fixed price. What if the transaction is a significant chunk of the market in the stock?

I think that executing limit orders always at close price is not that wrong, it's one of several trade-offs that can be made.
There are 3 alternative implementations:
1. Market price is close price. Market orders execute at close price.
1.1 Limit orders execute only at close price if marketable. This is the current situation
1.2 Limit orders execute only at limit price without having access to close price, but marketability is determined by [low..high]
2. Market price is open price. Market orders execute at open price.
2.1. If order is marketable at open price it executes at open price. If order is not marketable at open it executes at limit if within [low..high]. This is the only implementation of the 3, that can execute at market price (open) and limit price, because it has a whole bar of time after it has checked marketability.
Using close price for market orders shrinks the bar to a point, because market orders are immediate (first) and close price is last. First = Last, hence point.
Consulting close price and conditionally executing at limit introduces a subtle hindsight bias, it's not possible in reality.
I don't think that the current implementation is deceptively optimistic. If you believe that always executing at close price is optimistic, then you should never need a limit order.
I think that the 2.1 implementation is closest to reality (except in cases of daily opening volatility), but even if it existed, would it be accepted as default and make all the existing backtests irreproducible?
I have implemented 1.2 in https://github.com/quantopian/zipline/pull/1453 as class ConservativeCloseSlippage.
There are classes that use open price, but do not switch dynamically between market and limit price as described in 2.1.
I implemented these classes, not because I find the current implementation optimistic, but because I felt that it misses much of my trades by considering only close price instead of [low..high]. So which slippage model is best? Just try which will provide best results, I believe they are conservative enough not to provide unrealistically good results.

@Quantopian

I was wondering if you have any update on this long standing bug. A solution was provided both here and on github but after a year the bug is still present and the fix not integrated. From what I understood you have big plans for the slippage model so the bug fix has been postponed, but in the meanwhile wouldn't be better to simply accept the proposed fix? From time to time I stumble across this bug and I realize too late the great backtest results are not real, this is quite annoying and for users that are not aware of the bug is even worse. So this is just a gentle reminder about the existence of this bug ;)

Added tracking to the first backtest, these are the orders described originally, a partial sell at 111.02 instead of 111.00? See Source Code tab for the full logging output. You can use the tool in this backtest to help clarify. Why not add OHLC to it?

2015-08-03 06:45 sod:21 INFO SOD pos: 0
2015-08-03 06:45 sod:26 INFO 2015-08-03 13:45:00+00:00: SOD placing buy order for 921 of Equity(22739 [VTI]) at limit 110.500000 with current price 108.480000
2015-08-03 06:45 _trac:82 INFO   15   Buy 921 VTI _ at 108.48 limit 110.5              e2b7
2015-08-03 06:46 _trac:82 INFO   16      Bot 377/921 VTI (377) at 108.52 limit 110.5    e2b7
2015-08-03 06:47 _trac:82 INFO   17      Bot 413/921 VTI (790) at 108.51 limit 110.5    e2b7
2015-08-03 06:48 _trac:82 INFO   18      Bot all 131/921 VTI (921) at 108.58 limit 110.5    e2b7
2015-08-04 06:45 sod:21 INFO SOD pos: 921

2016-07-20 06:45 sod:21 INFO SOD pos: 921
2016-07-20 06:45 sod:31 INFO 2016-07-20 13:45:00+00:00: SOD placing sell order for -921 of Equity(22739 [VTI]) at limit 111.000000 with current price 110.680000
2016-07-20 06:45 _trac:82 INFO   15   Sell -921 VTI (921) at 110.68 limit 111.0        1727

2016-07-20 07:21 _trac:82 INFO   51      Sold -578/-921 VTI (343) at 111.02  (+1439) limit 111.0    1727
2016-07-20 07:22 _trac:82 INFO   52      Sold -91/-921 VTI (252) at 111.04  (+228) limit 111.0    1727
2016-07-20 07:23 _trac:82 INFO   53      Sold -37/-921 VTI (215) at 111.02  (+92) limit 111.0    1727
2016-07-20 07:24 _trac:82 INFO   54      Sold -27/-921 VTI (188) at 111.02  (+67) limit 111.0    1727
2016-07-20 07:26 _trac:82 INFO   56      Sold -42/-921 VTI (146) at 111.02  (+105) limit 111.0    1727
2016-07-20 07:27 _trac:82 INFO   57      Sold -54/-921 VTI (92) at 111.02  (+134) limit 111.0    1727
2016-07-20 07:28 _trac:82 INFO   58      Sold -60/-921 VTI (32) at 111.01  (+149) limit 111.0    1727
2016-07-20 07:33 _trac:82 INFO   63      Sold -921 VTI _ at 111.03 limit 111.0         1727
2016-07-21 06:45 sod:21 INFO SOD pos: 0

4
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5969e0976dc2845f10be6bab
There was a runtime error.

Can someone explain why Quantopian spends so much time adding additional functionality when the most fundamental of functions (limit order) is not being executed correctly!??

If exceptional functionality has to transact through garbage, the whole thing becomes garbage. I cannot believe how long this has gone without being fixed considering it invalidates any algo that uses limit orders.

The Quantopian tool is so great but is made practically useless from this error. You guys have something special here, PLEASE PLEASE fix this.