There may be a bug in the most commonly used Quantopian data for at least some stocks, or there is a gap in my understanding of how I am to use these data.
I've been trying to understand why I get so many partial orders with some strategies.
This naturally caused me to take a look at my trade sizes vs the historical daily volumes.
I do not understand the apparent inconsistency between data presented by pipeline and that presented outside of pipeline.
Here are some observations:
1) There is likely an inconsistency among price and volume data used by pipeline features like AverageDailyVolume() and that used to make/manage orders
2) The daily volume reported through pipeline and the normal interface do not match. The degree of mismatch is large and not constant in offset or multiple
3) Neither of the internally reported volumes match data published on yahoo.com
Attached is a backtest for a single stock [MTEX] from 7/1/2003 through 7/31/2003
The following transactions are made
7/1: 2963 shares ordered. 2963 purchased
7/8: -2963 shares ordered. 2963 sold
7/15: 5184 shares ordered. 1537 purchased 3647 shares were not filled
7/22: -1537 shares ordered. 1537 sold
7/29: 5509 shares ordered. 2239 purchased 3270 shares were not filled
Here is where my confusion begins.
The logged messages for total trading activity on the above trading days are below
These show that 6500 shares were traded on 7/15 and 3800 on 7/29.
Why then were my trades limited to 1537 and 2239, respectively?
2003-07-02 yesterdays_news:110 INFO yesterday MTEX traded 4600 shares at $ 7.41
2003-07-09 yesterdays_news:110 INFO yesterday MTEX traded 16400 shares at $ 6.00
2003-07-16 yesterdays_news:110 INFO yesterday MTEX traded 6500 shares at $ 6.62
2003-07-23 yesterdays_news:110 INFO yesterday MTEX traded 12300 shares at $ 8.49
2003-07-30 yesterdays_news:110 INFO yesterday MTEX traded 3800 shares at $ 8.06
Investigating further I find that the daily volume data in pipeline and the standard, nonpipeline interface do not match.
Note: that the data are for the trading date prior to the logging date, so they don't align with the data above
2003-07-01 periodic_rebalance:93 INFO Vpipe = 638965 Vquant = 29170
2003-07-08 periodic_rebalance:93 INFO Vpipe = 246630 Vquant = 46400
2003-07-15 periodic_rebalance:93 INFO Vpipe = 111950 Vquant = 3900
2003-07-22 periodic_rebalance:93 INFO Vpipe = 332663 Vquant = 10200
2003-07-29 periodic_rebalance:93 INFO Vpipe = 61250 Vquant = 6200
From yahoo.com I find that the actual trading volumes as shown below.
pipeline and nonpipeline values are show for comparison
2003-06-30 V(yahoo) = 64,900 Vpipe = 638965 Vquant = 29170
2003-07-01 V(yahoo) = 46,300 Vpipe not logged Vquant = 4600
2003-07-07 V(yahoo) = 26,400 Vpipe = 246630 Vquant = 46400
2003-07-08 V(yahoo) = 40,000 Vpipe not logged Vquant = 16400
2003-07-14 V(yahoo) = 13,000 Vpipe = 111950 Vquant = 3900
2003-07-15 V(yahoo) = 7,700 Vpipe not logged Vquant = 6500
2003-07-21 V(yahoo) = 43,500 Vpipe = 332663 Vquant = 10200
2003-07-22 V(yahoo) = 62,100 Vpipe not logged Vquant = 12300
2003-07-28 V(yahoo) = 6,200 Vpipe = 61250 Vquant = 6200 (6200 matches yahoo)
2003-07-29 V(yahoo) = 9,600 Vpipe not logged Vquant = 3800
This problem persists into the present with MTEX as seen in this data from 7/1/2016 through 7/31/2016
2016-07-06 periodic_rebalance:93 INFO Vpipe = 666 Vquant = 0
2016-07-12 periodic_rebalance:93 INFO Vpipe = 1926 Vquant = 893
2016-07-19 periodic_rebalance:93 INFO Vpipe = 34112 Vquant = 600
2016-07-26 periodic_rebalance:93 INFO Vpipe = 4474 Vquant = 161
|Returns||1 Month||3 Month||6 Month||12 Month|
|Alpha||1 Month||3 Month||6 Month||12 Month|
|Beta||1 Month||3 Month||6 Month||12 Month|
|Sharpe||1 Month||3 Month||6 Month||12 Month|
|Sortino||1 Month||3 Month||6 Month||12 Month|
|Volatility||1 Month||3 Month||6 Month||12 Month|
|Max Drawdown||1 Month||3 Month||6 Month||12 Month|