Seeing inconsistent results with the same code. The first two are two different tabs in the browser using the same code:
Returns Alpha Beta Sharpe Drawdown 4.52 0.82 0.08 7.95 -0.75 3.79 0.70 0.08 6.76 -0.85
The dictionary object in Python is stored and presented in an indeterminate order (same as Perl's hash and others) so looping through a dictionary offers the possibility of variance in ordering. So I wrapped all of those instances in sorted() with two new tabs, identical, and again, different results both from each other and the previous two examples:
Returns Alpha Beta Sharpe Drawdown 4.53 0.83 0.08 7.54 -0.77 4.84 0.87 0.08 8.42 -0.71
Next I thought it could be possible that pipeline might not be entirely consistent however I'm always printing some detail the first time through and these are all identical except for the wallclock time:
2016-01-05 05:45 before_trading_start:213 INFO pipe len 170 2016-01-05 05:45 before_trading_start:215 INFO lng 68 top DNR 0.005814 bottom KSS 0.001204 2016-01-05 05:45 before_trading_start:217 INFO shs 68 top SWKS -0.001204 bottom HRI -0.005814 2016-01-05 06:31 ti:482 INFO 2016-01-05 to 2016-01-26 $10000000 2017-10-18 01:35 US/Pacific
In ordering on the back end if anything is stored in a dictionary rather than ordered it could potentially result in this?
My only remaining theory would be timing in processing orders, something asynchronous. That is, if orders and cancels are all being issued in order, it is reasonable to assume for speed they may not be received back sequentially, a tiny difference in processing time might cause different results. Perhaps more concretely, for instance I could be checking to see if an order has been canceled, maybe sometimes it would be, and sometimes the memo hasn't been received yet? I don't feel good in suggesting that and yet thought it should be mentioned in case. If the development team might want to take a stab at this, one route might be to grab an existing backtest and be sure there will be a lot of partial fills should that possibly be part of it perhaps.
Unfortunately I can't boil this code down to a sharable level, will have to rely on any others who might at some point have a backtest example.
Anyone else notice this?