Back to Community
Inconsistent results, a mystery not yet solved

Seeing inconsistent results with the same code. The first two are two different tabs in the browser using the same code:

Returns  Alpha    Beta     Sharpe   Drawdown  
4.52     0.82     0.08     7.95     -0.75  
3.79     0.70     0.08     6.76     -0.85  

The dictionary object in Python is stored and presented in an indeterminate order (same as Perl's hash and others) so looping through a dictionary offers the possibility of variance in ordering. So I wrapped all of those instances in sorted() with two new tabs, identical, and again, different results both from each other and the previous two examples:

Returns  Alpha    Beta     Sharpe   Drawdown  
4.53     0.83     0.08     7.54     -0.77  
4.84     0.87     0.08     8.42     -0.71  

Next I thought it could be possible that pipeline might not be entirely consistent however I'm always printing some detail the first time through and these are all identical except for the wallclock time:

2016-01-05 05:45 before_trading_start:213 INFO pipe len 170  
2016-01-05 05:45 before_trading_start:215 INFO lng 68  top DNR  0.005814  bottom KSS  0.001204  
2016-01-05 05:45 before_trading_start:217 INFO shs 68  top SWKS -0.001204  bottom HRI -0.005814  
2016-01-05 06:31 ti:482 INFO 2016-01-05 to 2016-01-26  $10000000  2017-10-18 01:35 US/Pacific  

In ordering on the back end if anything is stored in a dictionary rather than ordered it could potentially result in this?
My only remaining theory would be timing in processing orders, something asynchronous. That is, if orders and cancels are all being issued in order, it is reasonable to assume for speed they may not be received back sequentially, a tiny difference in processing time might cause different results. Perhaps more concretely, for instance I could be checking to see if an order has been canceled, maybe sometimes it would be, and sometimes the memo hasn't been received yet? I don't feel good in suggesting that and yet thought it should be mentioned in case. If the development team might want to take a stab at this, one route might be to grab an existing backtest and be sure there will be a lot of partial fills should that possibly be part of it perhaps.

Unfortunately I can't boil this code down to a sharable level, will have to rely on any others who might at some point have a backtest example.

Anyone else notice this?

1 response

I don't know if it's related, but I had some problems with the code editor yesterday. It would sometimes not take my changes into account when building, and I had some wierd errors with code indentation.