I hope that my friends, the "usual suspects" James, Joakim, Grant, Karl, Blue, etc might pick up on this one, but just FYI for anyone who doesn't know me, i have been trading my own very modest little account for > 30 years and still going.
As newbie traders, perhaps some of us might remember having been attracted to advertising nonsense like "97% win rate with this amazing system" and hoped to be able to do even better because we thought we were pretty clever. After a few (few hundred more likely) losses, we maybe then succumbed to the trader psychology concept of "accepting our losses as just part of the cost of trading". And maybe we stayed with that. After all, if we understand the "traders equation" regarding expectancy and if ours is positive, then basically we are doing OK, right? However with time, we started to wonder if we could do better and, more specifically, just how much better could we do? So then we tried NN & ML & DL & AI & whatever. Did they help us? As much as we had hoped?
I have just been having some discussion with James here on Q about the topic of avoiding bad trades and how, from a practical traders perspective, even if not from a ML theory perspective, missing out on good trades is definitely NOT symmetrical with avoiding bad trades. I would now like to present the following intentionally provocative comment: You are probably OVER-TRADING and more-over, unless you have considered this very carefully, probably about 25 - 30% of your trades are DOOMED LOSERS ..... AND a significant proportion of those could be AVOIDABLE ..... and THIS issue probably merits a LOT more attention than most of the system-tweaking that you / we are doing!!!
Before you conclude that i must be looking for a fight, please forgive my deliberately provocative wording style. How can i know for sure what you are doing? Well of course i don't, but i would like to attempt to convince you, or rather to allow you to convince yourself, of the probable truth of my statements above, and so hopefully you might then engage with me in more discussion about the value of and methodologies for avoiding bad trades, even at the expense of missing a few of the good ones.
So, here is an experiment for you to try. You will (as far as i know) not be able to do this on Quantopian, so you will have to use some other platform of your choice, but hopefully after doing this experiment, you will then be able to bring your learning back here to Q and apply it. Here is an outline of the experiment. It is an experiment NOT a trading system, and you must be able to use look-ahead, as follows: For each bar, starting with the closing price, see how much you could have gained or lost (if stopped out) by the end of the next bar (1-bar look-ahead) if you had been Long and similarly if you had been Short. Do the same thing for look-ahead periods of 2, 3, ... up to 10 bars. Using John Sweeny's concepts of Maximum Favorable & Maximum Adverse Excursions, track your MFE & MAE for both Long & Short for all periods between 1 & 10 bars. Within this range, choose whatever you think is typically your longest trade duration (or just use 10 bars ahead) and calculate the maximum values of the MFEs & MAEs. Normalize everything to percentage moves or (even better) to units of ATR. Now decide what you think is the minimum trade result (in ATR units) that you would like to accept for your trades and set this as a threshold for "viable trade" MFEs, with anything less that that being "too small". Do the same thing for maximum MAEs that you would be comfortable allowing without being stopped out, and treat anything larger than that as "too large". What remains as trades bounded by these thresholds can be considered as "viable", by whatever threshold criterion you chose. Do the same thing for both Long & Short trades and plot the results along with the price series bars. What you will find is that, even IF you had the benefit of perfect foresight (look ahead) which of course we don't, you will find that looking ahead N bars, there are many times when we could not actually do "viable" trades EITHER Long OR Short, and that any trades placed at such times will inevitably be losers or "go-nowheres". This result is based only on price data and is independent of any particular trading system, so any actual system will inherently be even less successful unless it specifically takes these "non-viable trading bars" into consideration. Now, the shock to me was to find out just how large is the proportion of bars (for a wide range of stocks, indices, futures, FX or whatever else you might want to look at) that inherently represent non-viable trading conditions. Of course it depends on the ATR threshold chosen, but with 1 ATR as a threshold, i was surprised to find that somewhere between 15% and 40% (typically about 20-30%) of all bars were NOT useful as candidates for "viable trades" at all, irrespective of whether either Short or Long! Logically these are bars that we SHOULD be diligently seeking to exclude, even BEFORE presenting to our trading system. So, my conclusions from this little experiment were:
1) Even with the benefit of look-ahead, a surprisingly large proportion of bars are not suitable as entry candidates, either Short or Long.
2) For any real systems (with no look-ahead) the proportion of bars that we should logically exclude will be even greater.
3) If we are not excluding these "non-viable" bars and are entering trades on them, then we are inherently over-trading (and losing).
4) Trades entered on such bars will either be losers or will produce results that are so small as to be of minimal benefit.
5) The best way that i can think of to improve trading system (irrespective of the system) performance is to avoid these "useless trade bars".
If you actually try this for your self, then i look forward to reading and sharing your comments which will of course be most warmly welcome.
With best regards, Tony.