Back to Community
Run Summary

A way to print a summary at end of run or status on a specified date, code in the next post.
[Edit: Earlier code removed, added a later better version instead, grab that one]

Example

16 responses

Wow! thanks Gary, I'll be adding this to the quantopian framework I made. https://github.com/Novaleaf/QuantShim

Much improved in part thanks to this.

Clone Algorithm
48
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 543ecc12ec7c1a09018faf17
There was a runtime error.

Hi Gary, Jason, and all,

Keep in mind that the 'dir' hack is probably not officially supported (I'm guessing). Since it is not part of the user interface, it could change and one day the code above would not work.

Quantopian support team, is there a better approach that could be put on the roadmap for incremental improvements?

Grant

We recently made dir(obj) available in the IDE to see an object's attributes, and we're definitely planning to keep that functionality. But the string representation of the dir function itself, as opposed to the value returned by calling it, could certainly change in the future.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hello Alisa,

Is there any way to get the value of 'last_close' without applying the hack?

Grant

Grant, I think your version is the easiest (though not convenient!) route right now. I imagine we will make this process easier with new tools that we add.

Hello Gary,

One way to tell if you are backtesting is to measure the delay between calls to handle_data in real time (not simulation time). I figure that if you are backtesting, the time difference between calls should be less than 50 seconds. Under paper/live trading, the time difference should be approximately 60 seconds.

Grant

from datetime import datetime

def initialize(context):  
    context.spy = symbol('SPY')  
    context.prior_call = datetime.now()  
    context.first_call = False # set to True after first call to handle_data  
    context.backtest = None # True for backtest, False for paper/live  
def handle_data(context, data):  
    now = datetime.now()  
    dt = (now - context.prior_call).total_seconds()  
    context.prior_call = now  
    if not context.first_call:  
        context.first_call = True  
        return  
    if dt < 50:  
        context.backtest = True  
    else:  
        context.backtest = False  
    print dt  
    print context.backtest  
Clone Algorithm
2
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 545356f903af2c09287f27e8
There was a runtime error.

Here's an improved version that is now even easier to drop into your algorithm. Check it out, clone and run it quickly now, it's fast.

  • Run Summary now works with set_universe too and also fetcher's universe_func.
  • Accounting on bar after fill for more accuracy.
  • Self-contained, all variables within context.books.
  • Simplified. And fewer modules.
  • Only needs the function and one call to it in handle_data().
  • Detects backtest vs live/paper and daily vs minute.
  • Currently set to log summary at the end of every day when live/paper.

Takes only seconds to add to your algo, note though in handle_data be sure to place the call to summary above any 'return' or could be prevented from printing summary info on the last bar.

Just ignore all of initialize() and everything except the call to summary in handle_data onward (the strategy and graph you see at some 25000% was just sort of for the fun of it).
By the way I'm still not certain on what to do with partial fills, currently the original order is cancelled afterward however it seems another separate order already exists for the remainder.
I have tested this and worked thru some bugs by cloning over 50 of your algo's and dropping this summary into them, and yet it surely has a lot of room for improvement so ...
Anyone is invited to feel free to improve on it.
Currently uses the interim method for environment info and so I plan to replace this when the new environment method arrives, thank you.

Clone Algorithm
43
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5483e8745636e5091a390171
There was a runtime error.

Gary,

F-ing great I would say, really helpful.

To make it even more useful I would add some metrics:
Number of winning trades, number of losing trades in total and per Symbol + ratio, and the average return per trade in total and per symbol. Those are the metrics I use to determine whether a strategy is improving or good enough to improve. I don't mind to help to improve but then again your skills are clearly better than mine and I don't want to stuff up your script

Run Summary new version here can also cope with schedule_function (ironically also requires schedule_function to work).
Well, that's ok, the call to summary just moves out of handle_data to initialize().

I like those suggestions Peter. Meanwhile it seems like a miracle to me that I've been able to even progress this far. Hey, Grant K, you busy? (Or anyone)

Today is the worldwide debut of this great new tagline:
Run Summary stays out of your way.
Yeah I know, back to the drawing board for my marketing team.

Clone Algorithm
43
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5492335ae74643090d20bf77
There was a runtime error.

Hi Gary,

I haven't followed this too carefully, and yes, I'm generally busy. Is there something that is not working as you'd like? Where do you need help?

Grant

One problem you might like: The individual xval seems off in some scenarios. If you clone my backtest here and set so that context.symbols is only 'SPY' and run that, you'll see that the overall return is x0.817 while SPY shows x-0.3, a head-scratcher for me. Something involving this line: drawdown = min( avg_init_cash, abs(b[sym]['cash_low']) ) That's the main thing on my radar.

Then there are wish lists, my own and Peter Bakker's items above.

With the mod for schedule_function, long story short, summary() could no longer take arguments except context and data, so one cannot tell it to print whenever they want, say, after every order while the ordering is sparse. Maybe there is a way for someone more versed in Python. I tried a 'def summary_print()' indented inside summary() which I thought could be called like any other function, nope, uninitialized error in this case.
Sometimes people set an end date on a weekend and the summary won't happen when they do, it would be great to use trading calendar to adjust to the last trading day (usually the Friday before).
Or just anything that happens to catch your interest.
Thanks and I can be emailed with a click on my name if there are any details to discuss, anyone, I'd rather outside of the forum.

Gary,

Regarding being able to control printing within summary(), you might try:

def initialize(context):  
    context.print = False  

Then, from within handle_data, when you want to print, set:

context.print = True  

Within summary(), you would then use:

if context.print:  
    # print to log  

This way, summary() would still be called per the schedule, but output would be suppressed unless you want it.

Grant

Updated with the new additions in Q's get_environment(). No imports now. Couple of improvements. This also alerts you to negative cash. With get_fundamentals there might be 1000's of securities encountered and many not bought nor sold, you can turn off reporting those in the summary, just look for the word 'filter'.

[Removed backtest to replace with an update]

Improved

Clone Algorithm
43
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54ab3f3d1478200917c61d8e
There was a runtime error.

Run Summary can haz productivity increase.

In this new version:

Options added, alphabetical list ...

    #  - - - - - - - -  Options  - - - - - - - -  
    cancel_partials  = 1  # Cancel orders after being partially filled.  
    daily_live       = 1  # Log summary at end of each day when live.  
    drawdown_returns = 0  # Custom chart returns based on profit/drawdown.  
                          #  At 1/2015 Q returns are profit/init_cash.  
                          #  Custom chart only accepts 5 items so watch for that.  
    filter_zeros     = 0  # 0 or 1 to filter out those with no buy and no sell.  
                          # In get_fundamentals for example there can be over  
                          #   a thousand stocks processed with many not traded.  
    leverage_alert   = 1  # Log new lowest cash points reached.  
    percent_results  = 0  # Express results like 270% instead of x2.7  

It currently alerts to negative cash, that could be annoying if you're into leverag, can turn that off.

Upper output fewer lines, something like this ...

            Portfolio: 342690                                                              `  
         Initial Cash:   1000                          Buys: 217225 (147 trades)           `  
          Unused Cash:      0                         Sells: 77559 (11 trades)             `  
             Neg Cash:    -21                   Commissions: 1523                          `  
             Drawdown:   1021 (102%)             Shares Now: 139666                        `  
          Cash Profit:   -991                  Shares Value: 342681                        `  
         Total Profit: 341690  w/ shares               Cash: 9                             `  
              QReturn:   x342  Profit/InitCash                                             `  
               Return:   x335  Profit/Drawdown                                             `  
        2015-01-02 summary:616 INFO             200 average initial cash, 5 securities     `  
               Relativ   Buy|    By|Sl     By|Sl    Price    Draw    Cash    Shrs   Shrs   `  
         Symbol  Ratio   Hold    Count     Evnts   Strt|Now  Down     Now     Now   Value  `  
           RDNT   x32.0   2.2 16849|16849   20|2     3|9     -39502   39768       0       0`  
           NVAX   x9.71   2.0 23042|23009   18|3     2|6     -34756   10439      33     190`  
           EDAP   x21.2   0.5 143855|4419   34|1     2|2    -205336 -205336  139436  341618`  
           ACHN   x61.8   0.6 22133|22133   38|2     8|13    -49470   96132       0       0`  
           HGSH   x1548  10.8 11346|11149   37|3     0|4      -1019   58273     197     872`  

The backticks at the end of lines are for doing replace-all in an editor because after copy/paste everything is all run together at least in Win7/Firefox. [Edit: Seems changed ~Apr 2015, no longer necessary, kudos]

Nitty gritty:
I tried to find a way to calculate the individual securities values based on their maximum spent and did not succeed so those values are analogs (proportional) to the overall calculation (which does work), it is their mean.

This has references along the lines of Drawdown Returns in the code or just "Return" in the output, more here regarding a different way of looking at returns, a "Return" value based on maximum spent.
That "Return" value is constant even if you change initial capital, the Quantopian Returns value (QReturn) is tied to your initial capital setting, change initial capital and your returns will change, sometimes dramatically at least with most code.
My use of the word Drawdown is not the same as the DRAWDOWN you see in the IDE, I should probably change it, next version.

I haven't tested this as thoroughly as some in the past so let me know if you hit a bug.

Clone Algorithm
37
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54b6edcb09f5a109b19805f8
There was a runtime error.