Back to Community
[Research] Do you want parameter optimization? Click here to get started. [Heat Maps included]

Okay, I lied a little. This isn't quite parameter optimization. It's more of a brute force parameter search.

What I'm about to show you isn't the most efficient, it isn't the most powerful, and it isn't the fastest and most innovative method out there. But it works, it's simple, and it might just make your life a whole lot easier! I've used this a lot to help with my own algorithm writing and development and it's been super helpful while trying to figure out what parameters (weights, symbols, etc.) are best suited for my algorithms.

The notebook has a much more clear description so I highly recommend that you click 'View Notebook' in order to get started but if you want the basic summary, here it is:

The Scenario

Let's say I want an algorithm that only longs AAPL and SPY but I don't know how much to hold in each security. I could try hitting 'build algorithm' 50 times in the IDE but that's tedious and my macbook can't open that many tabs. So what do I do?
The answer is easy: Create that same algorithm in Zipline and spin up 50 algorithm runs. Each run will have different parameters and because I can get the results of each run through Zipline, I can see exactly which parameter led to the best returns or Sharpe ratio.

Clonable notebook here: https://www.quantopian.com/posts/research-clonable-do-you-want-parameter-optimization-click-here-to-get-started-heat-maps-included

Loading notebook preview...
Notebook previews are currently unavailable.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

40 responses

Awesome, this is what I used to do in picloud a few years ago!

Thanks Seong,

Not sure what the function is in Python, but in MATLAB it is interp2 (http://www.mathworks.com/help/matlab/ref/interp2.html). You could use cubic spline interpolation to pretty up your heat map.

Grant

Thanks for the feedback all,

Let me know if it serves useful in your development.

Seong

If you get a name error, please insert from zipline.api import order_target_percent in the first code cell of the notebook. Thanks!

also add import pandas as pd to make sure all_sharpes = pd.DataFrame(all_sharpes) in step two works as expected

Thanks for that, I plugged in a different algorithm and get an error "IndexError: indices are out-of-bounds" that I dont get in Quantopian IDE. My questions are
1. How compliant is the code between research to Quantopian where the differences are documented?
2. how to debug line by line and watch variables in the research ?

Thanks,

The research environment is new and we're slowly rolling it out to more and more people. Our goal is to make it easy and seamless to move algos between research and the IDE, but we're not quite there yet. There are still some differences that we're working to massage.

In an IPython nteobook, there isn't a built-in debugger, but you can slim down the code to isolate the error. If you're stuck you can always reach out to us ([email protected]) or ask here in the forums.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

As Alisa said, there are a few differences between the Quantopian IDE algorithm and the algorithm you can plug into Research. Mainly that Research relies on Zipline in (basically) it's pure format. I would check out some of the tutorials that come factory with Research like the 'CLICK ME FIRST' or anything starting with 'Tutorial'!

Seong

@Seong,

  1. Cool tool, saves a lot of work.

  2. You say:

The best Sharpe ratio seems to come from an AAPL weight of 0.8 and a SPY weight

If you had included 1.0 in the ranges, would you have found out that a portfolio of 100% AAPL and 0% SPY would have an even higher Sharpe ratio? Could you rerun your analysis with these wider ranges and post it here?

  1. Can you explain why you ran your analysis over combinations that allocate from 0 to 200% of available cash, rather than only those that only allocate 100%?

Andre,

I think you might be right. Weighting AAPL with 1.0 and SPY with 0 probably would have been the highest Sharpe. And I think that testing your theory would be a great homework assignment for you to complete and share with the group! Just copy and paste his notebook (that is how I got started). Here is a hint: Python uses half-open intervals (a <= x < b), so his code only tested weights up to 0.8. You just need to modify these two lines:

aapl_weights = [weight for weight in np.arange(0, 1, .2)]  
spy_weights = [weight for weight in np.arange(0, 1, .2)]  

Regarding leverage, we can easily limit the leverage to 1.0 by adding this code:

for aapl_weight in aapl_weights:  
    for spy_weight in spy_weights:  
        if (aapl_weight + spy_weight) <= 1.0:  
            # Same code as before, goes here  

What's the difference between this and a normal MVO with traditional estimation?

Im using Research... it doesnt work too many Errors...

File "", line 2
This cell is going to create the basic framework of the algorithm
^
SyntaxError: invalid syntax

File "", line 2
This cell is going to load in the data, run the algorithm, and print out the Sharpe Ratio
^

SyntaxError: invalid syntax

NameError Traceback (most recent call last)
in ()
----> 1 aapl_weights = [weight for weight in np.arange(0, 1, .2)]
2 spy_weights = [weight for weight in np.arange(0, 1, .2)]
3
4 #: Create a dictionary to hold all the results of our algorithm run
5 all_sharpes = defaultdict(dict)

NameError: name 'np' is not defined

NameError Traceback (most recent call last)
in ()
18
19 #: Plot our heatmap
---> 20 heat_map(all_sharpes)
21
22 print all_sharpes

NameError: name 'all_sharpes' is not defined

John,

It looks like you're missing a double quotation mark and also don't forget to add

import numpy as np  
import pandas as pd  
from zipline.api import order_target_percent  

to the way beginning of the notebook

Thanks

already imported
import numpy as np
import pandas as pd
from zipline.api import order_target_percent

however...

File "", line 76
sharpe = (perf_manual.returns.mean()*282)/(perf_manual.returns.std() * np.sqrt(282))
^
SyntaxError: invalid syntax

File "", line 42
backtest_count += 1
^
SyntaxError: invalid syntax

File "", line 2
print("Maximum Sharpe = {0:.2f}").format(Max_Sharpe)
^
SyntaxError: invalid syntax

John

Mind posting the block that you're trying to execute? It looks different from the original version.

Sure...

cell 1 along this line...

# Trading
if short_mavg[context.aapl] > long_mavg[context.aapl]:
# order_target orders as many shares as needed to
# achieve the desired number of shares.
order_target_percent(context.aapl, 1)
elif short_mavg[context.aapl] < long_mavg[context.aapl]:
order_target_percent(context.aapl, 0)

data = get_pricing(
['AAPL','SPY'],
start_date='2012-01-01',
end_date = '2013-01-01',
frequency='daily'
)

algo_obj = TradingAlgorithm(
initialize=initialize,
handle_data=handle_data
)

: Run the backtest and save the result.

perf_manual = ( (algo_obj.rundata.transpose(2,1,0))

: Get the sharpe

           sharpe = (perf_manual.returns.mean()*282)/(perf_manual.returns.std() * np.sqrt(282))  

print "The Sharpe ratio is %0.6f" % sharpe

File "", line 76
sharpe = (perf_manual.returns.mean()*282)/(perf_manual.returns.std() * np.sqrt(282))
^
SyntaxError: invalid syntax

cell 2 alng this line
# Used to warm-up moving averages
context.i = 0

        algo_obj = TradingAlgorithm(  
            initialize=initialize,  
            handle_data=handle_data  
        )  
        perf_manual = ((algo_obj.run(data.transpose(2,1,0))  

        # Keep track of how many backtests were run  
        backtest_count += 1  
        print("Backtest {0} completed...").format(backtest_count)  

        # Calculate the sharpe for this backtest  
        sharpe = perf_manual.returns.mean()*282)/(perf_manual.returns.std() * np.sqrt(282))  

        #: Add the result to our dict  
        all_sharpes[short_mavg_day][long_mavg_day] = sharpe

print " "
print "All backtest simulations completed!"
print " "

all_sharpes = pd.DataFrame(all_sharpes)
all_sharpes.index.name = "Long Moving Average Days"

File "", line 42
backtest_count += 1
^
SyntaxError: invalid syntax

finally cell 3 along ds line...

Max_Sharpe = (max(all_sharpes.max(axis=1))
print("Maximum Sharpe = {0:.2f}").format(Max_Sharpe)

File "", line 2
print("Maximum Sharpe = {0:.2f}").format(Max_Sharpe)
^
SyntaxError: invalid syntax

bold means the bug ;)

John,

It looks like you might be missing a parenthesis somewhere.

E.g.

sharpe = perf_manual.returns.mean()*282)/(perf_manual.returns.std() * np.sqrt(282))  

needs a ( in front of perf

And I think print("Maximum Sharpe = {0:.2f}").format(Max_Sharpe)should just be print "Maximum Sharpe = %.02f" % Max_Sharpe

If you're using Python 3 for print statements the correct syntax is:

print("Maximum Sharpe = {0:.2f}".format(Max_Sharpe))  

The .format() component needs to be enclosed within the print() parentheses.

sharpe = (perf_manual.returns.mean()*282)/(perf_manual.returns.std() * np.sqrt(282)) doesnt work...

Hm interesting..

Given that you're getting a syntax error it looks like there's somewhere that your code isn't formatted correctly.. I'm rerunning my notebook now with the correct imports (e.g. import pandas as pd and from zipline.api import order_target_percent added) and it's working OK.

Maybe do a comparison between both and see where your code is different?

As for
Max_Sharpe = (max(all_sharpes.max(axis=1))
print "Maximum Sharpe = %.02f" % Max_Sharpe

File "", line 2
print "Maximum Sharpe = %.02f" % Max_Sharpe
^
SyntaxError: invalid syntax

Seong... i did not create my own... I just copy paste... on what you have shared... in ur notes it shud be plug and play Right...? I made no effort.. on adding something since it already work.. if your also using Research... to implement does code... why shudnt it work out of the box... again... im just copy paste from your note nothing more... ;)

I see a lot of unbalanced parentheses.

John - you say you just copy and paste, but you really select, copy and paste. Do you always make sure you select correctly?

yap.. mostly SyntaxErrors... but I can email you the code and why isnt working on my end.. since.. clone notebook botton is not available... why not copy paste it cell by cell it shud technicly... work right?? same codes... :) same platform... :)

John,

Here's a clonable version of the notebook: https://www.quantopian.com/posts/research-clonable-do-you-want-parameter-optimization-click-here-to-get-started-heat-maps-included

Seong

Any ideas as to why I would be getting the following error:

TypeError                                 Traceback (most recent call last)  
<ipython-input-9-3589cc079717> in <module>()  
     16  
     17 #: See this perf_manual object? I'm going to use that to get ending portfolio value  
---> 18 perf_manual = algo_obj.run(data.transpose(2,1,0))  
     19  
     20 #: Get the sharpe ratio

TypeError: transpose() takes exactly 1 argument (4 given)  

In Seong's notebook notice that when he gets the data he doesn't specify any fields, consequently the data variable is a multidimensional Pandas object and the .transpose(2, 1, 0) is a multidimensional transpose. See the Pandas Docs on Panel.transpose() If I had to guess, when you got your data via get_pricing you selected a single field, in that case no transpose is necessary and you can feed the data directly into algo_obj.run()

If you still have issues post here with some more code so folks have a better context of what is going wrong.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

James,

So I commented out the transpose part; but that only led to me getting a syntax error that I really just don't understand. This is my first time attempting to migrate my algo over to the research environment for more testing so forgive me if I'm missing something obvious, but I've looked through the api reference and I believe my syntax is correct here. If someone could shed some more light on this I would greatly appreciate it.

---------------------------------------------------------------------------  
TypeError                                 Traceback (most recent call last)  
<ipython-input-18-01b4a9baaf94> in <module>()  
     17  
     18 #: See this perf_manual object? I'm going to use that to get ending portfolio value  
---> 19 perf_manual = algo_obj.run(data) #.transpose(2,1,0))  
     20  
     21 #: Get the sharpe ratio

/home/qexec/src/qexec_repo/zipline/algorithm.pyc in run(self, source, overwrite_sim_params, benchmark_return_source)
    513  
    514         # create zipline  
--> 515         self.gen = self._create_generator(self.sim_params)  
    516  
    517         # Create history containers

/home/qexec/src/qexec_repo/zipline/algorithm.pyc in _create_generator(self, sim_params, source_filter)
    401  
    402         if not self.initialized:  
--> 403             self.initialize(*self.initialize_args, **self.initialize_kwargs)  
    404             self.initialized = True  
    405 

/home/qexec/src/qexec_repo/zipline/algorithm.pyc in initialize(self, *args, **kwargs)
    291         """  
    292         with ZiplineAPI(self):  
--> 293             self._initialize(self)  
    294  
    295     def before_trading_start(self):

<ipython-input-17-09c2cafca2ca> in initialize(context)  
     33  
     34     # security  
---> 35     context.stocks = symbols('NUGT', symbol_reference_date='2015-01-01')  
     36  
     37     # parameters

/home/qexec/src/qexec_repo/zipline/utils/api_support.pyc in wrapped(*args, **kwargs)
     49     def wrapped(*args, **kwargs):  
     50         # Get the instance and call the method  
---> 51         return getattr(get_algo_instance(), f.__name__)(*args, **kwargs)  
     52     # Add functor to zipline.api  
     53     setattr(zipline.api, f.__name__, wrapped)

TypeError: symbols() got an unexpected keyword argument 'symbol_reference_date'  

Steve... I already gone thru that.. there's no solving... ur problem... why dont you clone... Seong's notes dats it... problem solve...

@ Steve

So this ends up being a name space conflict. What is happening is that there are two methods called symbols() one in Research and one in the IDE. We know this is confusing and have had conversations internally about making this more clear. If you want to use the symbols function from the IDE while running a backtest in Research you have to omit the 'symbol_reference_date' parameter. This is because while running the backtest it is going to look up the symbol based on the current date in the backtest.

The reason we have the 'symbol_reference_date' in Research is so you can specify exactly which companies you want in case you are doing some historical analysis, as we know that symbols change.

Check out the Research Cheat Sheet for more tips on how to move between Research and the IDE

@ James

So I'm definitely getting close to the root here, I managed to get it to run without errors, but when I run the following cell it outputs that "the sharpe ratio is nan"?

"""
This cell is going to load in the data, run the algorithm, and print out the Sharpe Ratio  
"""

data = get_pricing(  
    'NUGT',  
    start_date='2013-05-01',  
    end_date = '2015-07-24',  
    symbol_reference_date = '2015-01-01',  
    frequency='minute'  
)

algo_obj = TradingAlgorithm(  
    initialize=initialize,  
    handle_data=handle_data  
)

#: See this perf_manual object? I'm going to use that to get ending portfolio value  
perf_manual = algo_obj.run(data) #.transpose(2,1,0))

#: Get the sharpe ratio  
sharpe = (perf_manual.returns.mean()*252)/(perf_manual.returns.std() * np.sqrt(252))  
print "The Sharpe ratio is %0.6f" % sharpe  

I assume this must mean that I do need transpose in their for some purpose, I'm just not sure what it needs seeing as I'm only loading data for a single stock.

I've just seen this post. This is so useful. It would be nice to address the problem of incompatibility between quantopian vs zipline algos, but that's a little issue though. Thanks!

I'm getting an

argument of type 'TradingAlgorithm' is not iterable

error. Any thoughts on how to address before I reinvent the wheel?

Hi Seong,

I try to use your notebook to my algo but it failt. I was told many functions or parameters are not ḱnown such as the scheduler_function, sid etc. Here is my algo. Could you or someone here please give me some help?

Cheers

Clone Algorithm
3
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 58061558ab71a313199c767b
There was a runtime error.

The parameters to be optimized are the context.min_rsi and the context.max_rsi.

And here is my notebook.

Loading notebook preview...
Notebook previews are currently unavailable.

Hey Thomas,

The issue is your having is because you haven't defined schedule_function amongst other things in your research environment. To get those functions in Research, you must specifically import them.

ex:

from zipline import TradingAlgorithm  
from zipline.api import order_target, record, symbol, history, order_target_percent, order, schedule_function  
from zipline.api import attach_pipeline, date_rules, time_rules, pipeline_output  

However, we don’t yet support Pipeline in zipline algos in the research environment. This means that you can't bind a pipeline to a trading algorithm in the Research environment, so you wouldn't be able to generate the backtests as shown in this notebook. You should backtest your algorithm in the IDE for now.

Matt

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thomas,

The issue you're having is because you haven't defined schedule_function amongst other things in your research environment.

To do so, on the top of your notebook include:

from zipline import TradingAlgorithm  
from zipline.api import order_target, record, symbol, history, order_target_percent, order, schedule_function  
from zipline.api import attach_pipeline, date_rules, time_rules, pipeline_output  

However, we don't yet support Pipeline for zipline algos in the research environment. So - any algorithm that attaches a pipeline to a TradingAlgorithm won't work in research. You should stick to backtesting in the IDE for this one.

Hope this helps!

Hi Matthew,

Thanks for the answer first.

But does this mean I have no chance to do the "optimization" as Seong described here, so far I use the pipeline? I wonder if there is a way or alternative to do this since this will save a lot of time by opening many console/windows or by dozens of clicking start the backtesting in IDE.

Cheers

Hi Thomas, Matthew,

started playing around with optimization and backtest automation and was wondering if by now it is possible to do this using pipeline in the research environment? As all my algorithms written in the IDE are using Pipeline...

Many thanks in advance for the update!
Jens