Back to Community
Quantopian Tutorial with Portfolio Rebalance Algorithm: Lesson 2 - Universe, Fetcher, and Schedule_function

2/16/2016 This tutorial is out-dated, please view the new versions here: https://www.quantopian.com/posts/quantopian-tutorials

Welcome to the second lesson in the Quantopian Tutorial Series. In this lesson we extensively cover the universe of securities that will be in your algorithm, using fetcher to load in data to create your universe, and a portfolio rebalancing algorithm that uses schedule_function for minutely mode.

As a refresher, these videos covered the basics of the IDE.

Summary
We covered:
- Universe of Stocks: What actually belongs there and how you set your securities using multiple methods
- How to use Fetcher to set your Universe
- A daily rebalancing portfolio algorithm with schedule_function

Links
- Youtube: http://bit.ly/1swuNpW
- PDF Documentation and Guide: http://bit.ly/1Kvw03r
- First Quantopian Tutorial Series - Basics of the IDE: http://bit.ly/1wTmW7m
- Third Quantopian Tutorial Series - Basics of Fundamentals: http://bit.ly/16HhrNR

FEEDBACK
We're still building this out so I'd love to hear what you guys think. What was helpful? What wasn't helpful? What would you like to see next? Feel free to email us at [email protected] with your comments and suggestions.

** Credits to Jessica Stauth for writing up the algorithm found here

**EDIT:

On line 28 please use this new line instead:

symbols = [s.symbol for s in sids if s != 0] # instead of [s.symbol for s in sids]  
Clone Algorithm
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
'''
This algorithm uses the new universe_func callback inside of fetch_csv to define an investible
universe of sids based on an externally sourced list of index constituents. 

In this example we are using a csv file containing the list of SP500 constituents and GICS
sector codes from Quandl. This file is a static snapshot -- but you can point to a csv file
with varying universe composition over time and fetcher will correctly update the universe
with the current constituents.

'''


import datetime
import pandas as pd
import numpy as np

def preview(df):    
    log.info(' \n %s ' % df.head()) 
    return df

# Function for returning a set of SIDs from fetcher_data
def my_universe(context, fetcher_data):
    
    # Grab just the SIDs for the Financials sector w/in the SP500:
    financials = fetcher_data[fetcher_data['GICS Sector'] == 'Financials']
    
    sids = set(financials['sid'])
    symbols = [s.symbol for s in sids]
    context.count = len(symbols)
    print "total universe size: {c}".format(c=context.count)
    
    # Compute target equal-weight for each stock in the SP500 Financials universe
    context.target_weight = 1.0/context.count
    return sids

def initialize(context):
    
    # Fetch the SP500 constituents -- sourced from Quandl
    # https://s3.amazonaws.com/quandl-static-content/Ticker+CSV%27s/Indicies/SP500.csv
    # static snapshot of the SP500 constituents as of 10/2013, along with GICS sectors
    # I'm grabbing the data from a file on my dropbox folder which I modified by adding
    # a date column, alternatively you could add the date inside of a more complicated
    # pre_func and use the csv file as is.
    
    fetch_csv(
       "https://dl.dropboxusercontent.com/u/169032081/SP500.csv",
        pre_func=preview,
        date_column='date',
        universe_func=(my_universe))

    context.target_weight = 0.01
    
    schedule_function(func=rebalance,
                      date_rule=date_rules.every_day(),
                      time_rule=time_rules.market_open())
    
def rebalance(context, data):
    # Loop over every stock in the Financials sector and make sure we have an equal-weight
    # exposure using the order_target_percent() method.
    for stock in data:
        # Guard for missing stock data
        if 'price' in data[stock]:
            order_target_percent(stock,context.target_weight)
        else:
            log.warn("No price for {s}".format(s=stock))
    
def handle_data(context,data):
    pass
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

16 responses

You mentioned that the max allowed size of a universe is 200, but this does not seem to be the case.

set_universe(universe.DollarVolumeUniverse(floor_percentile=90, ceiling_percentile=100))  

results in a set of 802 securities outputted via
print len(data) Will it trade based on all of these and if not how does it choose a subset of 200?

Hi Colin,

So the 200 max allowed universe size applies to minutely mode, but is a bit different when you're working in daily mode (with a larger allowed universe size)

To use the DollarVolumeUniverse, set the universe in the initialize
method. You can set a universe interval up to 2% for backtests in
minute mode or up to 10% in daily mode. Each percentile in the range
will result in ~80 securities, so the following example consists of
the ~160 securities with the highest dollar-volume volume, with
changes coming on the quarter end.

Let me know if you have any questions

Ah OK, thanks for the fast response, makes sense. Another issue I am wondering if you are able to reproduce - when I clone the algorithm you listed above it functions correctly, but if I create a new one and paste in the exact same code it never hits breakpoints set in my_universe.

Hi Colin,

That is strange! I just tried copy/pasting from my algorithm into another one and it seems to be working for me.

Here's the screenshot: https://copy.com/DXphhjHrHeLZBBpj

Mind sending me a collab to your algorithm so I can take a closer look? [email protected]

Hi Colin,

I suspect on the algo you just shared with me that you're running into a similar error. Do you mind trying your prior solution and checking the dates and see if that works?

That is what I thought as well (start date in backtester not matching the one entered in the csv file), but it is not the case. When the dates match up there is still no trading activity.

Hi Colin,

The issue was that the date column in your FETCHER file started on a holiday. The best way to fix this is to make sure that your dates are on a trading day, not a holiday. In this case, I changed the date from 1/1/13 to 1/2/13 by adding in a df['date'] = '1/2/13' in your pre_func method.

If you don't mind, perhaps we could share snippets of your algorithm on this thread so that others who might run into the same problems might benefit as well?

I've also added in a 'if stock in data and 'price' in data[stock]' to make sure that we have data/volume for a security before trying to execute an order for it.

Thanks,
Seong

Sure that is fine with me. Thanks for figuring out the issue.

I've also tried backtesting with a start date that is after the one in my csv and no trading occurs.
-What is the best practice in setting a date for live trading? It seems based on this that it would need to be a future date since fetcher retrieves the universe from the file only once it hits the date.

Seong, I'm wondering the same thing as Colin, about the best practice in setting a date for live trading.

In other words, for a live trading algorithm, would you have to set df['date'] = datetime.today().strftime('%m/%d/%y') in your pre_func?

Audrey,

That's a great question. Let me try setting that up in a live algorithm and will post my findings here.

It looks like

    log.info("%r" % datetime.today().strftime("%m/%d/%y"))  
    df['date'] = datetime.today().strftime('%m/%d/%y')  

Did in fact change the dates of the CSV file to the current day

@Seong: Could you provide some more information about the maximum size of the universe in different cases. I mean how large can the universe be in case of daily data?
furthermore, I just tried to set the universe to 500 stocks using
"update_universe(fundamental_df.columns.values)" in the "before_trading_start(context)" method and I get the error message "UniverseExceedsSizeLimit: 0029 Universe size 500 exceeds limit of 200 for daily bars." so is it only possible to use a universe > 200 stocks when it is initialized from "initialize(context)"?

thanks in advance

The latest version of the backtest that fixes a broken fetch_csv universe method.

If you're coming from an old version, on line 28 please use this new line instead:

symbols = [s.symbol for s in sids if s != 0] # instead of [s.symbol for s in sids]  
Clone Algorithm
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
'''
This algorithm uses the new universe_func callback inside of fetch_csv to define an investible
universe of sids based on an externally sourced list of index constituents. 

In this example we are using a csv file containing the list of SP500 constituents and GICS
sector codes from Quandl. This file is a static snapshot -- but you can point to a csv file
with varying universe composition over time and fetcher will correctly update the universe
with the current constituents.

'''


import datetime
import pandas as pd
import numpy as np

def preview(df):    
    log.info(' \n %s ' % df.head()) 
    return df

# Function for returning a set of SIDs from fetcher_data
def my_universe(context, fetcher_data):
    
    # Grab just the SIDs for the Financials sector w/in the SP500:
    financials = fetcher_data[fetcher_data['GICS Sector'] == 'Financials']
    
    sids = set(financials['sid'])
    symbols = [s.symbol for s in sids if s != 0]
    context.count = len(symbols)
    print "total universe size: {c}".format(c=context.count)
    
    # Compute target equal-weight for each stock in the SP500 Financials universe
    context.target_weight = 1.0/context.count
    return sids

def initialize(context):
    
    # Fetch the SP500 constituents -- sourced from Quandl
    # https://s3.amazonaws.com/quandl-static-content/Ticker+CSV%27s/Indicies/SP500.csv
    # static snapshot of the SP500 constituents as of 10/2013, along with GICS sectors
    # I'm grabbing the data from a file on my dropbox folder which I modified by adding
    # a date column, alternatively you could add the date inside of a more complicated
    # pre_func and use the csv file as is.
    
    fetch_csv(
       "https://dl.dropboxusercontent.com/u/169032081/SP500.csv",
        pre_func=preview,
        date_column='date',
        universe_func=(my_universe))

    context.target_weight = 0.01
    
    schedule_function(func=rebalance,
                      date_rule=date_rules.every_day(),
                      time_rule=time_rules.market_open())
    
def rebalance(context, data):
    # Loop over every stock in the Financials sector and make sure we have an equal-weight
    # exposure using the order_target_percent() method.
    for stock in data:
        # Guard for missing stock data
        if 'price' in data[stock]:
            order_target_percent(stock,context.target_weight)
        else:
            log.warn("No price for {s}".format(s=stock))
    
def handle_data(context,data):
    pass
There was a runtime error.

Hey everyone, just out of curiosity how come we don't include a random walk with the fundamentals we've chosen in these graphs? We include a standard SPY benchmark and our algo, but I would think adding a random walk would also be important.

I got here from the tutorial notebook: " 2. Getting Started: Financial Modeling". But then at the top of this post it says:

2/16/2016 This tutorial is out-dated, please view the new versions here: https://www.quantopian.com/posts/quantopian-tutorials

So I follow that link to the main tutorials page where it refers users back here again as a more advanced tutorial. Can you please clarify the status of this post? Is it out-dated? More advanced?

Hi Chris,

Sorry for the confusion! I've removed the pointer from the basic tutorials to this one. This tutorial is indeed out of date, and we're working on replacing them with new 'intermediate' tutorials to follow the basic ones, all part of an effort to improve the learning experience for newcomers. We're working on creating written and video tutorials - stay tuned!

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.