Back to Community
Define custom universe via Fetcher using the new universe_func callback

Apparently it's new feature week here at Quantopian... here is another one we are excited to roll out!

You can now define your algorithm's universe of securities programmatically using a new callback function inside of fetch_csv.

In this example I grabbed a static list of the SP500 constituents and their GICS sector codes (as of 10/2013) from Quandl and used the universe_func callback to set SP500 Financials (list of 82 SIDs) as my investible universe.

There are a number of benefits to this functionality, perhaps the simplest one is allowing users to define a large universe of stocks without have to enter each SID manually in the initialize() method.

Because fetch_csv is smart about aligning external time series data with Quantopian's market data, you can also use this new callback to define dynamic universes. Simply make sure that your universe csv file has an 'as of' date associated with each ticker and your investible universe will be populated by the SIDs from the most recent day of data prior to the backtest date on a rolling basis (I didn't cover that option in this example -- but I will post a follow up example of this here ASAP).

I think this post may have beat the help docs being updated, but docs soon to follow. As always we would love to hear feedback or questions on using this new feature!

Clone Algorithm
392
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 53628e2cd324b607187eb39b
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

16 responses

What is the date convention here? Is it dd/mm/yy? Is it possible to specify how the date string is passed?

This looks awesome! I'm trying to do the same for stocks that hit a 52 week low on the NYSE, AMEX and NASDAQ.

This sort of data http://www.quandl.com/URC/AMEX_52W_LO-AMEX-Number-of-Stocks-Making-52-Week-Lows

@jessica How would I set that universe?

Hi Simon,

See an example algo attached that pulls this quandl time series in and plots it. I'm pretty sure this isn't exactly what you're going for though - that data series is simple a time-series of the number of stocks making 52 week lows, rather than a listing of which stocks are at 52 week lows each day.

Do you have another data source with the ticker, date pairs of the stocks you want in the universe?

Clone Algorithm
38
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5355a28f605d510725e72b13
There was a runtime error.

Is it possible to add this to the "Help & API Docs" as this thread is not longer on the front page.

If there are any other things that you have rolled out and has not being put into the doces is it possible to update this so that the docs section would be a on stop place to learn about ALL of Quantopian functionality.

Is it possible to explain what this piece of code does in the above example.

symbols = [s.symbol for s in sids]  

Also will the following have the same effect.
context.count = len(sids)

Also is it possible to explain set a bit more. As I understand this takes pandas series (will other data structure be legal?) either as symbols or sids and return a list of security objects.

Also what is are the date formats acceptable in the data column by default. Can different date for formats be used as in the Fecher Docs?

Hi Suminda,

The documentation is now available in the Fetcher Section under "Using Fetcher to create a custom universe" and includes another example. The help docs are now up-to-date with our new features!

To answer your other question, this line of code

symbols = [s.symbol for s in sids]  

Is saying iterate through the list of sids and convert it to a list of symbols. So for example, [Security(24), Security(300)..] will become ["AAPL", "AA", ...] using Python list comprehension.

The next line,
context.count = len(symbols) is counting the number of symbols in the list and printing the value. This helps identify how many stocks are in the universe during each bar of the backtest.

You can specify the date format of your data using the date_format parameter in fetch_csv.

Best,
Alisa

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi,

Is there a way to disable the forward filling while using Fetcher to create a custom universe?
Right now, universe_func will forward fill if no new data occurs on a new day. Is it possible to disable this behavior and instead have no universe / not trade the new day if there is no data on that day. I am writing an algorithm to daytrade a list of stocks that I import from Fetcher, and right now if there is no stocks on a trading day, it will trade the same stocks from the previous day for two consecutive days, which is what I do not want.

Thanks

Hey TingTing,
There are a couple ways. First, you can add rows to the CSV you are importing with the missing dates. With those missing dates, add a stock you don't care about then set up a line to ignore that stock when you are going through the data variable containing all the stocks for that day.

You can also use your existing CSV and add a pre_func to the fetch_csv function that fills in the missing dates with a stock you don't care about. Then similar to the first method, you ignore that stock in handle_data. Here's an example of that. I used the example from "Using Fetcher to create a custom universe" here as a starting point. You can see in the code there is a pre_func where I fill in the missing dates with AAPL, then ignore AAPL in handle_data. You will probably have to slightly adjust the stop_forward_filling function I made for your own use.

Does this work for you?

Clone Algorithm
148
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 53b5ba5edef7800727203814
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Jessica, it'd be great if you could share the quandl search criteria or code with us. I can't seem to figure out how to get data for the stocks and not the S&P index.

Hi Thomas,

I grabbed the SP500 index constituent from this page: http://www.quandl.com/resources/useful-lists

Scroll down to the Stock Index Constituents header - they have a static snapshot as of 10/10/2013

Hope this is useful!
Best wishes,
Jess

Hi Jessica, I was trying to use your example of defining a custom universe using fetcher but I seem to be stuck out of the gate.

Looks like fetcher is correctly loading my symbols file from dropbox since the preview function returns the right results. However, it never seems to call the my_universe function nor the handle_data function. Any ideas? Thanks!

Clone Algorithm
5
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 54527e05cd0f2a09295ef240
There was a runtime error.

Hi Claus,

Thanks for posting!

What's happening is that fetcher looks for the 'date' column in your CSV file and fills your universe with the stocks in your CSV starting from that date. So in this case, because your CSV contained the date '1/1/02' and the backtester date was set to start sometime at 2013, it never had the chance to hit '1/1/02' so your universe was never filled with stocks. In addition, the earliest trading date we have is '1/3/02' so you'd have to change '1/1/02' to '1/3/02' and either start your backtest on that date or sometime before.

I've done that here:

import datetime  
import pandas as pd  
import numpy as np

def preview(df):  
    log.info(' \n %s ' % df.head(n=10))  
    df['date'] = '1/3/02'  
    return df

# Function for returning a set of SIDs from fetcher_data  
def my_universe(context, fetcher_data):  
    log.debug("in my_universe")  
    # Grab just the SIDs for the one sector:  
    #market = fetcher_data[fetcher_data['market_group'] == 'Equities']  
    #sids = set(market['sid'])  
    sids = set(fetcher_data['sid'])


    symbols = [s.symbol for s in sids]  
    context.count = len(symbols)  
    print "total universe size: {c}".format(c=context.count)  
    log.info("total universe size: {c}".format(c=context.count))  
    context.target_weight = 1.0/context.count  
    return sids

def initialize(context):  
    fetch_csv(  
       "https://dl.dropboxusercontent.com/u/12406727/symbols.csv",  
        pre_func=preview,  
        date_column='date',  
        date_format = '%m/%d/%y',  
        universe_func=my_universe)

    context.target_weight = 0.01  
    log.debug("done with initialize")  
def handle_data(context,data):  
    log.debug("we're in handle_data")  
    for stock in data:  
        print stock

        if 'price' in data[stock]:  
            order_target_percent(stock,context.target_weight)  
            print stock  
        else:  
            log.warn("No price for {s}".format(s=stock))  

I just changed your 'date' column to be '1/3/02' instead of '1/1/02' and set my backtester date to start at 1/3/02. Try it out and let us know how things work out!

Seong

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Seong, thanks for quick reply. Looks like I got this to work, although not until I ran a full backtest (vs just building the script).

Thanks again!

Claus

Could @TingTIng solve his problem also by comparing data[stock]['Date'] to the current (simulation) date and process only when that's true?

Hi, I tried to run this and seems like it's looping through all the stocks in the fetcher csv, not just the "Financials". How can I make sure the def my_universe function runs properly?
When I changed the handle_data to "print stock", I got a list of all the stocks.

Thanks,
Nikki

bump, im seeing the same behavior