Back to Community
Fetching symbols, not found?

Hello,

I'm trying to use Quantopians backtester for a pre-calculated list of stocks. What I want to do is quite similar to what's done in the example "Using Fetcher to create a custom universe" at https://www.quantopian.com/help. I use that example to test my list. However, the code fails for some symbols. Output from loading my list:

0 NI -1 2/6/2015  
1 HME -1 2/6/2015  
2 CBM 1 2/6/2015  
3 AI -1 2/4/2015  
4 YOKU 1 2/4/2015  
2015-02-04 00:00:00+00:00 -1 0  
2015-02-04 00:00:00+00:00 1 Security(40562 [YOKU])  
2015-02-06 00:00:00+00:00 -1 Security(5310 [NI])  
2015-02-06 00:00:00+00:00 -1 Security(11654 [HME])  
2015-02-06 00:00:00+00:00 1 Security(1297 [CBM])  

As can be seen only four of the symbols could be identified. "AI" could not. The same thing happens for other tickers as well, "DAL" is another example.

Why does the symbol identification fail?

6 responses

I could mention that my own theory is that it's because of this:

All securities have a unique security id (SID) in our system. Since symbols may be reused among exchanges, this prevents any confusion and verifies you are calling the desired security. You can use our sid method to look up a security by its id, symbol, or name.

But the documentation only mentions the problem, without offering a solution - how do I set exchange? (NYSE in this case, in Mathematica I would write "NYSE:DAL" as an example.)

Tickers are reused on exchanges as companies go private, bankrupt, merge etc. In a simple algo, if you try to backtest using "symbol('AI')", you get an error because the symbol has been used by two companies. The backtester requires you to specify which stock you want to trade.

To fix the error, add the following to initialize() to trade Arlington Asset Investment Corp.

 set_symbol_lookup_date('2004-01-01')  
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

No, that won't fix the error. From the documentation:

Fetcher maps the symbol to the Quantopian security id (sid).

It's this mapping that fails, and it seems to me like that internal mapping isn't affected by set_symbol_lookup_date. I provide an example of this where I have replaced two stocks from Quantopian's example with EDS and AI, who suffer from this problem.

The log output shows:

1970-01-01preview:18INFO symbol start date stock_score
0 AI 2/13/12 11.7
1 EDS 2/13/12 15.8
2 FDX 2/14/12 12.1
3 M 2/16/12 14.3
1970-01-01preview:18INFO stock_score sid
dt
2012-02-13 00:00:00+00:00 11.7 0
2012-02-13 00:00:00+00:00 15.8 0
2012-02-14 00:00:00+00:00 12.1 Security(2765 [FDX])
2012-02-16 00:00:00+00:00 14.3 Security(2754 [M])

As can be seen, the mapping returns 0 for both EDS and AI instead of the right SID.

EDIT: In case someone has the same problem and finds this topic, my workaround is to add the following in the post_func, it removes tickers that weren't mapped properly:

  df = df[df.sid != 0]  
Clone Algorithm
3
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
import pandas as pd

def initialize(context):
    set_symbol_lookup_date('2015-01-01')
    # import the custom CSV data file
    fetch_csv("https://dl.dropboxusercontent.com/u/124064072/possibleBug.csv",
              pre_func=preview,
              post_func=preview,
              date_column='start date',
              universe_func=my_universe)

# my_universe returns a set of securities that define your universe.
def my_universe(context, fetcher_data):
    pass

# see a snapshot of your CSV for debugging
def preview(df):
    log.info(' %s ' % df)
    return df

def handle_data(context,data):
    pass
There was a runtime error.

Digging into this some more, those tickers are failing to map to your algo because of a bug in our backend. If a symbol is re-used (like 'AI') then it doesn't get mapped properly and is filtered out from your algo universe. If the ticker is unique, then it will work normally, as expected.

In the meantime, you have a good workaround in the post_func.

Ok, good to get that confirmed :)

I found a temporary fix for this. Use this as your pre_func

def pre_func(df):  
    df['sid'] = df['symbol']  
    return df  

It works for most symbols, FB included. Some symbols like TWTR still doesn't work for unknown reason.