Back to Community
Fetch short interest data for batch of Stocks and universe stocks

I am trying to fetch short interest data using fetcher and quandl.

My issues:
1. I get an unknown error in line 31
2. I don’t manage to retrieve it for list of stocks defined in context.stocks. If I use 'for' loop with fetcher I get an error it can be called only 5 times. and I want to fethc it for more than 5 stocks. I couldnt find a way in quandl or other to fetch a list all together of short interest data. any idea ?
2. how can I use universe and run fetcher on the universe selected stocks and/or on fundamentals filter ?

Clone Algorithm
12
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
def time_lag(df):
    # data from the exchanges is delayed by 8 business days
    # using pandas' tshift to lag the index by 8 business days
    df = df.tshift(8, freq='b')
    return df

def initialize(context):
    
    # set the second decile of stocks by dollar volume traded
    # as our universe. This universe updates quarterly.
    
    #set_universe(universe.DollarVolumeUniverse(80, 90))
    
    context.stocks = [sid(24),sid(46631), sid(7671)]

    # we run with a 50k initial cash balance, and we expect about 50 stocks in each
    # decile because the 10% universe above has about 500 members.
    # To limit total leverage to about 2x, we limit individual positions to
    # $1k notional value.
    context.position_limit = 1000
    context.top = None
    context.bottom = None
    
    for stock in context.stocks:
        url = 'https://www.quandl.com/api/v1/datasets/SI/{0}_SI.csv'
        url = url.format(stock.symbol)
        #fetch_csv(
        #    url,
        #    date_column='Settlement Date',
        #    date_format='%d/%m/%Y',
        #    post_func=time_lag
        #)

    
def handle_data(context, data):
    # get a map of sid -> days to cover. get_dtoc is defined below
    # it has a few guards on accessing the days to cover for the stock
    # codenote: this is a python dictionary comprehension.
    dtoc_map = {int(sid): get_dtoc(value) for sid, value in data.iteritems()}
    
    # cull the sids to the stocks in data AND with days to cover data.
    # codenote: itertools is full of win, google it.
    context.sids = data.keys()
    filtered = filter(lambda x: dtoc_map[x] is not None, context.sids)
    
    # calculate the count of items in a decile and plot it with record
    decile = len(filtered)/10
    record(decile_count=decile)
    
    # only proceed if we have a non-zero decile count
    if decile > 0:
        
        # rank the sids by days to cover, lowest to highest
        ranking = sorted(filtered, key=lambda x: dtoc_map[x])
        
        # find all the sids with a days to cover of one. this is interesting
        # because the NASDAQ reports the data with a floor of 1.
        ones = filter(lambda x: data[x]['days_to_cover'] == 1, ranking)
        record(dtoc_is_one_count=len(ones))
        
        # record the maximum days to cover (i.e. greatest short interest)
        record(highest_dtoc=data[ranking[-1]]['days_to_cover'])
        
        # slice the ranking to get the top and the bottom
        # codenote: python supports negative indexes, which count back 
        # from the end of the list.
        top = set(ranking[:decile])
        bottom = set(ranking[-1*decile:])
        
        if context.top != top or context.bottom != bottom:
            # turnover is defined as the number of stocks entering or leaving the
            # decile. it is useful as a way to guage the stability of our ranking.
            # turnover for top and bottom is recorded into a time-series plot
            record(bottom_turnover=calculate_turnover(bottom, context.bottom))
            record(top_turnover=calculate_turnover(top, context.top))
            
            log.info("Rebalancing on {d}".format(d=data[data.keys()[0]].datetime))
            context.top =  top
            context.bottom = bottom
            rebalance(context, data)

def calculate_turnover(curr, orig):
    """
    turnover reflects the churn in the decile from curr to orig.
    """
    if not curr or not orig:
        return 0.0
    
    # sets overload the behavior of +/- operators
    adds = curr - orig
    drops = orig - curr
    change = float(len(adds) + len(drops)) / len(orig) * 100.0
    return change
    
def get_dtoc(siddata):
    """
    The source data for days to cover has holes in the coverage. 
    Some stocks are not covered, and others have missing days. Here
    we have a convenience function to guard against missing days_to_cover properties.
    """
    if 'Days to Cover' in siddata:
        return siddata.days_to_cover
    else:
        return None    
 
def rebalance(context, data):
    """
    Rebalance the portfolio based on contents of the current portfolio,
    the top decile, and the bottom decile. top, bottom, and portfolio 
    are all expected to be in context. 

    The rebalance strategy is to calculate a difference between the
    current position and a desired position.
    It then places an order to make that adjustment.
    This should serve to avoid needless portfolio churn and 
    transaction fees.
    """
    
    for sid in context.sids:
        if sid not in context.top and sid not in context.bottom:
            order_target_value(sid, 0)
        
        if sid in context.top:
            order_target_value(sid, context.position_limit)
        if sid in context.bottom:
            order_target_value(sid, -1*context.position_limit)
            
There was a runtime error.
3 responses

You'll probably need to copy and paste your code or comment out the offending line so your backtest can finish. As of right now, I can't see any code to help you.

Please see the re-attached backtest commenting out the fetch lines. anyone has an ideas how to solve this issues ?

The algorithm does not allow you to do requests to external URLs. Use Self-Serve Data for this: https://www.quantopian.com/posts/short-interest-data-pipeline-for-quandl-dataset