Back to Community
Difficulty getting created columns out of 'fetched' data

Having difficulty applying transform columns to a dataframe of fetch'd data. The following is a minimal sample:

import numpy as np  
import pandas as pd  
import talib


url ='http://www.quandl.com/api/v1/datasets/YAHOO/INDEX_VIX.csv?request_source=python&request_version=2&sort_order=asc&trim_start=2002-01-01'

def fix_vix(df):  
    df  = df.rename(columns={'Close': 'price'})  
    df.fillna(method='ffill')  
    df['symbol'] = 'vix'  
    df['sma'] = talib.SMA(df['price'], timeperiod=5)  
    df['ema'] = talib.EMA(df['price'], timeperiod=5)  
    df = df[['price', 'sma', 'ema', 'symbol', 'sid']]  
    return df

def initialize(context):  
    fetch_csv(url, symbol='vix', date_column='Date', usecols=['Close'], post_func=fix_vix)  
    # The following results in an error, irrespective of the use of the 'symbol' param to fetch_csv, or constructing  
    # a 'symbol' column with the value 'vix' in the post_func.  
    # context.vix = symbol('vix')  
def handle_data(context, data):  
    if data is None:  
            return

    log.info(data['vix'])  
    log.info('price: %f' % data['vix'].price)  
    log.info('price: %f' % data['vix'].ema)  
    return  

The result is:

30  Error   Runtime exception: AttributeError: 'SIDData' object has no attribute 'ema'  

Despite the SiDData bits being logged clearly showing the 'ema' and 'sma' columns.

2011-01-04handle_data:29INFOSIDData({'ema': 17.448922258336495, '_sid': 'vix', '_freqstr': None, '_initial_len': 3, 'price': 17.38, 'sid': 'vix', 'source_id': 'PandasRequestsCSVb3e026392ead2d14e1bbc7c21b5e8ea4', 'dt': Timestamp('2011-01-04 00:00:00+0000', tz='UTC'), 'type': 9, 'symbol': 'vix', 'sma': 17.508000000000045})  

As commented in the code, I've also been unable to get the data into, say, context.vix by the tricks in the docs (at least as well as I did/didn't understand them).

Suggestions?

6 responses

Hello Michael,

It's long time since I looked at 'fetcher' so I can't explain what is happening here.

P.

Clone Algorithm
1
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
import numpy as np  
import pandas as pd  
import talib


url ='http://www.quandl.com/api/v1/datasets/YAHOO/INDEX_VIX.csv?request_source=python&request_version=2&sort_order=asc&trim_start=2002-01-01'

def fix_vix(df):
    df  = df.rename(columns={'Close': 'price'})  
    df.fillna(method='ffill')  
    df['symbol'] = 'vix'  
    df['sma'] = talib.SMA(df['price'], timeperiod=5)  
    df['ema'] = talib.EMA(df['price'], timeperiod=5)  
    df = df[['price', 'sma', 'ema', 'symbol', 'sid']]  
    return df

def initialize(context):
    context.sid = sid(8554)
    fetch_csv(url, symbol='vix', date_column='Date', usecols=['Close'], post_func=fix_vix)  
    # The following results in an error, irrespective of the use of the 'symbol' param to
    # fetch_csv, or constructing  
    # a 'symbol' column with the value 'vix' in the post_func.  
    # context.vix = symbol('vix')
    
def handle_data(context, data):  
    #if data is None:  
    #        return
    
    if 'ema' in data['vix']:    
        #log.info(data['vix'])  
        #log.info('price: %f' % data['vix'].price)  
        #log.info('price: %f' % data['vix'].ema)
        record(Price=data['vix']['price'], EMA=data['vix']['ema'])
    return  
There was a runtime error.

Hi Michael,

Fetcher is definitely a tricky subject. In this case, you were doing EVERYTHING RIGHT, except for two very, very small things:

1) checking that 'ema' existed in data['vix']. You need to do this because FETCHER only fills in 'ema' when the date in your CSV file matches up with the dates of your backtest. This is why we can do a universe call in fetcher, since the dates are used to align data.
2) you didn't set a STOCK in your universe. Because 'vix' isn't really a tradable security, you need something like context.stock = symbol('AAPL') just so that 'data' fills with some sort of security pricing for the duration of the backtest. That's as simple as setting context.stock to a valid security in 'initialize' and forgetting about for the duration of your backtest.

Check out my attached backtest. It's basically your code with just like two changes. Nice job and let me know if you have any questions

Clone Algorithm
9
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
import numpy as np  
import pandas as pd  
import talib


url ='http://www.quandl.com/api/v1/datasets/YAHOO/INDEX_VIX.csv?request_source=python&request_version=2&sort_order=asc&trim_start=2002-01-01'

def fix_vix(df):  
    df  = df.rename(columns={'Close': 'price'})  
    df.fillna(method='ffill')  
    df['symbol'] = 'vix'  
    df['sma'] = talib.SMA(df['price'], timeperiod=5)  
    df['ema'] = talib.EMA(df['price'], timeperiod=5)  
    df = df[['price', 'sma', 'ema', 'symbol', 'sid']]  
    return df

def initialize(context):  
    """
    Because you don't have a 'security' set in your universe.
    You need to set some stock in order that your algorithm 'data' fills with something
    for the time that you're running your backtest.
    """
    context.stock = sid(24)
    fetch_csv(url, symbol='vix', date_column='Date', usecols=['Close'], post_func=fix_vix)  

def handle_data(context, data):  
    """
    Because Fetcher only fills in data for dates where your CSV file
    matches up with the CURRENT date in your backtest, you need to make sure
    that you actually have data for 'vix'
    """
    if 'ema' in data['vix'] and 'price' in data['vix']:        
        log.info(data['vix'])  
        log.info('price: %f' % data['vix'].price)  
        log.info('price: %f' % data['vix'].ema)  
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thanks Seong Lee. Appreciate it.

Sorry to drag up an old thread, but when I clone this it no longer works, I get:
24 Error Error loading csv data from https://www.quandl.com/api/v1/datasets/YAHOO/INDEX_VIX.csv?request_source=python&request_version=2&sort_order=asc&trim_start=2002-01-01: 404 - Not Found

When I plug the URL into a browser the correct csv comes up, so not sure why its showing "Not Found"?

Quandl was having some issues with their API, it's working now. Try running the algo again

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Working great now, thanks!