Back to Community
Fetcher issues

trying to setup a portfolio example using quantopain fetcher, but am getting mysterious, non-informative errors with fetcher.

This builds and runs in daily mode, but for some reason skips the week june 15 though there is data for some of the securities during that week (example: SIXD)

when i try to run in minute mode I get a error telling me "Something went wrong. Sorry for the inconvenience. " Setting a breakpoint on the first line of handle_data doesnt help (debugger doesnt launch)

what am I missing?

Clone Algorithm
11
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 55aebdfb27f7880c6e013d1e
There was a runtime error.
10 responses

looks like the 'attach backtest' didn't work because of the error.

here is the source code. date ranges are 06/12/2015 to 07/15/2015, $1000 capital, minute mode.

import pandas as pd

def initialize(context):  
    # import the custom CSV data file  
    fetch_csv("https://docs.google.com/spreadsheets/d/1H8davFMCUhBg4ywZDg7uXYA3jlXepnMvkMl2FHBBmdc/pub?gid=0&single=true&output=csv",  
              pre_func=preview,  
              post_func=preview,  
              date_column='start_date',  
              universe_func=my_universe)  
    #set_benchmark(symbol('IWV'))  
    #context.IWV=symbol('IWV')  
    #context.IWV=symbol('SIXD')

# my_universe returns a set of securities that define your universe.  
def my_universe(context, fetcher_data):  
    # fetcher_data is the data resulting from the CSV file from fetcher.

    # set my_stocks to be every security in the fetcher_data  
    my_stocks = set(fetcher_data['sid'])

    # log the size of the universe for debugging  
    context.count = len(my_stocks)  
    print 'total universe size: {c}'.format(c=context.count)

    # return the securities we identified earlier  
    return my_stocks

# see a snapshot of your CSV for debugging  
def preview(df):  
    log.info('\n %s ' % df.head())  
    return df

def handle_data(context,data):  
    # Convert to EST timezone  
    exchange_time = pd.Timestamp(get_datetime()).tz_convert('US/Eastern')  
    log.info("handling data");  
    log.info(exchange_time);  
    for stock in data:  
        log.info(stock.symbol)  
        log.info(data[stock]['dt'].date())  
        log.info(data[stock]['dt'].date() == exchange_time.date())  

    # loop over the stocks in universe  
    for stock in data:  
      if(context.portfolio.positions[stock].amount == 0):  
          order_target_value(stock, 500)  
      #if data[stock]['dt'].date() == exchange_time.date():  
      #    log.info(stock)  
      #    order_target_value(stock, 500)  

I think I understand what's going on, but I'm not personally enough of a Python whiz to tell you the solution.

I turned the debugger on for the line

return my_stocks  

And then I stepped through from there. The debugger logs the return, which starts like this:

Returning from function with:  
set([0, Equity(10240, symbol='OSBC', asset_name='OLD SECOND BANCORP INC', exchange='NASDAQ GLOBAL SELECT MARKET', start_date=Timestamp('1993-11-16 00:00:00+0000', tz='UTC'), end_date=Timestamp('2015-07-21 00:00:00+0000', tz='UTC'), first_traded=None), Equity(45960,  

I believe the problem is that first entry in the set is '0' and that is blowing up the algo.

What you've got here is a repro case for a bug that I've seen in the logs, but not known what causes it. It was on my list of things to get fixed soon, and this will help fix it faster!

Until then, you need to modify your my_universe() function to remove that 0 from the set, and I bet it will work.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Adding on to Dan's point, you can remove sid(0) from the set by doing:

my_stocks = fetcher_data[fetcher_data['sid'] != 0]  

Also, I haven't seen Google Drive work successfully with Fetcher (maybe you found something new?). If you're having trouble reading the file, I'd recommend to use copy.com, it's a simple interface and allows you to publicly share the CSV with the algo.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thanks for the help,

@Alisa, it's not a security with a sid of zero, there's actually a set element of "0" being returned. I'm not a python guy but I lucked out on my first try:

my_stocks.remove(0)  

However now I get an error because the SidData doesn't have a price:

AttributeError: 'SIDData' object has no attribute 'price' There was a
runtime error on line 54.

So it appears there is some corrupted/invalid securities on your backend.... I will add some try/catch stuff to see what security is causing it.

Regarding using Google Drive for fetcher, I mentioned this solution on these forums, maybe a year ago. However when I tried this stuff yesterday I see google has made it easier. You just go to your google spreadsheet, choose "publish" from the dropdown, then choose csv. I highly suggest you give the google-spreadsheet ==> fetcher workflow a try, it's great! Just populate the spreadsheet and it's all ready for use in quantopian, no copying files around needed.

@Jason do you have a working sample for google drive... google-spreadsheet ==> fetcher workflow does historical prices in the docs can be use...against the the benchmark in quantopian??

@John: I think this thread is exactly what you need, it just doesn't work because (I think) some of the securities in my spreadsheet have invalid data on Quantopian's backend.

Here is the backtest (and code) with error handling for bad securities so it doesn't crash.

Clone Algorithm
11
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 55afee497888190c982b8af8
There was a runtime error.

@Jason.. Instead of using universe_function... Im using google docs to fetch historical prices w/c are not found.. on quantopian database is that possible? or we can some incompatibility issues... basically making it a substitute among for quandl.. if thats the word... ;) to be use... for benchmark against the index?

@Jason, we're fixing the issue with the element 0 being returned in fetcher. It happens if Fetcher can't read the security (ie the file has BF.A instead of BF_A). A fix is in the works.

And thanks for sharing the Google Doc workflow! They made is much easier to publish CSV's in the last year. We'll add it to the documentation.

@John keep in mind when you fetch external prices, your orders are still being traded and filled using Quantopian's database. You can use the external data as a signal, but the fill prices will be ones on our platform.

@Jason I tried putting in prices instead of stock symbol didnt work.... Im also curious about this fetch.csv locally in quantopian research can we also fetch it in the algo I havnt seen any samples on how it can be done...

jumping in late on this one but if anyone is still having trouble loading data from google drive let me know as it works for me