Back to Community
no XIV with zipline-live?

Newbie question: I'm trying to move my Q algo to zipline-live. I am able to run demo code with symbol('AAPL'), but not symbol('XIV') or symbol('VXX'):

asset = super(self.__class__, self).symbol(symbol_str)  

File "c:\python27\lib\site-packages\zipline\algorithm.py", line 1182, in symbol
"""Lookup an Equity by its ticker symbol.
File "c:\python27\lib\site-packages\zipline\algorithm.py", line 1212, in symbol
as_of_date=_lookup_date,
File "c:\python27\lib\site-packages\zipline\assets\assets.py", line 838, in lookup_symbol
return self._lookup_symbol_strict(symbol, as_of_date)
File "c:\python27\lib\site-packages\zipline\assets\assets.py", line 708, in _lookup_symbol_strict
raise SymbolNotFound(symbol=symbol)
zipline.errors.SymbolNotFound: Symbol 'XIV' was not found.

11 responses

you found us on slack, but posting it here for other who might also encounter this problem:

quandl does not have XIV included, but you can load it from google
first you should read this: http://www.zipline.io/bundles.html then
you can write your custom loader or take mine:
https://gist.github.com/tibkiss/2771ed5e80dc33e7a4d72462be1e6ae6 to

use splits & adjustments you need pandas-datareader==0.5.0.
You also need two patches on pandas-datareader:
https://github.com/pydata/pandas-datareader/pull/397/files
https://github.com/pydata/pandas-datareader/pull/399/files

Thanks, Tibor.

Would love to use this also. Where would you place the ingest_google.py file?

What else is needed to do besides running zipline ingest -b google_equities ?

IIRC zipline.io had all the information about setting up the new bundle, now its gone.

ingest_google.py goes to ~/.zipline/
you also need a ~/.zipline/extension.py with the following content

from zipline.data.bundles import register  
sys.path.append(os.path.dirname(__file__))  
from ingest_google import google_equities

symbols = ["SPY", "XIV"]  
register('google', google_equities(symbols))  

Once you did zipline ingest -b google you need to specify the same bundle (google) during your zipline-live run:
zipline run -b google

Thanks! That got me further (had to add import os,sys to the extension.py file also). But now I'm getting the same error I was getting with yahoo_equities:

Downloading Google pricing data: [##################------------------] 50%/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/zipline/data/us_equity_pricing.py:417: UserWarning: Ignoring 3 values because they are out of bounds for uint32: open high low close volume
Date
2017-07-31 NaN NaN NaN 246.77 65838659
winsorise_uint32(raw_data, invalid_data_behavior, 'volume', *OHLC)
Merging daily equity files: [####################################]
Downloading Yahoo adjustment data: [##################------------------] 50%
Traceback (most recent call last):
File "/Users/rrobinson/venv-zipline-live/bin/zipline", line 11, in
load_entry_point('zipline-live==1.1.0.2', 'console_scripts', 'zipline')()
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(
args, **kwargs)
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/zipline/
main.py", line 360, in ingest
show_progress,
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/zipline/data/bundles/core.py", line 451, in ingest
pth.data_path([name, timestr], environ=environ),
File "/Users/rrobinson/.zipline/ingest_google.py", line 110, in ingest
session=session,
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/pandas_datareader/data.py", line 126, in DataReader
session=session).read()
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/pandas_datareader/yahoo/actions.py", line 37, in read
splits['value'] = splits.apply(lambda x: 1/eval(x['value']), axis=1) # noqa
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/pandas/core/frame.py", line 2357, in _
setitem__
self.set_item(key, value)
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/pandas/core/frame.py", line 2422, in _set_item
self._ensure_valid_index(value)
File "/Users/rrobinson/venv-zipline-live/lib/python2.7/site-packages/pandas/core/frame.py", line 2404, in _ensure
valid_index
raise ValueError('Cannot set a frame with no defined index '
ValueError: Cannot set a frame with no defined index and a value that cannot be converted to a Series

You're hitting a pandas-datareader bug: there are no splits & adjustments returned for your instrument.
Solution is here: https://github.com/pydata/pandas-datareader/pull/399/files

Nailed it! Perfect - working now - thanks again!

Glad to hear. Feel free to join our slack channel for more immediate feedback: http://www.zipline-live.io/contact

Thanks Tibor for your sharing.

I added extension.py and ingest_google.py, what should I do next for the actual ingest?

Do I add all the symbols I want to include in symbols tuple in extension.py?
Then I just run extension.py for the bundle ingest?

right now when I run zipline ingest -b google it says "no bundle registered with name google"

Hi Bill,

I am having the same error: "no bundle registered with name google"; did you ever figure it out?

Ok I fixed the problem that Bill and I had, you need to add

include os
include sys

to the extension.py ,

but now I am getting another error:

Downloading Google pricing data: [############------------------------] 33%

Traceback (most recent call last):
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/zipline/utils/cache.py", line 259, in getitem
with open(self._keypath(key), 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/Derek/.zipline/data/google/.cache/SPY-ohlcv'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/Derek/.zipline/ingest_google.py", line 54, in pricing_iter
df = cache[path]
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/zipline/utils/cache.py", line 264, in __getitem
_
raise KeyError(key)
KeyError: 'SPY-ohlcv'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/Derek/venv-zipline/bin/zipline", line 11, in
sys.exit(main())
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/zipline/
main_.py", line 331, in ingest
show_progress,
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/zipline/data/bundles/core.py", line 451, in ingest
pth.data_path([name, timestr], environ=environ),
File "/Users/Derek/.zipline/ingest_google.py", line 85, in ingest
daily_bar_writer.write(_pricing_iter(), show_progress=show_progress)
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/zipline/data/us_equity_pricing.py", line 257, in write
return self._write_internal(it, assets)
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/zipline/data/us_equity_pricing.py", line 319, in _write_internal
for asset_id, table in iterator:
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/click/_termui_impl.py", line 259, in next
rv = next(self.iter)
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/zipline/data/us_equity_pricing.py", line 248, in
(sid, self.to_ctable(df, invalid_data_behavior))
File "/Users/Derek/.zipline/ingest_google.py", line 61, in _pricing_iter
session=session,
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/pandas_datareader/data.py", line 137, in DataReader
session=session).read()
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/pandas_datareader/base.py", line 181, in read
params=self._get_params(self.symbols))
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/pandas_datareader/base.py", line 79, in _read_one_data
out = self._read_url_as_StringIO(url, params=params)
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/pandas_datareader/base.py", line 90, in _read_url_as_StringIO
response = self._get_response(url, params=params)
File "/Users/Derek/venv-zipline/lib/python3.6/site-packages/pandas_datareader/base.py", line 139, in _get_response
raise RemoteDataError('Unable to read URL: {0}'.format(url))
pandas_datareader._utils.RemoteDataError: Unable to read URL: http://www.google.com/finance/historical?q=SPY&startdate=Jan+02%2C+1990&enddate=Jul+21%2C+2018&output=csv