zipline compatible script

I'm trying to write a script that's compatable with zipline for offline testing.

this looks very tedious, as it appears the quantopian script is actually the body of a class deriving from the zipline.TradingAlgorithm class

could anyone suggest an elegant way of doing this? I'm not a python person so I'm probably messing up.

edit: i ended up writing a framework here: https://github.com/Novaleaf/QuantShim but in the future I'll be dropping zipline due to its differences/defects.

23 responses

also i see required variables such as data_frequency are not exposed to the quantopian script. so it seems not possible .....

Use the following after checking out the code from github:

# -*- coding: utf-8 -*-
"""
Created on Thu Mar 20 12:17:18 2014

@author: Suminda
"""

# Delete unused imports once algo is finished

from datetime import datetime

from zipline import *
from zipline.algorithm import *
from zipline.api import *
from zipline.api import *
from zipline.data import *
from zipline.errors import *
from zipline.finance import *
from zipline.gens import *
from zipline.protocol import *
from zipline.sources import *
from zipline.transforms import *
from zipline.utils import *
from zipline.version import *

class commission:
def __init__(self):
self.PerShare = PerShare
self.PerTrade = PerTrade

commission = commission()

class slippage:
def __init__(self):
self.VolumeShareSlippage = VolumeShareSlippage
self.FixedSlippage = FixedSlippage

slippage = slippage()

# Logging. Following imports are not approved in Quantopian
####################################################################################

import logbook
import sys

log_format = "{record.extra[algo_dt]}  {record.message}"

zipline_logging = logbook.NestedSetup([
logbook.StreamHandler(sys.stdout, level=logbook.INFO, format_string=log_format),
logbook.StreamHandler(sys.stdout, level=logbook.DEBUG, format_string=log_format),
logbook.StreamHandler(sys.stdout, level=logbook.WARNING, format_string=log_format),
logbook.StreamHandler(sys.stdout, level=logbook.NOTICE, format_string=log_format),
logbook.StreamHandler(sys.stderr, level=logbook.ERROR, format_string=log_format),
])
zipline_logging.push_application()

log = logbook.Logger('Main Logger')

# Cut and past between the ### line below to Quantopian. Change symbols to sids. E.g. 'AAPL' to sid(24)
####################################################################################

# Place Quantopian approved imports here

def initialize(context):
pass

def handle_date(context, data):
order(symbol('AAPL'), 10)

####################################################################################
# Cut and past between the ### line above to Quantopian. Change symbols to sids. E.g. 'AAPL' to sid(24)

if __name__ == '__main__':
import pylab as pl
# Any other non approved imports to make things work or to plot results
start = datetime(2008, 1, 1, 0, 0, 0, 0, pytz.utc)
end = datetime(2014, 1, 1, 0, 0, 0, 0, pytz.utc)
# Reference any stocks data you need below
data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start,
end=end)
data = data.dropna()

algo = TradingAlgorithm(initialize=initialize,
handle_data=handle_date)

results = algo.run(data)
results.portfolio_value.plot()

pl.show()


You need to add following patch to api.py file:

class sid:
"""Default sid lookup for any source that directly maps the
sid a symbol.
"""
def __init__(self, sid_to_symbol):
"""Initialise the maping of sids to symbols
:Arguments:
sid_to_symbol : dict
Contains the maping
"""
self.sid_to_symbol = sid_to_symbol
def __call__(self, sid_num):
"""Retrive the symbol corresponding to the sid
:Arguments:
sid_num : int
sid for which the symbol needs to be retrived
:Returns:
Mapped symbol
"""
return self.sid_to_symbol[sid_num]

__all__ = [
'sid',
'symbol',
'slippage',
'commission',
'math_utils',
'batch_transform',
'FixedSlippage',
'VolumeShareSlippage'
]



Following improvements are needed:

• symbol and sid need to return a security object. Implication is that you cannot call any of the security objects methods as this is represented as a string.

Hope the above helps.

if you want to save the results use replacing last part of the file above with this.

####################################################################################
# Cut and past between the ### line above to Quantopian. Change symbols to sids. E.g. 'AAPL' to sid(24)

if __name__ == '__main__':
import pylab as pl
from pandas import ExcelWriter
import matplotlib.pyplot as plt

# Any other non approved imports to make things work or to plot results

start = datetime(1998, 12, 31, 0, 0, 0, 0, pytz.utc)
end = datetime(2014, 3, 22, 0, 0, 0, 0, pytz.utc)

# Reference any stocks data you need below
data = load_from_yahoo(stocks=['SPY', 'XLY', 'XLP', 'XLE', 'XLF', 'XLV', 'XLI', 'XLB', 'XLK', 'XLU'], indexes={}, start=start,
end=end)
data = data.dropna()
algo = TradingAlgorithm(initialize=initialize,
handle_data=handle_data, capital_base=10000, instant_fill=True)

results = algo.run(data)

results.to_csv('results.csv')


NB: This does not work with Zipline version 0.6 so you have to get the latest from Github.

Complete api.py file is below so you might not make a mistake:

#
# Copyright 2014 Quantopian, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Note that part of the API is implemented in TradingAlgorithm as
# methods (e.g. order). These are added to this namespace via the
# decorator api_methods inside of algorithm.py.

import zipline
from .finance import (commission, slippage)
from .utils import math_utils

from zipline.finance.slippage import (
FixedSlippage,
VolumeShareSlippage,
)

batch_transform = zipline.transforms.BatchTransform

def symbol(symbol_str, as_of_date=None):
"""Default symbol lookup for any source that directly maps the
symbol to the identifier (e.g. yahoo finance).

Keyword argument as_of_date is ignored.
"""
return symbol_str

class sid:
"""Default sid lookup for any source that directly maps the
sid a symbol.
"""
def __init__(self, sid_to_symbol):
"""Initialise the maping of sids to symbols
:Arguments:
sid_to_symbol : dict
Contains the maping
"""
self.sid_to_symbol = sid_to_symbol
def __call__(self, sid_num):
"""Retrive the symbol corresponding to the sid
:Arguments:
sid_num : int
sid for which the symbol needs to be retrived
:Returns:
Mapped symbol
"""
return self.sid_to_symbol[sid_num]

__all__ = [
'sid',
'symbol',
'slippage',
'commission',
'math_utils',
'batch_transform',
'FixedSlippage',
'VolumeShareSlippage'
]



If you want to use sid you have do the following at the top of your script file:

sid = sid({24 : 'AAPL'})


then you can use sid as usual as long you give the mapping for all the symbols you like above.

Another important thing to remember is that you have to have the Yahoo ticker list for all the tickers you would be using. (This appears towards the bottom of the file.)

data = load_from_yahoo(stocks=['AAPL'], indexes={}, start=start,
end=end)


thanks Suminda! I actually spent the last 3 hours writing a framework that should work in both places. I just got it working in quantopian, so now I need to get zipline operational to test with. after that, I'll make a github project for it.

the main difference with your code is i'm a python noob so I didn't think about wild-card importing, and i wrote shims for things, plus a new "Framework" class as a facade to abstract all the platform shims/glue away from the user's algorithm.

though i'm doing this for use in visual studio, so it's probably not very 'pythonic'

anyway, i'll post it once i get it tested under zipline.

trying to read your code Suminda... maybe too pythonic for me :P like i don't understand

sid = sid({24 : 'AAPL'})


wouldn't I want to use the context.mySid=sid(24) ?

i also like how you pass the initialize and handle_data methods to the framework. i'll have to do that too.

In you python script you have to use

... sid(24)


but Zipline does not know about sids so you have give a mapping before this will work. So when you define the sid object you have to pass the mapping to its constructor.

yeah i'm running into that problem now. also i see that the yahoo data loader is returning the data as a single dataframe. pretty disappointing to see all these discrepancies.

i guess it's working now.. no need to do any code changes to switch to/from quantopian.

fyi i circumvented the whole sid problem by just using sid to be a string on zipline, and int on quantopian. everything still works.

i'll put this on github in the morning. honestly going through all this code makes me either hate zipline's codebase, or python in general. not sure which. I probably won't continue improving this. it's just too retarded how the runtimes are so different.

Thanks for all this Suminda! It will be a big help for setting up a decent local env.

actually not sure there's interest in a cross-plat framework anyway. so I'll paste my current framework below. on this github will be my continued work, but i'll be abandoning zipline use because of it's various differences/defects.

current code that works on both is:


# Import the libraries we will use here
import datetime
import pytz
import math
import numpy
import pandas
import scipy
import scipy.stats
import zipline

#from zipline import *
#from zipline.algorithm import *
##from zipline.api import *
#from zipline.data import *
#from zipline.errors import *
#from zipline.finance import *
#from zipline.gens import *
#from zipline.protocol import *
#from zipline.sources import *
#from zipline.transforms import *
#from zipline.utils import *
#from zipline.version import *

#global constants
global true
true = 1
global false
false = 0

#quantopian shims
class Shims():
class Context():
def __init__(this , portfolio = zipline.protocol.Portfolio()): #, tradingAlgorithm = zipline.TradingAlgorithm()):
this.portfolio = portfolio
#this.tradingAlgorithm = tradingAlgorithm
pass
pass

class _Logger():
'''shim for exposing the same logging definitions to visualstudio intelisence'''
#def __init__(this):
#   pass

def error(this, message):
print("{0} !!ERROR!! {1}".format(this.framework.get_datetime(),message))
pass
def info(this, message):
print("{0} info {1}".format(this.framework.get_datetime(),message))
pass
def warn(this, message):
print("{0} WARN {1}".format(this.framework.get_datetime(),message))
pass
def debug(this, message):
print("{0} debug {1}".format(this.framework.get_datetime(),message))
pass
pass

class _TradingAlgorithm_QuantopianShim:
'''shim of zipline.TradingAlgorithm for use on quantopian '''
def __init__(this):
#this.logger = Shims._Logger()
#this.logger = log
pass

def order(this,sid,amount,limit_price=None, stop_price=None):
'''
Places an order for the specified security of the specified number of shares. Order type is inferred from the parameters used. If only sid and amount are used as parameters, the order is placed as a market order.
Parameters
sid: A security object.
amount: The integer amount of shares. Positive means buy, negative means sell.
limit_price: (optional) The price at which the limit order becomes active. If used with stop_price, the price where the limit order becomes active after stop_price is reached.
stop_price: (optional) The price at which the order converts to a market order. If used with limit_price, the price where the order converts to a limit order.
Returns
An order id.
'''
sec = this.context.framework.targetedSecurities[sid]
#this.logger.info(sec)
order(sec,amount,limit_price,stop_price)
pass

def order_percent(self, sid, percent, limit_price=None, stop_price=None):
"""
Place an order in the specified security corresponding to the given
percent of the current portfolio value.

Note that percent must expressed as a decimal (0.50 means 50\%).
"""
value = self.context.portfolio.portfolio_value * percent
return self.order_value(sid, value, limit_price, stop_price)

def order_target(self, sid, target, limit_price=None, stop_price=None):
"""
Place an order to adjust a position to a target number of shares. If
the position doesn't already exist, this is equivalent to placing a new
order. If the position does exist, this is equivalent to placing an
order for the difference between the target number of shares and the
current number of shares.
"""
if sid in self.context.portfolio.positions:
current_position = self.context.portfolio.positions[sid].amount
req_shares = target - current_position
return self.order(sid, req_shares, limit_price, stop_price)
else:
return self.order(sid, target, limit_price, stop_price)

def order_target_value(self, sid, target, limit_price=None,
stop_price=None):
"""
Place an order to adjust a position to a target value. If
the position doesn't already exist, this is equivalent to placing a new
order. If the position does exist, this is equivalent to placing an
order for the difference between the target value and the
current value.
"""
if sid in self.context.portfolio.positions:
current_position = self.context.portfolio.positions[sid].amount
current_price = self.context.portfolio.positions[sid].last_sale_price
current_value = current_position * current_price
req_value = target - current_value
return self.order_value(sid, req_value, limit_price, stop_price)
else:
return self.order_value(sid, target, limit_price, stop_price)

def order_target_percent(self, sid, target, limit_price=None,
stop_price=None):
"""
Place an order to adjust a position to a target percent of the
current portfolio value. If the position doesn't already exist, this is
equivalent to placing a new order. If the position does exist, this is
equivalent to placing an order for the difference between the target
percent and the current percent.

Note that target must expressed as a decimal (0.50 means 50\%).
"""
if sid in self.context.portfolio.positions:
current_position = self.context.portfolio.positions[sid].amount
current_price = self.context.portfolio.positions[sid].last_sale_price
current_value = current_position * current_price
else:
current_value = 0
target_value = self.context.portfolio.portfolio_value * target

req_value = target_value - current_value
return self.order_value(sid, req_value, limit_price, stop_price)

pass

class _TradingAlgorithm_ZiplineShim(zipline.TradingAlgorithm):
'''auto-generates a context to use'''
def initialize(this):
#delay initialize until start of first handle-data, so our portfolio object is available
#this.__isInitialized = false;
this.context = Shims.Context()
this.context.tradingAlgorithm = this
#this.context.portfolio = this.portfolio
pass

def handle_data(this,data):
this.context.portfolio = this.portfolio
#if not this.__isInitialized:
#    this.__isInitialized=true
#    this.context.portfolio=this.portfolio
this.context.framework.update_timestep(data)
pass
pass

class FrameworkBase():
def __init__(this, context, isOffline):
this.isOffline = isOffline
this.context = context
this.tradingAlgorithm = Shims._TradingAlgorithm_QuantopianShim() #prepopulate to allow intelisence
this.tradingAlgorithm = context.tradingAlgorithm
this.targetedSecurities = {}
this._targetedSecurityIds=[0]
del this._targetedSecurityIds[:]
if is_offline_Zipline:
this.logger = Shims._Logger()
this.logger.framework = this
else:
this.logger = log
pass
def initialize(this):
#do init here
if this.isOffline:
#passed to the run method
this._offlineZiplineData = this.abstract_loadDataOffline_DataFrames()
this._targetedSecurityIds = list(this._offlineZiplineData.columns.values)
else:
this._targetedSecurityIds = [sec.sid for sec in this.abstract_loadDataOnline_SecArray()]
#this.tradingAlgorithm.logger.info(len(this.securityIds))
#this.tradingAlgorithm.logger.info(this.securityIds)
pass

def abstract_loadDataOffline_DataFrames(this):
'''return a pandas dataframes of securities, ex: using the zipline.utils.factory.load_from_yahoo method
these will be indexed in .securityId's for you to lookup in your abstract_handle_data(data)'''
return pandas.DataFrame()
pass
def abstract_loadDataOnline_SecArray(this):
'''return an array of securities, ex: [sid(123)]
these will be indexed in .securityId's for you to lookup in your abstract_handle_data(data)'''
return []
pass

def abstract_update_timestep_handle_data(this,data=pandas.DataFrame()):

'''find the securities you loaded by .securityId.
if the security isn't present in data
, it's temporally unavailable (not yet listed or already removed from the exchange)
'''
pass

def update_timestep(this,data):
'''invoked by the tradingAlgorithm shim every update.  internally we will call abstract_update_timestep_handle_data()'''
this.targetedSecurities.clear()
for qsec in data:
if not this.isOffline:
sid = qsec.sid
else:
#if offline zipline, qsec is a string ex: "SPY"
sid = qsec;
qsec = data[qsec]

if len(this._targetedSecurityIds)==0 or this._targetedSecurityIds.index(sid)>=0:
this.targetedSecurities[sid]=qsec
this.data = data
this.abstract_update_timestep_handle_data(data)
pass

def get_datetime(this):
if is_offline_Zipline:
if len(this.targetedSecurities)==0:
return datetime.datetime.fromtimestamp(0,pytz.UTC)
else:
return this.targetedSecurities.values()[0].datetime
else:
return get_datetime()
pass

#entrypoints
def initialize(context=Shims.Context()):
'''initialize method used when running on quantopian'''
context.tradingAlgorithm = Shims._TradingAlgorithm_QuantopianShim()
context.tradingAlgorithm.context = context
context.framework = constructFramework(context,false)
context.framework.initialize()

pass

def handle_data(context=Shims.Context(),data=pandas.DataFrame()):
'''update method run every timestep on quantopian'''
context.framework.update_timestep(data)
pass

global constructFramework
def initalize_zipline():
'''initialize method run when using zipline'''
tradingAlgorithm = Shims._TradingAlgorithm_ZiplineShim()
context = tradingAlgorithm.context;
context.framework = constructFramework(context,true)
context.framework.initialize()
tradingAlgorithm.run(context.framework._offlineZiplineData)
pass

##############  CROSS PLATFORM USERCODE BELOW.   EDIT BELOW THIS LINE

class ExampleAlgo(FrameworkBase):
def abstract_loadDataOffline_DataFrames(this):
'''only called when offline (zipline)'''
# Load data
start = datetime.datetime(2002, 1, 4, 0, 0, 0, 0, pytz.utc)
end = datetime.datetime(2002, 3, 1, 0, 0, 0, 0, pytz.utc)
data = zipline.utils.factory.load_from_yahoo(stocks=['SPY', 'XLY'], indexes={}, start=start,
end=end, adjusted=False)
return data
pass
def abstract_loadDataOnline_SecArray(this):
'''only called when online (quantopian)'''
return [
sid(8554), # SPY S&P 500
]
pass

def abstract_update_timestep_handle_data(this, data = pandas.DataFrame()):
''' order 1 share of the first security each timestep'''
if(len(this._targetedSecurityIds)>0):
this.logger.info("buy {0} x1".format(this._targetedSecurityIds[0]));
this.tradingAlgorithm.order(this._targetedSecurityIds[0],1)
pass
else:
this.logger.info("no security found this timestep");
pass

##############  CONFIGURATION BELOW

def constructFramework(context,isOffline):
'''factory method to return your custom framework/trading algo'''
return ExampleAlgo(context,isOffline);

############## OFLINE RUNNER BELOW.  EDIT ABOVE THIS LINE
is_offline_Zipline = false
if __name__ == '__main__':
#import pylab
is_offline_Zipline = true

if(is_offline_Zipline):
initalize_zipline()
pass



fyi the github repo has an improved (but quantopian only) version of my framework now. I need this as an intraday/live trading framework, so I'll be continually updating it until it's on par with my previous (interday only) framework.

@Jason: Nice work! But why can't you use zipline.api for this? Should reduce your code burden.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Thomas, I didn't know such a thing existed (you replied acknowledging my rant about no docs) I searched online and I see this: https://github.com/quantopian/zipline/blob/master/zipline/api.py but it's not obvious to me how that would help? Could you please elaborate? Sorry I'm not a python person so I might be missing something obvious to the PEP inclined.

also fyi during my search just now I found what appear to be the built html version of the docs: http://zipline.readthedocs.org/en/latest/ Is that reasonablly up to date?

it looks like with zipline.api I should be able to add this to the offline-zipline version, is this correct?

import zipline.api
symbol("SPY")
#etc for the quantopian slippage/commision/etc global methods


Unfortunately the readthedocs is not up-to-date. We'll add new docs to zipline.io.

zipline.api is work in progress so that's why it's not advertised but the idea is to be able to write Quantopian algos that can be run on zipline (and vice versa). Thus if you do (although explicit imports are to be preferred in Python):

from zipline.api import *


You should get some of the functionality like the order methods, symbol() etc (if your zipline is from github master). See here for a simple example of instantiating an algorithm like that:
https://github.com/quantopian/zipline/blob/master/zipline/examples/quantopian_buy_apple.py

ok thank you! I'll work on getting the project zipline compatable again after I finish adding features.

Check out our new blog post on this topic:
http://blog.quantopian.com/unifying-zipline-quantopian/

OK Great

BTW, are the docs updated for the IPython Magic? (Pl. assume no knowledge in Python, IPython, Zipline, Quantopian)

Not yet, but they will be for the 0.6.2 release.

OK.

Another thing is we are in no man's land since the batch_trasform is deprecated and buggy and history is not yet fully functional and implementation is quaky and does not support minute mode at least. When writing an algo for live trading without backfilling it will not run immediately.