Unexplained number of stocks

Hi,
my algo is designed to enter in the market with a fixed number of stocks and it works like this:

for each sector:
- select good and bad stocks
- check if the number of good and bad stocks is enough (greather than c.num_stocks)
- save those stocks in a dict
update universe with those stocks

now for the order part...

1. check if the stocks is delisted
2. check if the stocks has open orders
3. exit all previous positions no matter what (to be shure that i'm flat every rebalance day)
4. order stocks

The problem i'm stuck with:
As you can see i never have the same amount of positions for the long side and for the short side as i would expect since the correct number of stocks is available for shure.
The leverage is never the same maybe this is related to the number of stocks....

What i have done to avoid this:
filter the pipeline with this

    price_filter   = (price >= 1.0)
volume_filter  = (volume >= 100 * 60 * 6.30)
mkt_cap_filter =  mkt_cap > 3000e6
dollar_volume_filter = dollar_volume > 10**7


and as i have written before i have avoided open orders stocks and delisted stocks.

Thank you very much for any help..

2
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Put any initialization logic here.  The context object will be passed to
# the other methods in your algorithm.
from quantopian.pipeline import Pipeline
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline import CustomFactor
from quantopian.pipeline.factors import Latest
from quantopian.pipeline.data import morningstar
from quantopian.pipeline.data.builtin import USEquityPricing
import pandas as pd
import numpy as np
import statsmodels.api as sm
import talib

def initialize(context):
set_do_not_order_list(security_lists.leveraged_etf_list)
c = context
c.num_stocks= 1
context.fund_factors_list = ['roic',
'profit_margin',
'growth']
'''
schedule_function(debugly,
date_rules.every_day(),
time_rules.market_close(hours=0,minutes=1))
'''
schedule_function(record_daily_values,
date_rules.month_start(),
time_rules.market_close(hours=0,minutes=30))
schedule_function(rebalance,
date_rules.month_start(),
time_rules.market_open(hours=0,minutes=30))
#############################################
# Pipeline definition
#############################################
pipe_columns = {
'roic' : Roic(window_length=1),
'mkt_cap' : MarketCap(),
'profit_margin' : Profit_margin(window_length=1),
'growth': Growth(window_length= 1),
'sector': Sector()
}

price = USEquityPricing.close.latest
volume = USEquityPricing.volume.latest
mkt_cap = MarketCap()

price_filter   = (price >= 1.0)
volume_filter  = (volume >= 100 * 60 * 6.30)
mkt_cap_filter =  mkt_cap > 3000e6
dollar_volume_filter = dollar_volume > 10**7

full_filter = price_filter & volume_filter & mkt_cap_filter & dollar_volume_filter

pipe = Pipeline(columns=pipe_columns)
pipe.set_screen(full_filter)

attach_pipeline(pipe, 'pipe')

c = context
c.best_stocks={} # { sector: dataframe of his stocks }
c.worst_stocks={}
c.output = pipeline_output('pipe')
grouped =  c.output.groupby('sector')

#########################################
# Pick stocks for each sector.
#########################################
for sector, df in grouped:

factors_dict = {factor: df[factor].mean() for factor in context.fund_factors_list}
best_fdm = [df[df[factor] > mean_value].index for factor, mean_value in factors_dict.iteritems()]
worst_fdm = [df[df[factor] < mean_value].index for factor, mean_value in factors_dict.iteritems()]
best_stocks = list(reduce(set.intersection,map(set,best_fdm)))
worst_stocks = list(reduce(set.intersection,map(set,worst_fdm)))
if len(best_stocks) > c.num_stocks:

best_df = df['mkt_cap'].loc[best_stocks]
ordered_best = best_df.order('mkt_cap', ascending = False)[:c.num_stocks]
c.best_stocks[sector] = ordered_best.index

else:

continue # skip this sector if it has not enough number of stocks
if len(worst_stocks) > c.num_stocks:

worst_df = df['mkt_cap'].loc[worst_stocks]
ordered_worst = worst_df.order('mkt_cap', ascending = False)[:c.num_stocks]
c.worst_stocks[sector] =  ordered_worst.index
else:

continue # skip this sector if it has not enough number of stocks

num_sectors = len(c.best_stocks.keys())
# each sector has a number of stocks equals to c.num_stocks * 2 so..
total_num_stocks = num_sectors * c.num_stocks *2
c.weigth = .95 / total_num_stocks
c.longs = sum(c.best_stocks.values(), [])
c.shorts = sum(c.worst_stocks.values(), [])

c.stocks = np.union1d(c.longs , c.shorts)

update_universe(c.stocks)

def rebalance(context, data):
c=context
c.currently_long = []
c.currently_short = []

open_orders = get_open_orders()
for stock in data:
if stock in open_orders:
continue
if (stock.end_date - get_datetime()).days < 365:
continue
if context.portfolio.positions[stock].amount != 0:
order_target(stock, 0)
if stock in c.longs:
order_target_percent(stock, c.weigth)
elif stock in c.shorts:
order_target_percent(stock, -c.weigth)

def handle_data(context, data):
pass

#
# PIPELINE CLEANING CLASSES
#
class Sector(CustomFactor):
inputs = [ morningstar.asset_classification.morningstar_sector_code ]
window_length = 1
def compute(self, today, assets, out, sector):
out[:] = sector

class Pct_Ret(CustomFactor):
inputs = [USEquityPricing.close]
def compute(self, today, assets, out, prices):
current = prices[-1]
previous = prices
ret = ((current-previous)/(1.0*previous))*100
out[:] = ret

class MarketCap(CustomFactor):
inputs = [USEquityPricing.close, morningstar.valuation.shares_outstanding]
window_length = 1
def compute(self, today, assets, out, close, shares):
out[:] = (close[-1] * shares[-1])

inputs = [USEquityPricing.close, USEquityPricing.volume]
window_length = 20
def compute(self, today, assets, out, close_price, volume):
out[:] = np.mean(close_price * volume, axis=0)

class Slope(CustomFactor):
inputs = [USEquityPricing.close]
window_length = 252
def compute(self, today, assets, out, prices):
Y = prices
X = range(len(prices))
results = sm.OLS(Y,A).fit()
(b, a) =results.params
slope = a / b * 252.0
out[:] = slope

class Growth(CustomFactor):
inputs = [morningstar.operation_ratios.revenue_growth]
window_length = 1
def compute(self, today, assets, out, growth):
out[:] = growth

class Roic(CustomFactor):
inputs = [morningstar.operation_ratios.roic]
window_length = 1
def compute(self, today, assets, out, roic):
out[:] = roic

class Profit_margin(CustomFactor):
inputs = [morningstar.operation_ratios.normalized_net_profit_margin]
window_length = 1
def compute(self, today, assets, out, pm):
out[:] = pm

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
def record_daily_values(context,data):

#####################################################################
#  Best Practice #6: Record leverage
#####################################################################

# The record() function allows you to create a custom time-series plot of any variable in your simulation.
# In this example we will record and plot the market exposure and leverage of our portfolio over time.
record(leverage = context.account.leverage)
record(market_exposure=context.account.net_leverage)
#record(stock_counte=context.min_num)
# We also want to monitor the number of long and short positions in our portfolio over time. This
# loop will check our positition sizes and add the count of longs and shorts to our custom time-series plot.
longs = shorts = 0
for position in context.portfolio.positions.itervalues():
if position.amount > 0:
longs += 1
if position.amount < 0:
shorts += 1

record(long_count=longs, short_count=shorts)

def debugly(context, data):

log.info('\n\t REBALANCE\nbest performing sector:{0}\nworst performing sector:{1}\ncurrently long with{2}\ncurrently short with{3}\nenter long with:{4}\nenter short with:{5}'
.format(int(context.best_performing_sector),
int(context.worst_performing_sector),
[s.symbol for s in context.currently_long],
[s.symbol for s in context.currently_short],
[s.symbol for s in context.longs],
[s.symbol for s in context.shorts]))


There was a runtime error.
2 responses

Hi Giuseppe,

One of the issues here is that when you are exiting all of your previous positions, an order is then sometimes placed (either long or short) in one of the positions that was just exited. Since this is being done in the same bar (there is no else before a long or short positions is opened), the ordering functions end up over ordering. Since order_target() or order_target_percent() converts to an order in number of shares behind the scenes, if an order is placed to close a position, and then an order is placed to open a new position, the order for the new position will calculate the number of shares to order based on the current position as opposed to the closed (i.e. 0 shares) position.

I've attached a version of your backtest that fixes this - see line 122 for my change.

Does this make sense?

It also appears that c.longs and c.shorts are not always the same length coming out of before_trading_start. Was this something that you had noticed before? I couldn't really tell from your question!

I'm not sure what you want to do

1
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Put any initialization logic here.  The context object will be passed to
# the other methods in your algorithm.
from quantopian.pipeline import Pipeline
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline import CustomFactor
from quantopian.pipeline.factors import Latest
from quantopian.pipeline.data import morningstar
from quantopian.pipeline.data.builtin import USEquityPricing
import pandas as pd
import numpy as np
import statsmodels.api as sm
import talib

def initialize(context):
context.counter =0
set_do_not_order_list(security_lists.leveraged_etf_list)
c = context
c.num_stocks= 1
context.fund_factors_list = ['roic',
'profit_margin',
'growth']
'''
schedule_function(debugly,
date_rules.every_day(),
time_rules.market_close(hours=0,minutes=1))
'''
schedule_function(record_daily_values,
date_rules.month_start(),
time_rules.market_close(hours=0,minutes=30))
schedule_function(rebalance,
date_rules.month_start(),
time_rules.market_open(hours=0,minutes=30))
#############################################
# Pipeline definition
#############################################
pipe_columns = {
'roic' : Roic(window_length=1),
'mkt_cap' : MarketCap(),
'profit_margin' : Profit_margin(window_length=1),
'growth': Growth(window_length= 1),
'sector': Sector()
}

price = USEquityPricing.close.latest
volume = USEquityPricing.volume.latest
mkt_cap = MarketCap()

price_filter   = (price >= 1.0)
volume_filter  = (volume >= 100 * 60 * 6.30)
mkt_cap_filter =  mkt_cap > 3000e6
dollar_volume_filter = dollar_volume > 10**7

full_filter = price_filter & volume_filter & mkt_cap_filter & dollar_volume_filter

pipe = Pipeline(columns=pipe_columns)
pipe.set_screen(full_filter)

attach_pipeline(pipe, 'pipe')

c = context
c.best_stocks={} # { sector: dataframe of his stocks }
c.worst_stocks={}
c.output = pipeline_output('pipe')
grouped =  c.output.groupby('sector')

#########################################
# Pick stocks for each sector.
#########################################
for sector, df in grouped:

factors_dict = {factor: df[factor].mean() for factor in context.fund_factors_list}
best_fdm = [df[df[factor] > mean_value].index for factor, mean_value in factors_dict.iteritems()]
worst_fdm = [df[df[factor] < mean_value].index for factor, mean_value in factors_dict.iteritems()]
best_stocks = list(reduce(set.intersection,map(set,best_fdm)))
worst_stocks = list(reduce(set.intersection,map(set,worst_fdm)))
if len(best_stocks) > c.num_stocks:

best_df = df['mkt_cap'].loc[best_stocks]
ordered_best = best_df.order('mkt_cap', ascending = False)[:c.num_stocks]
c.best_stocks[sector] = ordered_best.index

else:

continue # skip this sector if it has not enough number of stocks
if len(worst_stocks) > c.num_stocks:

worst_df = df['mkt_cap'].loc[worst_stocks]
ordered_worst = worst_df.order('mkt_cap', ascending = False)[:c.num_stocks]
c.worst_stocks[sector] =  ordered_worst.index
else:

continue # skip this sector if it has not enough number of stocks

num_sectors = len(c.best_stocks.keys())
# each sector has a number of stocks equals to c.num_stocks * 2 so..
total_num_stocks = num_sectors * c.num_stocks *2
c.weigth = .95 / total_num_stocks
c.longs = sum(c.best_stocks.values(), [])
c.shorts = sum(c.worst_stocks.values(), [])

c.stocks = np.union1d(c.longs , c.shorts)

update_universe(c.stocks)

def rebalance(context, data):
c=context
c.currently_long = []
c.currently_short = []

open_orders = get_open_orders()
print 'Longs: %d' % len(c.longs)
print 'Shorts: %d' % len(c.shorts)
for stock in data:
if stock in open_orders:
continue
if (stock.end_date - get_datetime()).days < 365:
continue
if context.portfolio.positions[stock].amount != 0 and stock not in c.longs and stock not in c.shorts:
order_target(stock, 0)
if stock in c.longs:
order_target_percent(stock, c.weigth)
elif stock in c.shorts:
order_target_percent(stock, -c.weigth)

def handle_data(context, data):
pass

#
# PIPELINE CLEANING CLASSES
#
class Sector(CustomFactor):
inputs = [ morningstar.asset_classification.morningstar_sector_code ]
window_length = 1
def compute(self, today, assets, out, sector):
out[:] = sector

class Pct_Ret(CustomFactor):
inputs = [USEquityPricing.close]
def compute(self, today, assets, out, prices):
current = prices[-1]
previous = prices
ret = ((current-previous)/(1.0*previous))*100
out[:] = ret

class MarketCap(CustomFactor):
inputs = [USEquityPricing.close, morningstar.valuation.shares_outstanding]
window_length = 1
def compute(self, today, assets, out, close, shares):
out[:] = (close[-1] * shares[-1])

inputs = [USEquityPricing.close, USEquityPricing.volume]
window_length = 20
def compute(self, today, assets, out, close_price, volume):
out[:] = np.mean(close_price * volume, axis=0)

class Slope(CustomFactor):
inputs = [USEquityPricing.close]
window_length = 252
def compute(self, today, assets, out, prices):
Y = prices
X = range(len(prices))
results = sm.OLS(Y,A).fit()
(b, a) =results.params
slope = a / b * 252.0
out[:] = slope

class Growth(CustomFactor):
inputs = [morningstar.operation_ratios.revenue_growth]
window_length = 1
def compute(self, today, assets, out, growth):
out[:] = growth

class Roic(CustomFactor):
inputs = [morningstar.operation_ratios.roic]
window_length = 1
def compute(self, today, assets, out, roic):
out[:] = roic

class Profit_margin(CustomFactor):
inputs = [morningstar.operation_ratios.normalized_net_profit_margin]
window_length = 1
def compute(self, today, assets, out, pm):
out[:] = pm

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
def record_daily_values(context,data):

#####################################################################
#  Best Practice #6: Record leverage
#####################################################################

# The record() function allows you to create a custom time-series plot of any variable in your simulation.
# In this example we will record and plot the market exposure and leverage of our portfolio over time.
record(leverage = context.account.leverage)
record(market_exposure=context.account.net_leverage)
#record(stock_counte=context.min_num)
# We also want to monitor the number of long and short positions in our portfolio over time. This
# loop will check our positition sizes and add the count of longs and shorts to our custom time-series plot.
longs = shorts = 0
for position in context.portfolio.positions.itervalues():
if position.amount > 0:
longs += 1
if position.amount < 0:
shorts += 1

record(long_count=longs, short_count=shorts)

def debugly(context, data):

log.info('\n\t REBALANCE\nbest performing sector:{0}\nworst performing sector:{1}\ncurrently long with{2}\ncurrently short with{3}\nenter long with:{4}\nenter short with:{5}'
.format(int(context.best_performing_sector),
int(context.worst_performing_sector),
[s.symbol for s in context.currently_long],
[s.symbol for s in context.currently_short],
[s.symbol for s in context.longs],
[s.symbol for s in context.shorts]))


There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

this really helped me a lot, i thought that exiting all position to start new orders from scratch was the simplest way to go and i was screwing the things up instead :)

However i haven't noticed that c.longs and c.shorts are not the same length every time, but i can correct them.

This is my first attempt to write a mkt neutral algo and the idea is to use fundamental factors to split good stocks from bad ones but inside each sector.
So actually from each sector i expect to find 2 good stocks and 2 bad stocks, finally combine the good stocks all together and go long in them and then do the same thing for the bad ones and short them, so yes c.longs and c.shorts has to be same length.

Thank you very much again :)