Simple Pipeline Help

I'm new...and I have been banging my head against the wall trying to create a simple algorithm that does a couple of very simple things that I have backtested in Think or Swim with individual stocks.

1. I want my base universe to be the NASDAQ 100.
2. I want to purchase a stock when it hits a 10 week high with.
3. I want to sell a stock if it falls below the 75 week low.
4. Each position should be around 1% of the whole.

Someone please tell me this is a simple problem and I am just over complicating it. :)

7 responses
1. I want my base universe to be the NASDAQ 100.

You have to download all the historical NASDAQ 100 entries, use self-serve data and then upload them. After that, create you're own filter that filters out companies not in that file.

2,3,4:

I don't think pipeline is a good fit for this strategy. Pipeline allows you to very easily create market neutral long/short portfolios. For example, you want each position to be 1%. But using the NASDAQ 100, how is that possible? Not all stocks in the NASDAQ 100 are hitting 10 week highs and falling below 75 week lows.

You need to reformulate your strategy in terms of holding a large number (500+) securities which are held both long and short. To choose these 500, you need to be able to assign a score to each security in your universe and then use that score to figure out portfolio weights. This is how you develop an alpha factor. Each and every stock in existence should have a numerical number that quantifies its exposure to your factor.

Even though this isn't a good fit for pipeline, you can easily do implement your strategy using the algorithm IDE. Look at some of the examples you you'll be able to figure it out.

Yep. Been trying to fit everything into pipelines. 🤦

Someone said impossible... darn, now I HAVE to do it!
I used a custom filter to approximate the criteria for the Nasdaq 100 and a custom factor to screen for the n-weeks-highs/lows. I didn't know what you mean by selling, exiting longs or going short. It now exits longs.

Btw,

you want each position to be 1%. But using the NASDAQ 100, how is that possible?

that was the easiest part ;)

order_target_percent(symbol, 0.01)

3
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
from quantopian.pipeline.data.morningstar import Fundamentals as f
from quantopian.pipeline.data.factset import Fundamentals as FF
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline import CustomFactor
from quantopian.pipeline import Pipeline, CustomFilter
from quantopian.algorithm import attach_pipeline, pipeline_output
import numpy as np

max_leverage = 1

def initialize(context):
schedule_function(
my_rebalance,
date_rules.every_day(),
# date_rules.week_start(),
time_rules.market_open(minutes=5)
)
schedule_function(
record_vars,
date_rules.every_day(),
time_rules.market_close()
)

attach_pipeline(make_pipeline(), 'my_pipeline')

class N100(CustomFilter):
inputs = [
f.morningstar_sector_code,
USEquityPricing.volume,
f.market_cap,
f.primary_exchange_id
]
window_length = 200

def compute(self, today, assets, out, sec, v, mcap, exch):

# Nasdaq has to be primary exchange
screen = exch[-1] == 'NAS'

# Finance sector is excluded (morningstar sector code 103)
screen &= sec[-1] != 103

# Need to have an average daily volume of 200k
avol = np.nanmean(v, axis=0)
screen &= avol >= 200000

# Pick out the biggest 103 of them in terms of market cap
top_cap = np.sort(mcap[-1][screen])[-103]
screen &= mcap[-1] >= top_cap

out[:] = screen

class Screener(CustomFactor):
inputs = [USEquityPricing.close]
window_length = 375

def compute(self, today, assets, out, c):
# 50-days-high

# 375-days-low
sell = (c[-1] < c[:-1]).all(axis=0)

res = np.zeros_like(assets).astype(float)
res[sell] = -1

out[:] = res

def make_pipeline():

wts = Screener()

return Pipeline(
columns={
'wts': wts,
},
screen=N100(),

)

def my_rebalance(context, data):
df = pipeline_output('my_pipeline')
wts = df.wts
wts = wts[np.isfinite(wts)]
wts = wts[wts != 0]

for sym in wts.index:
if sym in context.portfolio.positions:
if wts[sym] == -1:
order_target_percent(sym, 0)
print('long exit {}'.format(sym.symbol))
elif wts[sym] == 1:
if context.account.leverage > max_leverage:
return
order_target_percent(sym, .01)
print('long entry {}'.format(sym.symbol))

def record_vars(context, data):

wts = context.portfolio.current_portfolio_weights
longs = len(wts[wts > 0])
shorts = len(wts[wts < 0])
record(longs=longs,
shorts=shorts,
# both=longs + shorts
)

record(lever=context.account.leverage)


There was a runtime error.

It just came to me that symbols could drop out of the universe. For that case you should add a logic in my_rebalance or you might get stuck with them forever

    for sym in context.portfolio.positions.keys():
if sym not in wts.index:
order_target_percent(sym, 0)



Probably best before these lines:

    wts = wts[np.isfinite(wts)]
wts = wts[wts != 0]


that was the easiest part ;)

Yes, but assuming you want to invest all of your cash, how are you going to put 1% in each position with a universe of 100 names? Presumably you want to maintain a leverage ratio of 1, correct?

Yep. Been trying to fit everything into pipelines.

Why do you want to use pipeline? It would be much easier to just not use it. What's the value that pipeline adds when it's not suited for this particular strategy?

I wanted to see how close my universe selection is to nasdaq 100, so I applied the weighting methods used by nasdaq, rebalanced accordingly and calculated the correlation of the porfolio values to QQQ. It's very close, you could almost say identical with a correlation between 0.99 and 1.

3
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
from quantopian.pipeline.data.morningstar import Fundamentals as f
from quantopian.pipeline.data.factset import Fundamentals as FF
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline import CustomFactor
from quantopian.pipeline import Pipeline, CustomFilter
from quantopian.algorithm import attach_pipeline, pipeline_output
import numpy as np
from quantopian.pipeline.factors import SimpleMovingAverage as Sma
import quantopian.optimize as opt

max_leverage = 1

def initialize(context):
set_benchmark(symbol('QQQ'))
schedule_function(
my_rebalance,
# date_rules.every_day(),
# date_rules.week_start(),
date_rules.month_start(),
time_rules.market_open(minutes=5)
)
schedule_function(
record_vars,
date_rules.every_day(),
time_rules.market_close()
)

attach_pipeline(make_pipeline(), 'my_pipeline')
context.month = -1
context.value = []

class N100(CustomFilter):
inputs = [
f.morningstar_sector_code,
USEquityPricing.volume,
f.market_cap,
f.primary_exchange_id
]
window_length = 63
window_safe = True

def compute(self, today, assets, out, sec, v, mcap, exch):

# Nasdaq has to be primary exchange
screen = exch[-1] == 'NAS'

# Finance sector is excluded (morningstar sector code 103)
screen &= sec[-1] != 103

# must have traded for at least three full calendar months
screen &= ~np.isnan(v).any(axis=0)

# Need to have an average daily volume of 200k
avol = np.nanmean(v, axis=0)
screen &= avol >= 200000

# Pick out the biggest 100 of them in terms of market cap
top_cap = np.sort(mcap[-1][screen])[-100]
screen &= mcap[-1] >= top_cap

out[:] = screen

class N100QuarterlyWeighting(CustomFactor):
inputs = [f.market_cap]
window_length = 375

def compute(self, today, assets, out, mcap):

# stage 1:
# initial weighting by market cap:
wts = mcap[-1] / np.nansum(mcap[-1])
# if any weights exceed 24 % they are clipped to 20 %:
ecx24 = wts > .24
wts[ecx24] = .2

# stage 2:
# find weights that exceed 4.5 %:
exc40 = wts > .045
# if the sum of those weights is larger than 48 %, their sum is clipped to 40 %
if np.nansum(wts[exc40]) > .48:
wts[exc40] = wts[exc40] / np.nansum(wts[exc40]) * .4

out[:] = wts

class N100YearlyWeighting(CustomFactor):
inputs = [f.market_cap]
window_length = 375

def compute(self, today, assets, out, mcap):

# stage 1:
# initial weighting by market cap:
wts = mcap[-1] / np.nansum(mcap[-1])
# if any weights exceed 15 % they are clipped to 14 %:
ecx24 = wts > .15
wts[ecx24] = .14

# stage 2:
# If the aggregate weight of the subset of Index Securities with the five largest market capitalizations is
# less than 40%, Stage 1 weights are used as final weights; otherwise, Stage 1 weights are adjusted to
# meet the following constraints, producing the final weights:
#      The aggregate weight of the subset of Index Securities with the five largest market
#      capitalizations is set to 38.5%.
#
#      No security with a market capitalization outside the largest five may have a final index weight
#      exceeding the lesser of 4.4% or the final index weight of the Index Security ranked fifth by
#      market capitalization.
fifth_largest = np.sort(wts)[-5]
top5 = wts >= fifth_largest
if np.nansum(wts[top5]) >= .4:
wts[top5] = wts[top5] / np.nansum(wts[top5]) * .385

fifth_largest = np.sort(wts)[-5]
max_wt = min(.044, fifth_largest)
exc_mwt = (wts > max_wt) & ~top5
wts[exc_mwt] = max_wt

out[:] = wts

def make_pipeline():
# nasdaq 100 constituants are only checked once a year, so we downsample the filter:
nsd = N100().downsample('year_start')

# however, the weighting is adjusted quarterly

# and once a year there's a different weighting process

return Pipeline(
columns={
'qwts': qwts,
'ywts': ywts,
},
screen=nsd,

)

def my_rebalance(context, data):
# only adjust the weights every quarter
context.month += 1
if context.month % 3 != 0:
return

df = pipeline_output('my_pipeline')
wts = df.qwts

# once a year apply the other weighting
if context.month % 12 == 0:
wts = df.ywts

order_optimal_portfolio(
objective=opt.TargetWeights(wts),
constraints=[],
)

def record_vars(context, data):
context.value.append(context.portfolio.portfolio_value)
corr = 0
portval = np.array(context.value)
n = max(len(portval), 100)
if len(portval) >= n:
qqq = data.history(symbol('QQQ'), 'close', n, '1d').values
corr = np.corrcoef(qqq, portval)[0,1]

positions = len(context.portfolio.positions)
record(positions=positions)
record(lever=context.account.leverage)
record(correlation_to_QQQ=corr)


There was a runtime error.

you want to maintain a leverage ratio of 1, correct?

That depends on many factors, one being who you are testing for. For Quantopian funding, challenges and the contest it was a requirement - but none of this is an option right now. Not all hedge funds have this requirement and it also can be useful to have a cash reserve, for instance in case of a short squeeze and the related margin calls.