Back to Community
Simple Pipeline Help

I'm new...and I have been banging my head against the wall trying to create a simple algorithm that does a couple of very simple things that I have backtested in Think or Swim with individual stocks.

  1. I want my base universe to be the NASDAQ 100.
  2. I want to purchase a stock when it hits a 10 week high with.
  3. I want to sell a stock if it falls below the 75 week low.
  4. Each position should be around 1% of the whole.

Someone please tell me this is a simple problem and I am just over complicating it. :)

7 responses
  1. I want my base universe to be the NASDAQ 100.

You have to download all the historical NASDAQ 100 entries, use self-serve data and then upload them. After that, create you're own filter that filters out companies not in that file.

2,3,4:

I don't think pipeline is a good fit for this strategy. Pipeline allows you to very easily create market neutral long/short portfolios. For example, you want each position to be 1%. But using the NASDAQ 100, how is that possible? Not all stocks in the NASDAQ 100 are hitting 10 week highs and falling below 75 week lows.

You need to reformulate your strategy in terms of holding a large number (500+) securities which are held both long and short. To choose these 500, you need to be able to assign a score to each security in your universe and then use that score to figure out portfolio weights. This is how you develop an alpha factor. Each and every stock in existence should have a numerical number that quantifies its exposure to your factor.

Even though this isn't a good fit for pipeline, you can easily do implement your strategy using the algorithm IDE. Look at some of the examples you you'll be able to figure it out.

Yep. Been trying to fit everything into pipelines. 🤦

Someone said impossible... darn, now I HAVE to do it!
I used a custom filter to approximate the criteria for the Nasdaq 100 and a custom factor to screen for the n-weeks-highs/lows. I didn't know what you mean by selling, exiting longs or going short. It now exits longs.

Btw,

you want each position to be 1%. But using the NASDAQ 100, how is that possible?

that was the easiest part ;)

order_target_percent(symbol, 0.01)  
Clone Algorithm
3
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
from quantopian.pipeline.data.morningstar import Fundamentals as f
from quantopian.pipeline.data.factset import Fundamentals as FF
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline import CustomFactor
from quantopian.pipeline import Pipeline, CustomFilter
from quantopian.algorithm import attach_pipeline, pipeline_output
import numpy as np
from quantopian.pipeline.filters import QTradableStocksUS


max_leverage = 1

def initialize(context):
    schedule_function(
        my_rebalance,
        date_rules.every_day(),
        # date_rules.week_start(),
        time_rules.market_open(minutes=5)
    )
    schedule_function(
        record_vars,
        date_rules.every_day(),
        time_rules.market_close()
    )

    attach_pipeline(make_pipeline(), 'my_pipeline')



class N100(CustomFilter):
    inputs = [
        f.morningstar_sector_code,
        USEquityPricing.volume,
        f.market_cap,
        f.primary_exchange_id
    ]
    window_length = 200

    def compute(self, today, assets, out, sec, v, mcap, exch):
        
        # Nasdaq has to be primary exchange
        screen = exch[-1] == 'NAS'
        
        # Finance sector is excluded (morningstar sector code 103)
        screen &= sec[-1] != 103
        
        # Need to have an average daily volume of 200k
        avol = np.nanmean(v, axis=0)
        screen &= avol >= 200000
        
        # Pick out the biggest 103 of them in terms of market cap
        top_cap = np.sort(mcap[-1][screen])[-103]
        screen &= mcap[-1] >= top_cap

        out[:] = screen


class Screener(CustomFactor):
    inputs = [USEquityPricing.close]
    window_length = 375
    mask = N100()

    def compute(self, today, assets, out, c):
        # 50-days-high
        buy = (c[-1] > c[-50:-1]).all(axis=0)
        
        # 375-days-low
        sell = (c[-1] < c[:-1]).all(axis=0)

        res = np.zeros_like(assets).astype(float)
        res[buy] = 1
        res[sell] = -1

        out[:] = res


def make_pipeline():

    wts = Screener()

    return Pipeline(
        columns={
            'wts': wts,
        },
        screen=N100(),

    )


def my_rebalance(context, data):
    df = pipeline_output('my_pipeline')
    wts = df.wts
    wts = wts[np.isfinite(wts)]
    wts = wts[wts != 0]

    for sym in wts.index:
        if sym in context.portfolio.positions:
            if wts[sym] == -1:
                order_target_percent(sym, 0)
                print('long exit {}'.format(sym.symbol))
        elif wts[sym] == 1:
            if context.account.leverage > max_leverage:
                return
            order_target_percent(sym, .01)
            print('long entry {}'.format(sym.symbol))



def record_vars(context, data):

    wts = context.portfolio.current_portfolio_weights
    longs = len(wts[wts > 0])
    shorts = len(wts[wts < 0])
    record(longs=longs,
           shorts=shorts,
           # both=longs + shorts
           )

    record(lever=context.account.leverage)


There was a runtime error.

It just came to me that symbols could drop out of the universe. For that case you should add a logic in my_rebalance or you might get stuck with them forever

    for sym in context.portfolio.positions.keys():  
        if sym not in wts.index:  
            order_target_percent(sym, 0)

Probably best before these lines:

    wts = wts[np.isfinite(wts)]  
    wts = wts[wts != 0]  

that was the easiest part ;)

Yes, but assuming you want to invest all of your cash, how are you going to put 1% in each position with a universe of 100 names? Presumably you want to maintain a leverage ratio of 1, correct?

Yep. Been trying to fit everything into pipelines.

Why do you want to use pipeline? It would be much easier to just not use it. What's the value that pipeline adds when it's not suited for this particular strategy?

I wanted to see how close my universe selection is to nasdaq 100, so I applied the weighting methods used by nasdaq, rebalanced accordingly and calculated the correlation of the porfolio values to QQQ. It's very close, you could almost say identical with a correlation between 0.99 and 1.

Clone Algorithm
3
Loading...
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
from quantopian.pipeline.data.morningstar import Fundamentals as f
from quantopian.pipeline.data.factset import Fundamentals as FF
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline import CustomFactor
from quantopian.pipeline import Pipeline, CustomFilter
from quantopian.algorithm import attach_pipeline, pipeline_output
import numpy as np
from quantopian.pipeline.filters import QTradableStocksUS
from quantopian.pipeline.factors import SimpleMovingAverage as Sma
import quantopian.optimize as opt



max_leverage = 1

def initialize(context):
    set_benchmark(symbol('QQQ'))
    schedule_function(
        my_rebalance,
        # date_rules.every_day(),
        # date_rules.week_start(),
        date_rules.month_start(),
        time_rules.market_open(minutes=5)
    )
    schedule_function(
        record_vars,
        date_rules.every_day(),
        time_rules.market_close()
    )

    attach_pipeline(make_pipeline(), 'my_pipeline')
    context.month = -1
    context.value = []



class N100(CustomFilter):
    inputs = [
        f.morningstar_sector_code,
        USEquityPricing.volume,
        f.market_cap,
        f.primary_exchange_id
    ]
    window_length = 63
    window_safe = True

    def compute(self, today, assets, out, sec, v, mcap, exch):

        # Nasdaq has to be primary exchange
        screen = exch[-1] == 'NAS'

        # Finance sector is excluded (morningstar sector code 103)
        screen &= sec[-1] != 103

        # must have traded for at least three full calendar months
        screen &= ~np.isnan(v).any(axis=0)

        # Need to have an average daily volume of 200k
        avol = np.nanmean(v, axis=0)
        screen &= avol >= 200000

        # Pick out the biggest 100 of them in terms of market cap
        top_cap = np.sort(mcap[-1][screen])[-100]
        screen &= mcap[-1] >= top_cap

        out[:] = screen


class N100QuarterlyWeighting(CustomFactor):
    inputs = [f.market_cap]
    window_length = 375

    def compute(self, today, assets, out, mcap):

        # stage 1:
        # initial weighting by market cap:
        wts = mcap[-1] / np.nansum(mcap[-1])
        # if any weights exceed 24 % they are clipped to 20 %:
        ecx24 = wts > .24
        wts[ecx24] = .2

        # stage 2:
        # find weights that exceed 4.5 %:
        exc40 = wts > .045
        # if the sum of those weights is larger than 48 %, their sum is clipped to 40 %
        if np.nansum(wts[exc40]) > .48:
            wts[exc40] = wts[exc40] / np.nansum(wts[exc40]) * .4

        out[:] = wts


class N100YearlyWeighting(CustomFactor):
    inputs = [f.market_cap]
    window_length = 375

    def compute(self, today, assets, out, mcap):

        # stage 1:
        # initial weighting by market cap:
        wts = mcap[-1] / np.nansum(mcap[-1])
        # if any weights exceed 15 % they are clipped to 14 %:
        ecx24 = wts > .15
        wts[ecx24] = .14

        # stage 2:
        # If the aggregate weight of the subset of Index Securities with the five largest market capitalizations is
        # less than 40%, Stage 1 weights are used as final weights; otherwise, Stage 1 weights are adjusted to
        # meet the following constraints, producing the final weights:
        #      The aggregate weight of the subset of Index Securities with the five largest market
        #      capitalizations is set to 38.5%.
        #
        #      No security with a market capitalization outside the largest five may have a final index weight
        #      exceeding the lesser of 4.4% or the final index weight of the Index Security ranked fifth by
        #      market capitalization.
        fifth_largest = np.sort(wts)[-5]
        top5 = wts >= fifth_largest
        if np.nansum(wts[top5]) >= .4:
            wts[top5] = wts[top5] / np.nansum(wts[top5]) * .385

        fifth_largest = np.sort(wts)[-5]
        max_wt = min(.044, fifth_largest)
        exc_mwt = (wts > max_wt) & ~top5
        wts[exc_mwt] = max_wt

        out[:] = wts


def make_pipeline():
    # nasdaq 100 constituants are only checked once a year, so we downsample the filter:
    nsd = N100().downsample('year_start')

    # however, the weighting is adjusted quarterly
    qwts = N100QuarterlyWeighting(mask=nsd)

    # and once a year there's a different weighting process
    ywts = N100YearlyWeighting(mask=nsd)

    return Pipeline(
        columns={
            'qwts': qwts,
            'ywts': ywts,
        },
        screen=nsd,

    )


def my_rebalance(context, data):
    # only adjust the weights every quarter
    context.month += 1
    if context.month % 3 != 0:
        return

    df = pipeline_output('my_pipeline')
    wts = df.qwts

    # once a year apply the other weighting
    if context.month % 12 == 0:
        wts = df.ywts

    order_optimal_portfolio(
        objective=opt.TargetWeights(wts),
        constraints=[],
    )



def record_vars(context, data):
    context.value.append(context.portfolio.portfolio_value)
    corr = 0
    portval = np.array(context.value)
    n = max(len(portval), 100)
    if len(portval) >= n:
        qqq = data.history(symbol('QQQ'), 'close', n, '1d').values
        corr = np.corrcoef(qqq, portval)[0,1]

    positions = len(context.portfolio.positions)
    record(positions=positions)
    record(lever=context.account.leverage)
    record(correlation_to_QQQ=corr)


There was a runtime error.

you want to maintain a leverage ratio of 1, correct?

That depends on many factors, one being who you are testing for. For Quantopian funding, challenges and the contest it was a requirement - but none of this is an option right now. Not all hedge funds have this requirement and it also can be useful to have a cash reserve, for instance in case of a short squeeze and the related margin calls.
And if you are trading with your own money it's even more advisable not to blow up all your cash with a trading strategy ;)