Back to Community
problem w/ DVMT?

There appears to be a problem w/ the ticker DVMT. I'm using pipeline, and data.can_trade and still getting:

Something went wrong. Sorry for the inconvenience. Try using the built-in debugger to analyze your code. If you would like help, send us an email.
Cannot order DVMT on 2016-09-07 19:45:00+00:00 as there is no last price for the security. Try using data.can_trade to check whether an asset is delisted.
There was a runtime error on line 154.

However, the attached notebook shows that there was a price for 2016-09-07.

Loading notebook preview...
Notebook previews are currently unavailable.
9 responses

Backtest. Still getting error, even though I think I'm removing DVMT from the list of stocks to be traded.

Clone Algorithm
370
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
import numpy as np
from quantopian.algorithm import attach_pipeline, pipeline_output
from quantopian.pipeline import Pipeline
from quantopian.pipeline.data import morningstar as mstar
from quantopian.pipeline.filters import Q1500US
import cvxpy as cvx
from quantopian.pipeline.classifiers.morningstar import Sector
import pandas as pd
import quantopian.experimental.optimize as opt
import quantopian.algorithm as algo

# Constraint Parameters
MAX_GROSS_LEVERAGE = 1.0
MAX_SHORT_POSITION_SIZE = 0.015
MAX_LONG_POSITION_SIZE = 0.015

def initialize(context):
    
    set_slippage(slippage.FixedSlippage(spread=0.00))  
    set_commission(commission.PerShare(cost=0, min_trade_cost=0))
    
    # parameters
    # --------------------------
    context.n_stocks = 200 # universe size, top market cap
    context.N = 4 # trailing window size, days
    context.leverage = 1.0 # gross leverage
    # --------------------------
    
    schedule_function(run_optimization, date_rules.week_start(days_offset=1), time_rules.market_close(minutes=15))
    
    schedule_function(allocate, date_rules.week_start(days_offset=1), time_rules.market_close(minutes=15))
     
    attach_pipeline(make_pipeline(context), 'my_pipe')
    
def make_pipeline(context):
    
    market_cap = mstar.valuation.market_cap.latest
    
    my_screen = market_cap.top(context.n_stocks, mask = Q1500US())
    
    return Pipeline(columns={
            'sector': Sector(),
        },screen = my_screen)
    
def before_trading_start(context,data):

    context.output = pipeline_output('my_pipe')
    
    context.stocks = context.output.index.tolist()
               
    record(leverage = context.account.leverage)
    
    num_secs = 0
    b_t = np.zeros(len(context.portfolio.positions.keys()))
    for i,stock in enumerate(context.portfolio.positions.keys()):
        if context.portfolio.positions[stock].amount != 0:
            num_secs += 1
        b_t[i] = context.portfolio.positions[stock].amount*data.current(stock,'price')
            
    record(num_secs = num_secs)
    
    denom = np.sum(np.absolute(b_t))
    if denom > 0:
        ls_sum = np.sum(b_t)/denom
    else:
        ls_sum = 0
        
    record(ls_sum = ls_sum)
    
def run_optimization(context,data):
    
    N = 390*context.N
    prices = data.history(context.stocks, 'price', N, '1m').dropna(axis=1)
    context.stocks = list(prices.columns.values)
    prices = prices.ewm(com=15).mean().as_matrix(context.stocks)
    
    m = len(context.stocks)
    
    a = np.zeros(m)
    w = 0
        
    for n in range(1,26*context.N+1):
        (a,w) = get_weights(context,data,prices[-n*15:,:])
        a += w*a
        w += w
    
    a = a/w
    a = a - np.mean(a)
                   
    denom = np.sum(np.absolute(a))
    if denom > 0:
        a = a/denom
    
    context.weight = a
    
def allocate(context, data):
    
    # remove DVMT to avoid error
    # see https://www.quantopian.com/posts/problem-w-slash-dvmt
    if sid(50242) in context.stocks: context.stocks.remove(sid(50242))
    
    pipeline_data = pd.DataFrame({'alpha': context.weight},index = context.stocks)
    
    df_pipeline = context.output.ix[context.stocks]
    pipeline_data = pipeline_data.join(df_pipeline)
    pipeline_data = pipeline_data.loc[data.can_trade(pipeline_data.index)]
    todays_universe = pipeline_data.index
    
    # Objective
    # ---------
    # For our objective, we simply use our naive ranks as an alpha coefficient
    # and try to maximize that alpha.
    # 
    # This is a **very** naive model. Since our alphas are so widely spread out,
    # we should expect to always allocate the maximum amount of long/short
    # capital to assets with high/low ranks.
    #
    # A more sophisticated model would apply some re-scaling here to try to generate
    # more meaningful predictions of future returns.
    objective = opt.MaximizeAlpha(pipeline_data.alpha)

    # Constraints
    # -----------
    # Constrain our gross leverage to 1.0 or less. This means that the absolute
    # value of our long and short positions should not exceed the value of our
    # portfolio.
    constrain_gross_leverage = opt.MaxGrossLeverage(MAX_GROSS_LEVERAGE)
    
    # Constrain individual position size to no more than a fixed percentage 
    # of our portfolio. Because our alphas are so widely distributed, we 
    # should expect to end up hitting this max for every stock in our universe.
    constrain_pos_size = opt.PositionConcentration.with_equal_bounds(
        -MAX_SHORT_POSITION_SIZE,
        MAX_LONG_POSITION_SIZE,
    )

    # Constrain ourselves to allocate the same amount of capital to 
    # long and short positions.
    market_neutral = opt.DollarNeutral()
    
    # Constrain ourselve to have a net leverage of 0.0 in each sector.
    sector_neutral = opt.NetPartitionExposure.with_equal_bounds(
        labels=pipeline_data.sector,
        min=-0.0001,
        max=0.0001,
    )

    # Run the optimization. This will calculate new portfolio weights and
    # manage moving our portfolio toward the target.
    algo.order_optimal_portfolio(
        objective=objective,
        constraints=[
            constrain_gross_leverage,
            constrain_pos_size,
            market_neutral,
            sector_neutral,
        ],
        universe=todays_universe,
    )
    
def get_weights(context,data,prices):
    
    d = np.ones(len(context.stocks))
    
    x_tilde = np.mean(prices,axis=0)/prices[-1,:]
    y_tilde = 1.0/x_tilde
    
    d[x_tilde < 1] = -1
    
    x_tilde[x_tilde < 1] = 0
    y_tilde[x_tilde != 0] = 0
    
    x_tilde = x_tilde + y_tilde

    return (d*x_tilde, np.sum(x_tilde))
There was a runtime error.

Backtester shows no trade on 2016-09-07, but research platform does.

1969-12-31 19:00 PRINT 2016-09-07 00:00:00+00:00
2016-09-08 08:45 PRINT 2016-09-07 13:31:00+00:00 NaN
2016-09-07 13:32:00+00:00 NaN
2016-09-07 13:33:00+00:00 NaN
2016-09-07 13:34:00+00:00 NaN
2016-09-07 13:35:00+00:00 NaN
Name: Equity(50242 [DVMT]), dtype: float64

Clone Algorithm
3
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
def initialize(context):
    context.stock = sid(50242)
    
    print context.stock.start_date
 
def before_trading_start(context, data):
    
    prices = data.history(context.stock, 'price', 390, '1m').head(5)
    print prices
There was a runtime error.

Hello Grant,

It looks like your algorithm holds an EMC (sid(2518)) position on 09/07/2016, which is the pay date for a stock dividend that pays out in DVMT (Dell). This causes the algo to acquire an unexpected position in DVMT and when it calls order_optimal_portfolio with no target weight for this asset, the optimizer tries to sell out of the position. However, the backtester doesn't have trade data for DVMT until 2016-09-08, so it thinks DVMT isn't tradable and rejects the attempt to close the position.

A temporary fix would be to filter out EMC from your tradable universe either by removing it from the pipeline output in before_trading_start, or excluding it from your pipeline using the built-in filter StaticAssets. Adding a more general solution to the optimization API is on our list of issues to tackle.

The get_pricing function actually uses a different data feed than the one used by the backtester and the pipeline API. We are currently working on migrating get_pricing to use the new data feed, so the datasets match in both environments.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thanks Ernesto -

I never would have figured that one out!

It sounds like the trick might be to also apply the data.can_trade test to stocks in the currently held portfolio, since sometimes one ticker can magically convert into another one overnight. Or I guess one would still be hosed, since universe=todays_universe only applies to the stocks being added--everything else gets dropped automatically under the hood.

Why does the optimization routine not apply the data.can_trade test upon selling? Could the test be added? It'd be a one-line change, right? Or is the optimization API on the back-burner? Or maybe you are batching up a bunch of changes? Is it on github? Maybe I could submit a request there?

Hi Ernesto,

Any feedback on a fix? Seems really straightforward.

Thanks,

Grant

We resolved this problem by changing the way we model EMC's transaction. Rather than model it as a cash dividend and a stock dividend, it is now modeled as a regular delisted security. Your algorithm gets an all-cash payout at the closing price, just like we do for delistings. Of course this isn't a perfect representation of the transaction, but it is a good analog on the important aspects of calculating returns.

None of the can_trade options you suggest are available; the problem occurs well before it gets to the orders being placed. The optimization routine can't construct an optimal portfolio when the portfolio includes a security with an unknown (and unknowable) price. My favorite quote about software development: "Seems really straightforward."

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Thanks Dan,

What about live trading at IB? If I'm thinking about this correctly, if EMC were held the prior day at IB, then it would magically be converted to DVMT at the open the next day, right? And it would appear in the portfolio. Wouldn't there still be a problem, under live trading with a broker? There would be no price until the first trade, so presumably, the optimization API and ordering would choke, right?

Yes, there is still an open bug about this particular case.

If you are integrated with a broker, and if your portfolio holds an equity that is listed on a major exchange but has never traded, and you run the optimization API, the optimizer can't handle it and will throw an exception.

Thanks. I suppose it tries to look up the price, but there is no key; the sid doesn't exist, or something like that. Maybe a brute-force try-except? Anyway, for me, the broker use case is not of concern presently; just thought I'd point it out, since it came to mind.