problem understanding context.portfolio.cash and capital_used

I have been watching the shared algorithms that are posted. It is fun and interesting to see the ideas people come up with.

However I'm also starting to see a trend where the comment replies inevitably say something like "the algorithm borrows wildly, many times beyond the starting cash". These sort of comments are useful feedback but also distract from the main idea being shared.

So I started to investigate the reason behind the massive over-spending by algorithms.

The documentation lists two properties of the portfolio object that should help determine how much cash has been used versus how much is available for spending: context.portfolio.cash and context.portfolio.capital_used.

I have tried to understand how to use these variables but I simply cannot make them work for me. They seem to only be useful in a strictly long portfolio. Adding in short positions causes their usefulness to evaporate.

In the attached backtest I show a portfolio composed of equally balanced long and short positions. Each day $10k is added to both the long and short side. Each month the overall position is closed out and started all over. I plotted context.portfolio.cash and capital_used and found that neither one changed meaningfully over the period of the backtest. Even though the algorithm was investing$20k per day there was negligible change.

I now believe that the main culprit is context.portfolio.capital_used. The formula for calculating the amount of capital used must be incorrect. Once it is fixed then context.portfolio.cash should also start working (since it likely uses capital_used as an input).

I came to this conclusion by calculating my own version of context.portfolio.capital_used which you will see in the backtest as abs_cap_used. If you look at the custom recorded variables graph the only useful information is from abs_cap_used and abs_cash.

If anyone has the patience and willingness to review the source code I would appreciate any feedback or corrections.

If the Quantopian team sees this please consider changing the formula for context.portfolio.capital_used (or adding a new property that reflects the modified calculation).

Thanks,
Dennis

28
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
 Returns 1 Month 3 Month 6 Month 12 Month
 Alpha 1 Month 3 Month 6 Month 12 Month
 Beta 1 Month 3 Month 6 Month 12 Month
 Sharpe 1 Month 3 Month 6 Month 12 Month
 Sortino 1 Month 3 Month 6 Month 12 Month
 Volatility 1 Month 3 Month 6 Month 12 Month
 Max Drawdown 1 Month 3 Month 6 Month 12 Month
def initialize(context):
set_commission(commission.PerTrade(cost=0))  # ignore commission costs for clarity
context.daycounter = 0

def handle_data(context, data):
# use virtually identical securities for net zero position
spy = sid(8554)     # SPY, S&P 500
ivv = sid(21513)    # IVV, S&P 500

# go long and short the same amount
order(spy, -10000 / data[spy].price)   # short $10k of SPY order(ivv, +10000 / data[ivv].price) # long$10k of IVV

# record built-in portfolio variables: cash and capital_used
record(cash = context.portfolio.cash)
record(capital_used = context.portfolio.capital_used)

# calculate capital_used the Quantopian way: -amount*cost_basis
pos = context.portfolio.positions
q_capital_used = sum(-pos[s].amount * pos[s].cost_basis for s in pos)
record(q_cap_used = q_capital_used)

# calculate capital_used using absolute amount: abs(amount)*cost_basis
pos = context.portfolio.positions
abs_capital_used = sum(abs(pos[s].amount) * pos[s].cost_basis for s in pos)
record(abs_cap_used = abs_capital_used)

# calculate free cash: starting_cash + pnl - capital_used
port = context.portfolio
abs_cash = port.starting_cash + port.pnl - abs_capital_used
record(abs_cash = abs_cash)

# close all positions once per month
context.daycounter += 1
if context.daycounter > 20:
for stock in pos:
order(stock, -pos[stock].amount)
context.daycounter = 0

This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.
11 responses

Hi Dennis,

I took a quick look at Zipline (https://github.com/quantopian/zipline/blob/master/zipline/finance/performance.py) and found:

 def __core_dict(self):
rval = {
'ending_value': self.ending_value,
# this field is renamed to capital_used for backward
# compatibility.
'capital_used': self.period_cash_flow,
'starting_value': self.starting_value,
'starting_cash': self.starting_cash,
'ending_cash': self.ending_cash,
'portfolio_value': self.ending_cash + self.ending_value,
'cumulative_capital_used': self.cumulative_capital_used,
'max_capital_used': self.max_capital_used,
'max_leverage': self.max_leverage,
'pnl': self.pnl,
'returns': self.returns,
'period_open': self.period_open,
'period_close': self.period_close
}


Grant

Thanks Grant. I had tried to find 'capital_used' with a Google search but it didn't locate this file.

I think the portfolio object is set in its own function. I wasn't able to use the properties from core_dict in my algorithm (e.g. max_leverage).

def as_portfolio(self):
"""
The purpose of this method is to provide a portfolio
object to algorithms running inside the same trading
client. The data needed is captured raw in a
PerformancePeriod, and in this method we rename some
fields for usability and remove extraneous fields.
"""
# Recycles containing objects' Portfolio object
# which is used for returning values.
# as_portfolio is called in an inner loop,
# so repeated object creation becomes too expensive
portfolio = self._portfolio_store
# maintaining the old name for the portfolio field for
# backward compatibility
portfolio.capital_used = self.period_cash_flow
portfolio.starting_cash = self.starting_cash
portfolio.portfolio_value = self.ending_cash + self.ending_value
portfolio.pnl = self.pnl
portfolio.returns = self.returns
portfolio.cash = self.ending_cash
portfolio.start_date = self.period_open
portfolio.positions = self.get_positions()
portfolio.positions_value = self.ending_value
return portfolio


The code also reveals the calculations for capital_used (period_cash_flow) and cash (ending_cash).

I found the code for context.portfolio.capital_used (aka period_cash_flow). It is adjusted by subtracting the net sum of transaction dollars. So that would cause short orders (negative dollars) to mask long orders. And it's also why it is overall negative instead of positive in long positions. This matches my hypothetical q_cap_used equation in my backtest.

    self.period_cash_flow = 0.0               # initialized to zero
self.period_cash_flow += cash_payments    # dividends
# subtract transaction spot price x number of shares
self.period_cash_flow += -1 * txn.price * txn.amount


I also found the code for context.portfolio.cash (aka ending_cash). As expected it is calculated from period_cash_flow:

    self.ending_cash = starting_cash   # initialized to starting_cash
# sum of starting_cash and 'capital_used' (aka period_cash_flow)
self.ending_cash = self.starting_cash + self.period_cash_flow


There doesn't seem to be a zipline calculation for absolute capital used and remaining cash. So I guess I will just have to keep using my home-brew solution:

    # calculate capital_used using absolute amount: abs(amount)*cost_basis
pos = context.portfolio.positions
abs_capital_used = sum(abs(pos[s].amount) * pos[s].cost_basis for s in pos)
record(abs_cap_used = abs_capital_used)
# calculate free cash: starting_cash + pnl - capital_used
port = context.portfolio
abs_cash = port.starting_cash + port.pnl - abs_capital_used
record(abs_cash = abs_cash)


@Quantopian: Can you address this in zipline?

I've been bitten by this on essentially every algorithm I have designed or considered, it should be something that I can set as a startup parameter and then not have to worry about. Something similar to the max_notional and min_notional that you recommend in your algorithms, but enforced by the backtester itself versus reinventing the wheel every. single. time.

Ryan, do you think Grant's homebrew solution is the right one? Or is there a different, similar calculation?

Once we have the useful calculations identified, figuring out how to do limits can be next.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Dan,

I think it was Dennis C who posted a "homebrew solution," not me.

Grant

It looks correct, but the other part of the problem is that I don't know what it's supposed to look like. I'm a n00b at all of this, I can understand the idea of notionals and limits, but I can't seem to make them work when it comes to application.

I've done the equivalent of these calculations before, but the numbers never added up. If I update "cash on hand" like this, the values don't match the zipline cash-used values due to slippage / commission / other:

self.cash_on_hand = self.cash_on_hand - (data[stock].price * order_size)


This is the idealized equation if I'm pursuing a long-only strategy (it scares me to bet money I don't have), but it leaves out the other bits. I can't even get the numbers to balance between cash_on_hand and some summation of portfolio.* or a combination thereof. If I want to add shorting into the mix, it gets even worse. Multiplying by cost_basis seems like a good idea, but again, I don't know if that's correct.

My ideal situation would be:
- Set the initial cash-on-hand to a fixed value, defaulting to portfolio size.
- Make that number available in the init function so other vars can be set relative to it (i.e. short_limit = start_value * 0.5).
- Set the short limit to a fixed value, defaulting to zero.
- Have zipline process transactions as it does now, but tracking both long positions and shorts, separately.
- Reject transactions that would violate either limit (OVER_MAX_LONG, OVER_MAX_SHORT).

Without having these kinds of limits enforced in the engine, I don't see any point to specifying an initial investment amount. That only changes where you put the zero-mark, it doesn't change the behavior of the backtester in any way. With these limits in place, things like credit, dividends, commissions, slippage, etc. could all just affect the cash_on_hand value, and the engine would respond appropriately.

Want to short something? Check your credit (MAX_SHORT), and if you have "cash" put through the transaction.
Get a big dividend? cash_on_hand++
Long on 10 stocks and short on 6? The numbers still add up (and the limits still work!) because you're under MAX_LONG and over MAX_SHORT.

Making this happen I leave as an exercise to the reader - I certainly don't know how to do it, or I would've gotten at least one of my algorithms to reconcile. =)

@Dan and Ryan. My understanding is also limited as I'm new to this as well.

I posted a rather detailed explanation of how I believe margin works in another thread. That is based on Reg-T Margin which is going to be the case 99% of the time. See the margin description at IB.

The other kind of margin is Portfolio Margin which has a minimum account requirement of \$110k and very different rules (certain baskets of securities are considered risk balanced and available margin goes way up).

Note that IB has additional margin available for same-day positions (details vary depending whether it is long or short and stock price). But the end-of-day requirement is always the same: 50% margin for Reg-T accounts. So I just use 50% margin as a simple, easy to understand default. (There are some exceptions: leveraged ETFs have less margin available).

One nice thing about having extra margin for same-day positions is that we don't have to worry as much about the order of buys and sells. Meaning that your algorithm can issue a sell order and a buy order at the same time and not care if the buy order executes first (unless the orders are huge). That is to say if you are aiming to maintain a 50% margin level it's okay if you accidentally rely on another 15% of margin before it is returned via the accompanying sell order. This greatly simplifies the logic in portfolio rebalancing strategies for instance.

Because of all of this I think any backtester imposed limits should be configurable (just like we do with commission and slippage model). But I think the default model should be an end-of-day 50% margin limit.

Even without enforced limits we need accurate variables for capital used and available cash. From that we can calculate buying power and margin levels.

My home-brew calculation determines the capital_used variable first and then the available cash. This is only accurate at the start of handle_data and assumes no open orders (which tie up available cash and reduce new buying power). Typically I take that 'cash' variable and add or subtract any buys and sells for orders sent by my logic. That way I know how much unencumbered cash I have at any particular moment. Since I recalculate available cash every event I don't worry too much about commission fees.

The bottom line is the existing portfolio variables 'capital_used' and 'cash' are not useful for Reg-T margin calculations. And I seriously doubt they are even useful in a Portfolio Margin account. Making the change in calculation suggested by my home-brew formula would not impact the use of these variables in a non-margin account (e.g. long only position). And the change would make the variables suddenly useful for Reg-T long/short positions.

Hi Dennis,

What's the status of this effort? It sounds like you are on the right track for capturing the regulations and the Interactive Brokers rules.

I agree that Quantopian should consider some default settings that would conform with limits on a typical account. Perhaps some sort of warning could be provided (e.g. on the returns chart, highlight time periods over which the rules were violated).

Grant

@Grant, from Dan's comment above it seems like they are willing to listen. But as users we have to identify our needs clearly. I think I've gone a long way in that direction. However I'm not an expert and I may have overlooked some things.

It would be useful if someone here with more of a financial / broker background could weigh in. I'm coming at things from the perspective of a programmer so while it may seem logical to me that doesn't make it necessarily so.

Has there been any update to this topic recently? I asked a similar question in relation to live trading, however I haven't been able to find a solution. My issue is that there doesn't appear to be a way to limit your starting_cash in live trading that I can see - this makes it a little tricky to run an algorithm if you don't want it to be your only one!

In addition, your capital_used parameter doesn't seem to increase as the algorithm accumulates profits, so I'm doubtful that this issue has been fixed yet?

Any updates on this? I'm working on an algo that often trades somewhat illiquid equities and constantly having problems with a negative cash balance as open orders do not affect the available cash.

I've figured you could write a function to tally up the cost of all orders but that brings forth another problem. I don't think there's a way to find the limit price on a given order, only the sid and amount of shares. You could keep a list of orders and their prices as you place them but that's very cumbersome.

Anyways; thought I'd check in and see if this has been worked out yet, thanks!