Futures Have Launched - Research, Backtesting, Lectures, Tutorial, and More

Futures are now available in backtesting. You can now access futures data and trade contracts. You can implement algorithms that trade futures alone, or algorithms that trade both futures and equities. New data and new tradable assets means new opportunities to find alpha. If you have your eye on an allocation, this is the first opportunity for futures algos to be evaluated on Quantopian, so correlation to other algos is as low as it will get.

We didn't just expand the API, we launched a new curriculum to help you learn about futures. We added a tutorial to help get you started with the new Futures API, and a handful of lectures including an introduction to how futures work and some advanced lessons on trading futures quantitatively.

There's a lot of new stuff, so here are some more details on what we've added.

Futures Data in Research

Futures data for 72 different US futures is available in Research for you to analyze and formulate ideas. The data includes 24 hour as-traded pricing and volume data, 5 days a week. Similar to what we have for equities, futures data is available at a daily or minutely frequency and includes open, high, low, close, and volume data (OHLCV)

The Futures API includes continuous futures. You can use continuous futures to request for windows of pricing/volume data for a particular underlying commodity/asset without having to worry about all the different contracts trading simultaneously for that same underlying.

In case you missed it, we posted an example notebook that walks through the new API.

Futures Available in Backtesting

Futures data is now available in backtests so you can now develop algorithms that trade futures contracts. Similar to research, your algorithms can now access 24 hour futures pricing/volume data at a daily or minutely level. In order to get futures data in an algorithm, you will need to run your backtests on the US Futures trading calendar. There is a new dropdown menu where you can pick between the US Equities or US Futures calendar. Equities data is available on either calendar but futures data is only available on the US Futures calendar. The US Futures calendar runs from 6:30am-5pm ET. The US Equities calendar is the 9:30am-4:00pm ET calendar that you are used to on Quantopian.

Continuous futures are available in backtests. The same function that allows you to build continuous futures in research is available in the backtester. You can get the history of a continuous future via data.history().

You can trade futures contracts in your algorithms. A very common technique to ordering futures contracts is to use the continuous future to get a reference to the current active or front contract for a particular underlying commodity/asset. To do this, you can pass a ContinuousFuture to data.current(), and ask for the 'contract' field. For example, if you want the contract to which a crude oil ContinuousFuture is current pointing, you can do something like this:

context.cl = continuous_future('CL')
...
current_contract = data.current(context.cl, 'contract')


Once you have the current contract, you can place an order for futures contracts using order_optimal_portfolio.

Lectures

We added 4 new lectures to the Quantopian Lecture Series to teach you about futures, and how to think of them in the context of quantitative finance:

The lectures are a great way to learn about some of the theory behind trading futures. We are expecting to add more lectures in the near future.

Tutorial

We launched a Getting Started With Futures Tutorial that teaches you how to use futures data in research and how to include futures in your algorithms. The tutorial walks through the new Futures API in Research and Backtesting and builds up to a simplified version of the Pair Trading Example in the Mean Reversion on Futures lecture.

Help Documentation Updated

We added the new Futures API to the help documentation so you can learn about the new functions that we added and how to use them. The existing documentation was updated to explain how certain functions work on the US Futures calendar (e.g. schedule_function()).

Slippage & Commission

We made new slippage and commission models for algorithms that trade futures. When you run backtests on algorithms that trade futures, they will use per-future models with parameters that depend on the particular underlying commodity/asset being traded.

The default commission model used for futures on Quantopian is Interactive Broker's fixed commission model and includes per-trade exchange fees.

The default slippage model used for futures is a special volatility volume share model that was fit to empirical data.

Note that if you run a backtest on an algorithm that trades equities and futures, the equity slippage + commission models will be applied to your equity trades, while the futures models will be placed to any contracts that you order.

We look forward to working with the community as you develop strategies that trade futures. Over the next few months, we will start to evaluate these strategies. We look forward to making our first allocation to a futures algo!

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

73 responses

any way i can use pipeline in futures mode? currently i got the following error:

CannotUsePipeline: Pipeline is not yet supported on the Futures calendar. You can use Pipeline by switching to the US Equities calendar. If you would like to run an algorithm on the futures calendar, you will have to remove Pipeline from it.

Hi Neo,

Unfortunatley, Pipeline isn't yet supported on the futures calendar, so futures cannot be accessed via pipeline. I'm curious, are you looking to use futures data in an algo that trades equities via pipeline, or were you looking to write a cross-sectional futures strategy using the Pipeline API?

Hi Jamie,

Thanks for the prompt reply. I am trying to write a simplistic algo that trades VIX futures and needs VIX spot prices as part of my signals. I guess one way to get around the pipeline issue is to use fetch_csv instead.

Been looking forward to futures implementation for a long time. Thanks for this Jamie & team.

Is there any data available for Indian markets?

Hi Jamie,

I want to try out latest version of zipline from github for backtesting futures. I have futures data with roll date and spread info. I was able to ingest dividend and split info for equities by supplying dividends and splits dataframes to adjustment_writer. How can I ingest futures data with roll date and spread info in zipline?

Can one trade futures and stocks simulatenously?

Hi Jamie,

Another question. Is there a way to construct continuous futures based on a customized roll rule (for example, always roll X business days before expiration)? It seems to me that right now there are only 2 ways to roll the futures, by volume and by calendar.

@diwakar: We currently have data for US Equity and US Futures. Going forward, we will be looking to add data for international markets including India.

@Mayur: I can help you with any questions about Quantopian, but I'm probably not the best person to help you with Zipline questions. Your best bet is to post in the GitHub repository. Our engineers monitor the repo and should be able to answer your questions.

@Tim: Yes, you can trade equities and futures simultaneously on the US Futures calendar. However, pipeline is not yet supported on the futures calendar so you will have to reference equities the old fashioned way (sid(...)).

@Neo: Currently, there's no way to customize the roll attribute of a continuous future, but you can definitely customize how your algorithm rolls between positions. I've attached a backtest that holds a long position in CL and always rolls 2 days prior to the auto_close_date of the front month contract.

The roll attribute on continuous futures defines when the continuous future decides to point to the 'next' contract. There's no way to customize this right now, but it might be something that we look into going forward. Out of curiosity, were you looking to change the rolls for customizing lookback windows, or were you looking to customize when your portfolio moves?

You guys are the best!

Will Q publish criteria for Futures strategies to be eligible for the Contest or Fund? For example, any rules on position concentration, or futures equivalent of low beta / sector neutral?

Dan: Yes, we're working on figuring out what those metrics will be. Right now, we're thinking about risk metrics as leverage doesn't quite apply to futures in the same way it does to equities. We have been talking about using some sort of expected CVaR metric to control how much an algo can include in a portfolio. We're also going to make a contest for futures algos to give you a good sense of the types of strategies we're looking for. I can't say for sure at this point, but I imagine the contest will also prefer low beta-to-SPY, low drawdown, and high sharpe.

Jamie: Do you expect one of the requirements to be also (full) hedging of the long positions with short ones?

I'm don't have a good sense of that one yet, Tim. We will update the community when we have a better idea of what types of futures algos will score well.

Jamie: one idea would be to prefer algos that work equally well across all (or many) futures as possible, especially unrelated ones like copper and corn. This would give some indication of "breadth"

When will we be able to combine pipeline equities and non pipeline futures, in the Alpha I did this successfully but in the live version it doesnt work. I use future term structures to trade stocks so I need to use both

thank, Peter

How do I set stop losses or limit orders?

@Peter: We don't yet have a timeline on when pipeline will be available on the futures calendar. While pipeline was technically available in the alpha, it was not returning results that were properly aligned with the futures calendar. This is something that we need to fix with pipeline before it works on the futures calendar.

Are you hoping to use pipeline for cross sectional equity factors, or are you trying to use it to use VIX data in your algo?

@Cary: Stop loss and limit orders can be specified in most of the order methods. The most commonly used is probably order_target_percent with the style specified as a LimitOrder or StopOrder (reference). That being said, when we evaluate and select algorithms for an allocation, the execution component of the algorithm is expected to be handled by our trading team. The author can focus his/her efforts on creating the alpha signal, timing, and target portfolio, and the trading team will execute the corresponding orders with the prime broker. Because of this, we encourage the use of order_optimal_portfolio instead of the other order methods.

Will it be possible to trade the individual contracts on the curve? So far all the examples included only continuous futures. This is quite a limitation regarding spread trading.

@M. H.: Yes, it is possible to trade individual contracts on the curve. Have a look at the data.current_chain function which gets you the forward looking chain of contracts of a continuous future at a given point in time. You should always use a continuous future to maintain a dynamic reference to a particular underlying commodity/asset. But you must use other functions like data.current or data.current_chain to trade the appropriate contracts associated with that continuous future.

I've attached an example that trades multiple contracts on the same chain. Let me know if this helps.

@Jamie: Thank you very much, that definitely helps!

I wonder how quantopian will support future contracts portfolios in tools like pyFolio. Are you guys working on this?

It's a whole different animal, and there's no much info/tools to handle them properly. In my experience things like calculating a Markowitz mean-variance portfolio based on future contracts it's a hassle because of the integer weights. Any ideas?

Victor,

Great question. We have started working on expanding Pyfolio to analyze algos that trade futures. I don't have any info to share yet, but we are actively working on it. We will provide an update in the community when we know more.

@Jamie that sounds really good. Can't wait!

Sometimes when trading both futures and equitys there is trouble getting data for the equitys.

Hi Cary,

Thanks for reporting this. We are working on a fix and hope to have it patched up soon. Sorry for the trouble.

For all the quantopian data sets like sentiment or some of the cboe indexes, do they all have to go through pipeline? That seems to be the only way I have seen of accessing the data in the examples. I was curious if there was another way to get at that data while using the futures calendar.

Thanks,

Hi Jamie,

Thanks for adding futures. Is it possible to trade live using futures?

Thanks

@Cary: The issues with getting data for equities on the futures calendar should now be fixed. If you are still having trouble, please send us an email at [email protected] describing the problem.

@Kevin: Right now, yes. Unfortunately you can't yet use partner datasets on the futures calendar. However, solving this problem is high up on our list. We certainly want you to have the ability to trade futures using non-pricing data for signal or risk mitigation.

@Ramesh: Futures are currently available in research and backtesting, but not live trading. Right now, we are working on building a contest for Futures which we expect to use Quantopian paper trading.

Hi Jamie,

Thanks so much for adding futures in Quantopian! I'm currently trying to backtest VX futures, however, I noticed that there seems to be a slight issue when I try to reference the "future_symbol" call to an assigned variable rather than just the string name of the future. For example if I backtest using this sample:

context.single_contract = future_symbol('VXM17')
print data.current(context.single_contract,'price')


Everything runs smoothly. However, if I backtest using this sample:

contract_sym = 'VXM17'
context.single_contract = future_symbol(contract_sym)
print data.current(context.single_contract,'price')

Error InvalidFutureSymbolCall: 0077 symbol argument must be a string.

Blockquote

Would you happen to know what is causing this issue for me? I'm testing an algorithm that's not using continuous futures in which case I wanted to specify the different contracts.

Cheers,
Francis

Hi Francis,

Unfortunately the future_symbol method does not accept variable input, only a string literal. Sorry for the inconvenience. A workaround for this problem is to instantiate separate references to each contract:

future_symbol('VXM17')
future_symbol('VXZ17')


etc.

Can you tell me more about the need for specific contracts? It would surprise me if you can't get the contracts you need using a continuous future. Maybe I can point you to an example that helps.

Hi Jamie,

Thanks for the quick response! Currently, I'm just trying to compare days to expiry between the contract chains, however, it seems that all contracts are sharing the same expiry date when I'm pulling the continuous futures chain. I've attached my tester backtest for your reference and you can see in the Logs that every contract seems to be having the same number of days to expiry. I thought that I would need to make a specific contract call to get the longer maturities from today.

As well, I'm having trouble closing out specific contracts held within my context.portfolio.positions. For example, suppose that I have multiple contracts held and want to close out the one with the closest maturity, is there a way to do this without using something like this:

held_contract = map(lambda x: x.symbol, context.portfolio.positions)[i]
order(future_symbol(held_contract), 0)


Cheers,
Francis

Hi Francis,

Thanks for sharing. In general, I would recommend against using the end_date to determine when a contract will end. If a contract is still trading (hasn't expired yet), the end_date will be before expiration. For example, the end_date of a contract with delivery in Jan. 2018 will be today, since today is technically the last date on which there was a trade. If you want to order contracts by their expiration, I'd recommend using the expiration_date or the auto_close_date which are determined when the contract is created in our database.

For closing out positions, you can iterate over context.portfolio.positions to get references to the contracts you currently hold. For example, if you wanted to close out all of the contracts which you currently hold, you can do something like this:

for held_contract in context.portfolio.positions:
order_target_percent(held_contract, 0)


Note that you don't need to get the symbol property from the contract since we already have a reference to the Future object (held_contract). I also changed it from order to order_target_percent since order(asset, 0) tells the program to order 0 shares/contracts of the specified asset.

I hope this helps.

Guys,

If you want to broaden your futures coverage, I would advise that you get in touch with CRB (Commodity Research Bureau, crbtrader.com). I am not affiliated with them in any way, but they were a provider in one of my previous jobs, and they:
a) have great coverage
b) have great service (honestly, it was a pleasure)
c) have futures data nice, commoditized, and easily available

Just FYI.

Kind regards,

Wojtek

Hi guys,

Fore-apology: I'm new to this ("this" being Quantopian). There will be a lot of basic-level questions. Like this one:

Why do futures seem to follow a different coding logic than equities? With equities I can code a transaction in literally one line e.g. order_target_percent(context.NT, 1.0). Why for futures do I have to use a much more roundabout schedule_function + order_optimal_portfolio combo? What if I just want to purchase 1 futures contract?

Many thanks!

Wojtek

Hi Wojtek,

You can use the same order functions including order_target_percent for futures. We chose to show order_optimal_portfolio in the tutorial because it's important to think about weighting assets for your entire portfolio when you are trading more than one asset. Additionally, we are working to add more constraints/objectives to order_optimal_portfolio so that you can easily move your portfolio from one state to another according to different types of logic.

Using order_target_percent is certainly a more simple approach when you are just trying to order one name. But when you want to consider different criteria like risk metrics when moving your portfolio from one state to another, order_optimal_portfolio will be your best friend. Even without adding constraints, you can think of it like a batch order_target_percent for all of your orders on one line.

greeat stuff

Hey guys, is there any chance you will be adding CME Currency futures contracts as well? E.g. contracts like 6E, 6B, 6J, etc. It would definitely help bridge the gap into having access to real forex data. Thanks in advance.

@Evil Speculator:

We would certainly like to keep adding to the list of assets available on Q, but we are not planning on adding more currency futures in the near term. Out of curiosity, what's the functional difference between the 6E, 6B, 6J, etc. and the EU/EC/EE, BP, and JY/JE? (apologies if this is a noob question!). I'm looking to get a better sense of what it is you need that we don't have.

@Jamie - actually I hadn't realized that you had those (EU/EC/EE, etc.). Will check them out for sure :-)

What is the source of those? I have never seen them before. If those are the CME futures under the hood then it would help if you would link some aliases so that they match their CME symbols.

Upon quick inspection, it looks like some of the CME products have multiple symbols. We can certainly consider adding an alias for these assets. I'll notify our product team so they can prioritize appropriately. If you would like to know the asset of a particular contract or continuous future, I'd recommend doing something like this in research:

from quantopian.research.experimental import continuous_future, history

history(continuous_future('EE'), 'contract')[0]


That should print out a the attributes of a contract on the continuous future which will have the exchange as well as the name of the asset.

Hi, does anyone know if is there a way to filter the futures time series to only include certain time like 9:30 - 4pm EST by any chance? Thats after I get the series through history() I am trying to back-test a strategy based on a futures and a stock pair and it seems like I run into problems outside of time when equities are traded. I am doing this in notepad. Or maybe you have a better suggestion how to sync minute data between future and stock series in a notepad. Thanks for your help.

Hey Jamie - you're right about it going by multiple symbols. Never seen those others though on any data provider and I have been trading futures for almost a decade now. So yes, those aliases would be much appreciated as most futures traders are going to recognize 6E immediately but might stumble over EC. Just a thought, it's not a big deal in the end as long as the data is good ;-)

Joseph - well a simple solution would be to do a left join via pandas. It should only grab the rows which have a matching date index (which of course is the equities symbol).

Good Idea Evil, that should work. Thanks!

Another question most likely for Jamie (I know I only get one per day - take it out of my weekly ration ;-)

Is there a way to account for roll cost for futures contracts in backtesting? I am probably mistaken as I'm new to the platform but it seems to me that the continuous contract assumes that positions are rolled for free. We do have roll dates if I understand the pertinent comments above correctly. What would be great is an estimated cost per contract which would be added to open positions on roll-over dates.

Also, I take it for live trading the continuous contract won't do for order submission, right? I assume it needs to be the front month.

@Evil:

Rolling contracts in your portfolio is actually separate from the roll attribute on continuous futures. When you order contracts in the backtester, you do so by getting a particular contract from a continuous future, and then ordering that contract. Typically, the algo will maintain a reference to a continuous future and use that to pick the appropriate contract to trade on a given day. In this process, you sometimes end up moving your portfolio from one contact to the next. When this happens, commissions and slippage are applied as normal.

This topic is explored in the futures tutorial. I also discussed it a bit further up in this post, but the tutorial is probably a better reference :)

The roll attribute on continuous futures defines the rules for when a continuous future should start pointing at the next contract in the chain. It doesn't set up any automatic rolling in your portfolio unless you wrap code around it to do so (which is actually a good way of doing it!).

We don't have live trading setup with futures yet, but when we do, I imagine it will work like the backtester where you have to place orders for specific contracts. This would not be limited to the front contract.

@Jamie: Do you know if there is a way to get data for continuous future but for 2-3 month period not just 1. So fo example if I want to look at rolling data for month 6 and 7 so essentially restarting every 2 month instead of 1. Thank you!

Hi Joseph,

Unfortunately there isn't a way to specify that right now. That's certainly an interesting feature request. That being said, there are some futures that don't roll every month, so I would recommend looking into the data to see if it lines up with what you're looking for.

hi Jamie,
thanks for posting this information. I was wondering if we can have a draft of an algorithm like the multifactor for futures?
Ioannis

I love the analysis provided here for the options and futures. I take tips from here to help my students at Stock Market course understand the concepts better.

@Jamie Do you know by any chance if you guys support weekly futures ? specifically vix weekly futures and if you do what are the tickers. Thanks.

@ioannis: We don't have a futures template like the long-short multi-factor one with equities, but we are working on one. I just posted a set of allocation criteria for futures algos today. Hopefully the description there can help guide you in your algo design.

@joseph: The full list of supported futures can be found here. We do not currently support weekly futures, sorry for the inconvenience.

Any word on when futures will be available for live trading?

Co-asking: when will futures algorithms be allowed to enter contests? Thanks!

We don't have a timeline available for either live trading or the contest, but I can tell you that we have started to work on risk metrics for futures algos in anticipation of a contest. We also recently published allocation criteria for futures algos, so you should check those out if you want a sense of what the contest guidelines will look like (or if you are interested in an allocation!).

@Jamie could we get the futures template soon. More than the futures contest I am looking forward to some real examples to get started with futures. Some pointers/links/reference material/books etc. on how to transition from equity trading strategies to futures trading would also help.

Yes, Can you see me?

Is there a general way to get the underlying current value for the index that the future is for? It seems that some simple strategies could make use of simply knowing the spot value vs. the future value.

When will we be able to use pipeline data with futures?
CannotUsePipeline: Pipeline is not yet supported on the Futures calendar. You can use Pipeline by switching to the US Equities calendar. If you would like to run an algorithm on the futures calendar, you will have to remove Pipeline from it.
There was a runtime error on line 23.

And when can we use Futures on the US Equities calendar? I have some strategies that involve trading between futures and equities together - for example SPX and /ES correlations.

At quantopian, the US Futures calendar runs from 6:30am-5pm ET. The current CMEGROUP products (e.g. forex / gold) are mostly trading at US central time 1700 - 1600 (e.g. almost 24 hours). Before quantopian can fix this, is there anyway (or workaround) that i can use my backtesting / paper trade using the real cmegroup trading hour ? Thanks.

Is there an example alogirthm available that lists all the supported futures in a ready-to-trade manner, so that an algo can be written that invest in all of them simulatenously, applying a certain ranking scheme for determining long and short positions, much like this is done with a long-short equity strategy?

@Jamie: Experimenting with the futures API, I'm running up against the same restrictions as Kevin, Randy, Neo, and some others. My first idea was to create an algorithm hedging Q1500US equities and futures against one another, but I can't do that. My next idea was to use spot prices from the datasets to help trade futures, but that is also not possible at the moment.

The issue is the combination of (1) having pipeline as the only way to access datasets and (2) having pipeline unavailable on the futures calendar. Maybe you could resolve (1) more easily than (2)? It's anyway cumbersome and inefficient to have to write a CustomFactor to access asset-independent data like the VIX. It be way nicer if we could reference cboe_vix.value anywhere in our code, instead of just in CustomFactors.

Still, awesome to have futures on here now -- I'm looking forward to additional updates.

Any updates on if futures are allowed for real trading or the contest yet?

Any updates on lifting the restrictions to futures trading? Can we at least trade futures on the equities calendar so we can access pipeline? I'm fine with my algorithms only trading futures during the equities calendar.

@Tony real trading has been shut down.

@Peter do you know why?

My opinion (and I have some data to back this up) is that most of the people live trading were trading in a way that was not consistent with the way Quantopian wanted them to trade. They were trading things that made them money but would not be considered by Quantopian for one of its allocations. So it was mostly a win-win for Quantopian, they no longer had to support the code/service and a few users may have left. The users who left were perhaps not focused on the competition but perhaps they were spreading trading ideas/strategies that were never going to be considered for an allocation.

Is there any way to set futures commission and slippage costs to zero for a backtest?

Ryan, you can set futures commission and slippage costs to zero for a backtest with these lines:

set_commission(us_futures=commission.PerFutureTrade(0))
set_slippage(us_futures=slippage.FixedSlippage(0))


Thanks Jamie!

Hello, I am still learning this language.
Is there a way to roll 10 different futures without having to write the same codes 10x?

Thank you.