Back to Community
Futures Data Now Available in Research

Update: Futures are available in Research and Backtesting. We also added a tutorial and futures lectures. For more details, see this post.


Today, we added futures data to Quantopian Research for 72 U.S. futures going back to the start of 2002. Data includes daily and minutely open, high, low, close, and volume (OHLCV) data for 24 hours x 5 days a week and is collected from electronic trade data. The list of available futures can be found at the bottom of the notebook.

Along with the data, we added some new API functions to the quantopian.research.experimental library to make it easier to interact with this new asset class. Attached is a notebook which gives a brief introduction to futures data on Quantopian as well as the new futures-related API.


Highlights from the notebook include:

Pricing and Volume Data for Futures Contracts: The notebook includes examples of pricing and volume lookups for particular futures contracts.

Continuous Futures: A continuous future is a specifier for retrieving a chain of contracts all associated with the same root symbol, and the dates at which activity has rolled from one contract to the next. Continuous futures maintain a reference to the active contract (or history of active contracts) for the particular underlying.

Continuous Futures vs Individual Contracts: The notebook demonstrates the value of continuous futures by comparing them to the the individual contracts they comprise. This comparison includes a look at price adjustments when a continuous future rolls from one contract to the next.

Term Structure: By comparing contracts with consecutive delivery dates, the notebook looks at the term structure of a particular underlying.


Many of you will be new to futures. We are actively working on lectures and tutorials that will cover the basics of futures, the new Quantopian API, and some more advanced statistical content for futures that you're already used to seeing for equities in the Lecture Series. This notebook assumes a basics knowledge of futures. Until we have more educational content on Quantopian, here are a few resources that I found to be helpful when learning about futures:

Introductory
- Forward and Futures Contracts, Khan Academy (video)
- Futures, Investopedia (article) -- there are a lot of other good articles about futures on Investopedia too

More Advanced
- Following the Trend, Andreas Clenow (book)


P.S. We are planning to launch the full futures product, including backtesting, documentation, and lectures at QuantCon.

Loading notebook preview...
Notebook previews are currently unavailable.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

38 responses

Thanks for implementing this. It will definitely bring in a lot of new insight and improve a lot of existing algorithms.

Just one question.... If minutely data will be available for backtesting, does this mean that live (deployed) algorithms will be able to look at live data (rather than 10-20 minute delayed, as futures such as $VX are)? Or, will all minute-resolution backtests need to be run with timeshifted data?

That's a great news. When you will implement European Futures as well?

The Bean Report - a quick look at bean futures on Q:

p.s. most code borrowed from Q tutorials.

Loading notebook preview...
Notebook previews are currently unavailable.

Are there any sample codes based on reading futures data?

@John, Are you asking about Quantopian paper trading? Currently, algorithms that are broker backed (paper or real $) all receive live updating data each minute. I'm not yet sure what paper trading will look like in terms of delayed data for futures. At the very least, a delayed algorithm will not run with timeshifted data. If it's delayed, it simply runs 15 minutes delayed. For example, the first bar (9:31) is simulated 15 minutes later (9:46), but runs as though it's the first bar of the day, and uses data from the first bar (9:31).

@Vincenzo, No timeline yet, but we definitely want to expand on the current list of available futures.

@Frank, That looks like an awesome study! Have you considered making a standalone post for it? You might have an easier time getting eyes/feedback on it. Also, have you considered generalizing the study by using ContinuousFutures?

@Macro Investor, If you're asking about backtesting, futures are currently only available in research. We will be launching futures in backtesting, as well as tutorials, lectures, and documentation at QuantCon.

@ Jaime

It was just a Saturday afternoon project because I was excited to see that futures were available in research. I will create a stand alone post next time I go back to the study and enhance it. Will also consider generalizing with Continuous Futures. Thanks again for Futures. Awesome to have this available !!!!

Frank, this is a really cool notebook. I only got through a quarter of it and even that was a ton of cool content.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

@ Delaney, thanks, will make a stand alone post after fixing some issues and adding some features to the model. I have not checked the Q Data availability, but am wondering if any agricultural or weather data is available?

Sounds like a plan, Max may even be interested in borrowing snippets to use as exercises in our upcoming futures lectures. He'll reach out to you if so. I don't believe we currently have that data, but it is certainly in the realm of things we would like to integrate at some point. Seong would know better about this.

Sounds good!

We're always on the lookout for future partners. We've spoken to a few agricultural providers as well - and if you know any potential data firms that provide that data, you can send them my way. SLEE @ quantopian.com

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

For continuous_future, it will be useful to have a bit more flexibility in specifying the roll, e.g. number of days offset vs the auto_close_date in the case of calendar, and/or rolling a deferred future on the same day as the nearby future. This is because if I want to look at strategy of futures spreads, then having them all roll on their respective expiry severely restricts what I can do.

Hi Han,

Thanks for the feedback. Continuous futures are not assets that can be ordered, so there won't be any automatic rolling from contract to contract. To roll from one contract to the next, you will be able to get the current contract of a continuous future, as well as the next contract (from a function that will be available in backtesting). So you can decide when you would like to roll from one contract to the next. The roll logic in continuous futures is simply for looking at price histories.

That being said, I recognize that being able to set a custom roll logic (both for historical time series' of data, and for rolling from one held contract to the next), so that's something we may add going forward.

Are there any sample codes available to clone?

Futures are not yet available in backtesting. The launch of futures in backtesting will happen at QuantCon at the end of April at which point there will be sample algorithms to clone.

Very happy to see VIX Futures included. Excited for full launch

If I live trade via Robinhood (which does NOT support futures), will my algorithm even be able to look at live futures data? I will need to use Interactive Brokers to facilitate this, right?

Is it normal that VIX prices are available from July 2012 only? Will earlier data be added at some point?

@everyone: Futures have launched in backtesting, check out the official announcement.

@John: Futures are available in research and backtesting right now. I'm not yet sure how futures data would interact with Robinhood accounts.

@Samuel: I was a little surprised by the start of data in Jul. 2012 for our VIX data as well. Our provider only started collecting pricing and volume data for VIX futures in 2012 it seems. I actually called the exchange and spoke with a few people trying to figure out why this may have been the case. From what I understand, VIX futures techincally started trading in 2004, but they were quite illiquid for a long time. My best guess now is that there just wasn't enough activity for our vendor to get data prior to 2012. We don't have immediate plans to add more data for VIX futures, but in the long run, we would like to be able to have histories going back to 2002 for all the instruments we added. I'm working on a notebook to show the 'lifespans' of the futures we have in our database which should paint the picture nicely.

@Jamie Thanks for the update. It would be definitely interesting to have VIX data around 2008 for the stress-testing of volatility strategies.

@Jamie I'd like to bump up what @Han Zheng wrote above.
There very much ought to be a way to build a continuous contract based on calendar but able to roll +/- N days before the auto_close_date. I imagine most interest will be in rolling a few days earlier. There are many reasons to do this, but among the most important are that for physically delivered commodities, those dates can be particularly volatile in ways that many traders prefer to avoid.
Is there a plan to do this?

@knuckledragger: I definitely agree with you and Han. I actually posted an example algorithm that rolls a continuously held position in CL 2 days before the auto close. The key to note with continuous futures is that they don't actually determine any sort of rolling of your portfolio. The auto_close_date of a contract is the only attribute that actually takes any sort of automated action on your positions. Even if you use a continuous future that roll before the auto close, you'd need to include some logic in your algo to close out your existing position.

I'm hoping the example I shared in the other thread demonstrates how you can customize the roll behavior in your algo. Let me know if you find it helpful.

Thanks, Jamie. I'm more interested in the behavior of the continuous contract for right now. I am using the research / ipython notebook rather than the backtester.

Ah, I see. A limitation for continuous futures right now is that they can't really be used to simulate a return stream if your algo roll logic doesn't line up with either of the default rolls on continuous futures. At some point, we'd like to build out that sort of customizability. In the meantime, the backtester is your best bet at getting a return stream since it computes returns based on the contracts you hold.

Note that you could then pull the return stream back into research via get_backtest().

This is my first time on Quantopian and this was a great place to start. this was awesome!

I modified the original notebook a bit to include a section that looks at the spread between Crude Oil and Gasoline. There's also a tearsheet for the final algorithm in the Futures Tutorial which runs a pair trade between these two futures.

Loading notebook preview...
Notebook previews are currently unavailable.

Here is the backtest that was used to generate the tearsheet in the above notebook. It was copied from the Futures Tutorial.

Clone Algorithm
28
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 59038d88b5d7f761e170ab62
There was a runtime error.

Hi @Jamie, how to backtest if the data is outside of the trading calander?

I tried to test grians in IDE, but seems it can just be traded from 6:30-5pm(future). Since the soybean's trading time is 19:00-7:45 & 8:30-13:20.....So every day it miss 10-hour data (from 19:00 to 05:30 UTC) when backtest in IDE..... I want to know how can I add those missing data by setting trading time frame? otherwise the data set is not continuous, and is hard to do algo...

Hi Manlin,

Right now, trades can only be scheduled between 6:30am-5pm Eastern Time. While this doesn't capture the entire trading window for corn, it captures the vast majority of it (see attached notebook). You can place trades any time during that window.

Additionally, you have access to 24 hour data. If you want data from 3:30am-6:30am ET, you can make a data.history call with 'm' frequency for 180 minutes at 'market open' of the US Futures calendar. The 6:30am-5pm restriction is only on schedule function so returns and historical lookups will be continuous.

There are some strategies that depend on executing trades around the clock. Unfortunately, those can't be implemented with the current US futures calendar. We would like to extend the calendar at some point in the future to support these types of strategies. In the meantime, we're hoping that there are still a large number that can be written that place trades in the 6:30am-5pm window.

Does this help?

Loading notebook preview...
Notebook previews are currently unavailable.

@Jamie Thanks for your quick reply . It's helpful to me!

I'm getting nans in the NG futures when using data.current for price vs data.history. Does anyone know why?

            print(data.current(sec,'close'))  
            print(data.history(sec, 'price', 1, '1m')[0])  
2011-11-28 06:31  PRINT 3.714  
2011-11-28 06:31  PRINT 3.714  
2011-11-29 06:31  PRINT nan  
2011-11-29 06:31  PRINT 3.543  
2011-12-27 06:31  PRINT 3.128  
2011-12-27 06:31  PRINT 3.128  
2011-12-28 06:31  PRINT 3.137  
2011-12-28 06:31  PRINT 3.137  
2012-01-26 06:31  PRINT 2.826  
2012-01-26 06:31  PRINT 2.826  
2012-01-27 06:31  PRINT 2.667  
2012-01-27 06:31  PRINT 2.667  
2012-02-24 06:31  PRINT nan  
2012-02-24 06:31  PRINT 2.776  

Hi Honver,

The close is the close price from the previous minute bar, but is not forward filled which is why you see the nan. The price is also the close price from the previous minute bar but it is forward filled.

I hope this helps.

@Jamie after I clone the algorithm "Futures Trend Follow " available in section "Sample Algorithms" at https://www.quantopian.com/help and run a backtest starting with 1/1/2017 I get following Error Message:

NoDataAfterDate: No data on or after day=2017-02-23 00:00:00+00:00 for sid=1034201612
There was a runtime error on line 63.

Could you please help?

^ This error has been there for a month or so, Q said they are working on this, however, there seems to be no progress...

Yes, I've been waiting for several weeks for this issue to be fixed. But its still been unresolved.

I'm thinking to switch over to Zipline, the open source backend of Quantopian, but I found that its lacking 1) ingestable Futures data, and 2) examples to get started. It seems these requests have been in GitHub dating back to 2015 with no resolution.

Its frustrating there's no straightforward way to make progress with Futures backtesting and trading. At this point, I'm getting the feeling Quantopian isn't taking Futures trading seriously.

I would like to see Futures on Zipline.
@Jeremy Cohn, Quandl does have free Futures data. You could download that and load it into a Zipline bundle.

@Peter Harrington,
Thanks, yes I am aware of those. Unfortunately Quandl only offers daily bars, but I'm an intraday trader and require minute bars. In any case, yes, I'll have to access minute bars elsewhere and figure out how to ingest into Zipline.

When querying historical data on continuous futures with data.history, the closing price for the same date can be different on different backtesting dates. Does anyone know if historical pricing on continuous futures is somehow dynamically calculated? This unexpected behavior of unreproducible pricing can make debugging code more challenging. If this is not a bug, then it would be very helpful to have an option that ensures pricing reproducibility over time. Attached is an example backtest showing this effect in the log output, also reproduced here:

2016-03-24 12:00 fetch_prices:33 INFO query on 2016-02-17 has a price of 1885.25 on 2016-02-15 for ES
2016-03-24 12:00 fetch_prices:35 INFO query on 2016-03-21 has a price of 1876.60 on 2016-02-15 for ES

Clone Algorithm
0
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5996048d702753510cdacf8a
There was a runtime error.