Back to Community
contributors and reviewers needed for Zipline-Live

Hi beautiful people,

As you know the engine of Quantopian is zipline and there is a branch/fork "zipline-live" (www.zipline-live.io) that makes it possible to live-trade your signals with Interactive Brokers. Although the core works we are expanding it with more brokers (Robinhood), a frontend to access performance data, bringing zipline's changes into zipline-live, connecting other exchanges (Cryptos), enhancing the ingestion processes, etc etc,

We are looking for people who can contribute to this awesome project. Skills we are looking for: python,zipline, css, Django,

Cheers, Peter

30 responses

@Peter,
Have been monitoring on the Slack channel, and you've been making good progress there!
Hope to contribute soon, as I now have a potential use-case.
alan

Hi Peter -

It seems like the end game would be to get zipline-live up and running on a hosted server, with integration with IB, Robinhood, etc. and data feeds (i.e. what Quantopian was doing). Quantopian already knows how to do this, so is there a lot of duplication of effort here? Or does "on premise" mean getting it running on a local server (e.g. in one's basement)? If the latter, this would seem like a step backwards.

Also, I'm wondering how much interest there could be in this? My understanding is that Quantopian only had a few hundred real-money traders. Even if they all came back, it wouldn't amount to much participation in the grand scheme of things.

Perhaps you could elaborate on where you see this going?

I'm sure there is actually certainly a lot of interest & a poll could probably confirm that. I would guess a lot more interest to live trade than to participate in the contest. Ever since they stopped live trading, I've wondered why anyone would even want to pay for any of the premium bundles?

Can you outline a bit more as to how the zipline-live solution will look? Will you need to deploy it on a cloud server & it automatically hooks up to a centralized source of data feeds or the exchange? I.e. if you want to use fundamental data, where would you get it from?

I'd be interested to follow along at least, what is the slack channel called?

FYI, QuantRocket are looking at doing something similar - https://www.quantrocket.com/

Hi Peter,

I am Director of the Quantitative Finance Club at Texas Tech University. The Quantopian platform has been an invaluable learning tool. We have a few programmers who are interested in a functioning live server such as zipline-live. We are familiar with the open source zipline repository and host backtesting locally. My contact information is [email protected].

Hi All,

Someone has already mentioned QuantRocket and while they seem to be moving forward with development, their business model is to charge fees to use platform based on size of account traded. The larger the account the higher the fees. While I know some software companies use this software licensing model, I will never use a platform to with such a licensing model.

Another software that is also zipline based and is working specifically on Cryptos is Enigma's Catalyst. They are using another open source software ccxt which is used for all the various Crypto exchange integrations. They are providing as open-source and while I am looking at using this software they don't yet have pipeline functionality built and it does not connect to any equity brokers like IB.

I want a tool that I can build automated algos for all markets; Crypto, Equities, and Futures. I have looked at the zipline-live repository and it seemed that the momentum of dev has slowed. If we can all work towards an effective open source solution based on zipline, ccxt, and an effective IB connector where we control the intellectual property of our models and can run the workflow from a dedicated server .... This would benefit all!

I'm In! :)

thanks, Alan, our idea are alike! Are you member of the Slack channel? In would love to discuss the roadmap further with interested people.

@Peter

Kudos to you for this effort! I’m not sure I have the coding skills you’re looking for but would can offer assistance with testing and general feedback.

@Dan, Peter
I second Dan‘s offer

My general feeling about Zipline-live is that while it could be better, it does work and given the circumstances that is great.

Explaining the circumstances

  • Many of the people writing code for trading cannot share their code. There are a number of reasons why they cannot share it: company
    policy, an asshole boss, compliance issues, etc. This is understandable.
  • Many of the huge OSS projects were done by employees of a company. Python has been supported by Google for many years, IBM has supported many Apache projects, etc. While Zipline and Quantopian are humongous contributions to the open source community, they were done so with a financial incentive, and they were written by employees of a company benefiting from the use of this software. The takeaway here is: don't feel bad that the project with 4-5 part time contributors looks slow and shitty compared to one run by 40 full time employees.

Some ideas I had for getting more contributions to Zipline-live
0. We could put some effort into highlighting contributors, this will incentivize developers who want their work to be seen. (I wanted to translate my book from Python 2 to Python 3 and I was too busy to do it myself so I asked the community to do it with the promise of recognizing the translator and it was finished quickly.)
1. GSOC - applying for a grant from Google to support an intern for the summer to do something big.
2. Asking some of the more forward-thinking hedge hedge funds to contribute. Traditionally hedge funds have been a stiff-ass place to work where it is against the rules to wear headphones or jeans, etc. There are some large and small ones that behave more like a software company than a bank. We could approach these and ask that they contribute, while explaining benefits.

Remember fawce (when they killed brokerage integrations)

Moving forward, we want to help people find other ways to trade their own money algorithmically. One possible option is the zipline-live open source project that is derived from our open source backtester, zipline. This project has the potential to be an alternative for personal trading. We've talked to the project leaders, and we've agreed to support the future development of zipline-live. Interactive Brokers has agreed to provide some assistance as well. We support more than a dozen different open source projects already and zipline-live fits our mission. We are also soliciting additional corporate sponsors to help support the project. If you're interested in using this project for your own personal trading, or in helping to build this project please visit http://www.zipline-live.io/.

I haven't been extensively following the zipline-live project until recently. Did any of these three ever actually happen?

I'm not sure if there's any validity to it, but I've noticed a tendency for Quantopian to be less enthusiastic about sharing lately. To my knowledge, the code for the Optimization API, the risk model, and the new SimpleBeta has not been shared yet. Regarding the Quantopian support for the zipline-live project, I would have expected an announcement that they'd published the details of their integration with IB and Robinhood, but I haven't seen anything (maybe this is a red herring, if the details and code would be irrelevant). Regarding the IB support mentioned by Fawce, has Quantopian provided an IB point of contact? Perhaps he/she could be invited to comment here. Any candidate corporate sponsors could be invited, as well.

Another potential opportunity for Quantopian would be to provide a zipline-live update and facilitate discussion at the upcoming QuantCon (perhaps inviting IB and candidate corporate sponsors to meet face-to-face). This would be a substantive way of showing support.

On a separate note, I'm wondering if someone could take a crack at articulating a business case here, from the perspective of Quantopian, IB, Robinhood, corporate sponsors, etc.? It kinda smells like throwing good money after bad, given Quantopian's effort to make it fly and having it crash with limited engagement.

noticed a tendency for Quantopian to be less enthusiastic about sharing lately

Natural as humans given the many quite-less-than-civil words that are all-too-frequently dished out in their direction perhaps. Even with interesting contributions happening, it's a tough realm here.
Or could be from lessened optimism. One might think of that as a bad thing and yet on the other hand for example the 3% downturn could be a good thing if it brings about adjustments toward 30% profit per year instead of being stuck perhaps around +5%. Hoping some will see my point.
So much of the stock market is a battle against our own human foibles, that's why I love algorithms, even though we still have to struggle with ourselves in creating them. Meanwhile it is true in any endeavor to some degree also, the battle against our own human qualities. The accomplishments here are astounding but those who have made them possible really need to almost sort of hold their breath staving off pride for when the bank account numbers go up, then breathe a sigh of relief. Part of avoiding pride is to really consider that some of the considerable brain-power and insights out here in the little people can be practically manna from heaven. I have been sidelined on something I thought would be beneficial and that's no fun, I still wish it would be considered.
We can all assist toward maximizing the big picture returns by taming our words to some degree I think, perhaps. Remember the animated character in the movie Inside-Out representing anger, with flames coming out of the top of his head. When things reach that point it may be time for a personal email to feedback or direct only to an author/member involved. Personal attacks publicly harm this entire thing. But then there's the historic death-by-a-thousand-cuts phenomenon, persistent digs that can wear one down. I don't have the answers, it just feels to me that there could be some current trusted paths worth revisiting, double-questioning, and maybe some overlooked worth reviving. Personally I like some of GK's suggestions above and elsewhere, but would replace the word crash with suffer. Another contest with no restrictions might be a bit over the top but generally thinking way outside the box might be something to consider. I believe I might even contract many outsiders for mere days at a time each for their objective viewpoint and say let us have it, criticize everything you can in our methodology, don't hold back, just be cordial.

Sign me up

The case for zipline-live is that there are very, very few people who care about it, but for those that do it is very, very important.

Looking from Quantopian perspective I understand that they have almost no incentive to help the project - as it is taking any good algos away from their own fund .

But even with all features, zipline-live will still not be able to mimic historical, real-time and fundamental data available for free at Quantopian. So in order to replicate it, you really have to do it in a serious way - forming a startup probably. Which then hits the problem of a relatively small target audience. I estimate a few hundred people ready to pay ~$50/month for it which is just too small of a market.

With respect to data vendors, my impression is that Quantopian has been able to get favorable terms by showing that they could prevent wholesale data theft. In this fashion, they could offer free retail trading (with the idea that the retail traders would also write hedge-fund-viable algos). The shortest put would be for Quantopian to revisit their decision, to see of there would be any way to offer a development sandbox for zipline-live that would allow users to hook up to their basic data feeds. The problem comes in, however, that to convince the data vendors that they aren't getting ripped off, they'd need to host everything end-to-end, or have some sort of spin-off business entity that would manage things. Everything comes back to the potential size of the market. My sense is it probably needs to be something like 10,000 active users, but I recall that Quantopian only had a few hundred. Frankly, I think Quantopian made the right decision (although the execution was kinda poor, giving only a 1-month notice), given the information I've gleaned. Presumably, they still have an offer on the table of $250M from Point 72 and should do whatever they can to execute on the hedge fund business model.

Looking from Quantopian perspective I understand that they have almost no incentive to help the project - as it is taking any good algos away from their own fund .

Apparently the userbases do not overlap, that's why they killed brokerage integrations. So I'm not really sure how it would take away from Quantopian to support zipline-live's development, because the algos that Quantopian is using and that zipline-live users are using are probably fairly different.

I think this discussion quickly derailed. I'm well past caring about the end of the brokerage integrations or what made sense for who and why. Still not happy about it and never will be, but life goes on.

My question is that there was a stated commitment by Quantopian to support zipline-live in a serious fashion. Does anyone know if they actually followed through in any meaningful sense at all?

From what I saw some members from Q's engineering team listened in on the Zipline-live Slack channel and provided guidance if we got stuck. They were there to answer questions as well. The engineers at Q have always been quick to respond to me, and helpful.

The code for SimpleBeta has been published, go to the Zipline Github repo and search for SimpleBeta.
The code for the optimizer hasn't been published, from what I understand it is not that involved. I actually would like to take a crack at writing my own version, to understand it better.
Some of the other code like the broker integrations with IB may not make sense for them to share.

Regarding the optimizer, I’d think it would be a welcome addition to zipline. Using the existing Q code would make sense, modifying where necessary.

Regarding business model, the goal is individual profit by combining community quant and coding skills. In the context of often asserted low capacity winning algos of interest only to smaller accounts, I am willing to give my coding time in exchange for implementing an algo in my personal account that will compensate me for that time. I expect our community has members who would take the opposite side of this trade, sharing algo knowledge in exchange for an execution platform. Together, we build out the platform for public use and private benefit.

I am open to joining a club of quants and coders with such a mission statement.

Regarding data, it would be nice to be able to import zipline into the Q research platform and run backtests there. Is this supported?

I have read the recent posts. I am now thinking that the tool just might get mature more quickly if IB were convinced that this tool could generate significant commissions in the long run. In reality, they really need a tool that supports algo development and automated trading.

I would think a conversation with the right person at IB and they could be convinced to be a major competitor. After all, they are broker that most would want to trade through given their access to markets and fee structure.

Has their been discussions with IB?

I'm sorry , hoping for a white knight is not a strategy. Chasing a white knight is a fruitless exercise, because of the relatively tiny target market. Instead of trying to disprove these statements, I advocate joining in an effort that benefits each individual member financially for their efforts as outlined in my initial post to this thread.

Thanks for continuing to work on zipline-live. I wish I could be of some use, but when it comes to python and css, I know just enough to get by.

I think it's important to not get discouraged by the idea that only a couple hundred people were actually using Quantopian's live brokerage trading feature at the time it was discontinued. The number of people who were using the platform working towards live trading is easily multiples of that -- people who were still working on trying to develop a solid algorithm or still watching their paper trades, confirming out-of-sample performance before moving ahead with real money. That part of the process takes a lot of time, effort, and includes endless potential setbacks. Though those people weren't actively using the live trade feature, they were on the platform to use it, and shouldn't be discounted.

I routinely check up on my paper trades here on Quantopian to see how they're continuing to hold up out-of-sample, and after the effort that went into developing them it's such a shame they're only making theoretical money. If there were a drop-in replacement, with Robinhood support, that works, I'd happily pay to use it (so long as it doesn't cut too much into my PvR). Though my experience live-trading with Quantopian was that it did not behave like the backtester and thus required lots of debugging/modification of my algo, until it stopped being able to sync altogether, which was the end of that. So even with Quantopian's infrastructure it wasn't smooth sailing (more like drowning). Without Quantopian's centralized infrastructure and data streams, it all seems even more far fetched.

@Peter

I would be happy to help out with the project. Where are u in the timeline of this project? I'm strong in Python and CSS and have used Zipline in many of my algorithms while running them locally. I also have dabbled in Django and could be of some use there as well.

While I was travelling (snowboarding in Japan) I followed this thread and I'm actually quite heartened by the responses and I had quite a few in my inbox as well. Lots of ideas to chase still. The gist of my thinking is that starting a funded startup and put some good development power behind it would be the most fruitful avenue. Zipline and Zipline-live can become a very important part of the financial ecosystem but in my opinion, Quantopian has taken an US-centric institutional route, I would like to take a tiered approach by developing and contributing to zipline and zipline-live and make it such that it's easy to keep an algo's code for yourself but let other people trade with it and they will pay a contribution. The crux in my view is data, multi-asset capabilities (trade on multiple of exchanges (IB, Robinhood, cryptos, etc) at the same time: enable multiple currencies, enable European and Asian exchanges, etc, etc ). Then the hard part: a server farm that is stable where one can move an algo to without disclosing the code and being able to rent out your work. Take zipline, Quandl , C2, Quantconnect and a proper reputation and verified trade system, hustle and mix and try to do this in a proper way.

So lots of work that will need funding.

I have spoken to several people who would like to be involved and would like to provide funds and I'm exploring several avenues, from self-funding (by me and a founding team), VC funding, ICO offering that would power the ecosystem and some other ideas that got shot down (robbing banks and old ladies)

Let me take a week and nail some stuff down and I'll come back. Zipline is too beautiful for it just to be a hedgefund tool. Hedgefund power to the people is a more wholesome framework... and hedge and other funds can join the party as they want.

Thanks, Peter

Peter,

If you can answer the Heilmeier questions, you'll be off to a good start.

I'd be interested in helping, if we can get some clarity on the overall enterprise.

Hi Peter,

As background, I spent a lot of time trying to get crypto data working with zipline.
I eventually found the solution, forget about it and use backtrader!
I have also contributed code to ccxt.
So I'll definitely be watching this project with interest.

Hi Peter:

Some of this code may be of use to you as well:

https://github.com/ScottFreeLLC/AlphaPy

Regards,

MC

Thanks Conal, CCxt integration seems a good way, Enigma/Catalyst managed so zipline-live can, but I would love to discuss the gotcha's with you

Thanks Mark, I'll definitely play with your framework and see where and how we can use it