Back to Community
CVXOPT module

Hi Quantopian team,

Great product with easy to read api docs. Had my algorithm running in no time.

Any plans on including the CVXOPT module in the near term? Currently using the scipy.optimize package but would ideally like to use sparse matrices (I'm solving a sparse QP problem at each step).

Thanks!

26 responses

I too would like to have this added to the supported modules, but the Quantopian team has delayed in adding it for a while now (~2 years). I inquired about this about a month ago and they informed me that it currently would only be possible to use it in their Research Environment. However, this obviously doesn't help much since using the module within a live trading strategy is the main purpose of needing it.

I have resorted to using scipy.optimize as well and I also need CVXOPT for efficient sparse matrix solvers. The former is noticeably slower in comparison and much nastier in regards to quadratic programming implementations.

Is it a matter of slow backtests, or are you exceeding the ~50 second time-out of handle_data under live trading? --Grant

It is a matter of exceeding the timeout. For example, I want to optimize some function by running multiple scenarios where the number of scenarios should be large, but I am restricted to a certain scenario count due to the restricted time. This is a problem because the less scenarios I run, the less accurate the solution in regards to the target function.

Ah...so can you carry over intermediate results, so that the computation is done over multiple calls to handle_data?

I haven't tried that as of now, but that should work since the scenarios are independent of one another. I guess I would have to save the intermediate results within some structure stored in the context, which is a hack that could be easily avoided with the inclusion of the CVXOPT module. Another good reason to use CVXOPT is because it has a much easier API interface, especially when considering equality/inequality constraints, boundary values, and data structures.

Yeah, I worked my way through an example (https://www.quantopian.com/posts/long-only-minimum-variance-portfolio-using-scipy-dot-optimize-dot-minimize) and I'd agree that scipy.optimize has a learning curve. Aside from a more efficient optimization module, there are a couple other issues:

  1. The 50-second time-out seems overly restrictive. It should be possible to have background processes running over multiple calls to handle_data (and even overnight).
  2. Your problem sounds well-suited to some form of parallel processing. One should be able to launch all trials at once, and then collect the results for global optimization. I wonder how many processor cores and how much memory is available to each algo? Perhaps they could be allocated (up to some limit) when the algo is initialized?

It would be interesting to hear from Quantopian regarding plans for 1 & 2 above.

Grant

Hi Desmond, Grant, and Grant,

Great news, the CVXOPT module was added to the whitelist last night.

We'll have an example algo soon, but wanted to let you know it's there now.

Have fun!

KR

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

That's great to here Karen! Thanks! It would also be nice if the Quantopian team could tackle points 1 and 2 from (other) Grant's most recent post :)

I think I can speak on behalf of all the other Grants on Quantopian when I say that this is great news.

Great stuff. Thanks Karen !

Thought I'd try the example code give on http://cvxopt.org/examples/tutorial/qp.html. But, when I try to run the code below, I get:

14 Error Runtime exception: ImportError: cannot import name blas

Any ideas how to get it to work?

from cvxopt import matrix, solvers

def initialize(context):  
    context.spy = sid(8554)  
def handle_data(context, data):  
    Q = 2*matrix([ [2, .5], [.5, 1] ])  
    p = matrix([1.0, 1.0])  
    G = matrix([[-1.0,0.0],[0.0,-1.0]])  
    h = matrix([0.0,0.0])  
    A = matrix([1.0, 1.0], (1,2))  
    b = matrix(1.0)  
    sol=solvers.qp(Q, p, G, h, A, b)  
    print sol['x']  

I have also attempted to implement something similar and I receive the following error:

ImportError: cannot import name misc

I was informed last week that it would be looked into, so hopefully the issue will be addressed soon.

We'll take a look to see what's going on!

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

did somebody get an algo to work with the CVXOPT module

Nope.

Nope, same problem as the Grants.

Did anyone ever get this to work?

Hi Grant K, it works in the research environment but not in live trading yet.

Any update on this? Why is it not working in live trading yet? --Grant

The CVXOPT module isn't currently available, we need to fix some things in the backend. When it's ready, we'll post an update in this thread.

I'm not sure why but it seems like cvxopt is now working on Quantopian (at least for me it did) -- yay! But do let us know if you encounter any problems.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

I still get an error for the code below:

14 Error Runtime exception: ImportError: cannot import name misc

from cvxopt import matrix, solvers

def initialize(context):  
    context.spy = sid(8554)  
def handle_data(context, data):  
    Q = 2*matrix([ [2, .5], [.5, 1] ])  
    p = matrix([1.0, 1.0])  
    G = matrix([[-1.0,0.0],[0.0,-1.0]])  
    h = matrix([0.0,0.0])  
    A = matrix([1.0, 1.0], (1,2))  
    b = matrix(1.0)  
    sol=solvers.qp(Q, p, G, h, A, b)  
    print sol['x']  

Pretty odd, does the cloned algorithm from https://www.quantopian.com/posts/the-efficient-frontier-markowitz-portfolio-optimization-in-python-using-cvxopt work for you? It might somehow not call the optimization I suppose.

It runs (see attached). --Grant

Clone Algorithm
22
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 55279bca9b08930d5167a690
There was a runtime error.

I need the cvxopt module. It is officially available now in the backtester?

Attached backtest may demonstrate a workaround for the import error.

At least, Grant's test runs with it and another cvxopt algo I have works in the backtester.

I added a run once latch like is done in zipline to warm up history before calling cvxopt.

How I found this. I noticed I had left in history warm up code that is only needed in zipline in a backtest I posted to this thread:
https://www.quantopian.com/posts/the-efficient-frontier-markowitz-portfolio-optimization-in-python-using-cvxopt When I removed it, I got the import error.

Richard

http://quant-coder.prokopyshen.com/ContactMe

Providing custom quant programming, tutoring, linux cloud support and troubleshooting services.

Clone Algorithm
2
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 5533d4a331ee610d47846aef
There was a runtime error.