Back to Community
Whatever changed with memory reclamation...

is broken with at least with sklearn.cluster and scipy.optimize. I can't even pull off a simple mean-variance optimization without the kernel reloading now.

5 responses

+1. And I’m not even using any ML stuff (yet). Running a single NB from a fresh start (around 5% memory utilisation - I can never get any lower than that) results in maxing out memory and a kernel restart all the time for me.

To be fair, memory reclamation does appear to happen (for me anyway), but not ALL memory gets reclaimed and the accumulated memory that doesn’t get reclaimed in the end causes the kernel to restart...

Wasting a lot of time in Research this way is very frustrating!! Maybe I just have unreasonable expectations? Unfortunately I’m oftentimes ‘forced’ to go back ‘testing’ factors in the IDE due to the memory limitation in Research, which is far from ideal.

Hi Rob, Joakim,

Thanks for the report of problems. We're looking into the issue.

Thanks
Josh

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

+1

Thanks Josh!

I determined in Rob's previous post that his notebook wasn't being idle-killed, there was something going on in the notebook execution itself. Rob, did you send in a support request as I mentioned to you in email? Without permission to access the notebook for debugging, it's not obvious what could be causing this problem.

I'll describe below how the behavior changed. If it sounds like the issue any of you are having, please get in touch (Help > Contact Support) and mention this thread.

Prior to Nov 21st, our Research Notebook service would never clean up notebook servers that were idle. The result was that our servers would (with increasing regularity) get overwhelmed with memory requirements of inactive Notebooks. We would have to force-kill ALL notebook servers every several days just to make room for new notebooks to run. After the 21st, we modified the Notebook service to cull idle notebook servers. Notebooks are considered idle if they aren't performing any active cell executions, aren't transferring data and if there is not a web browser connected to the notebook. If all three conditions are true for at least an hour, Notebook kernels are killed and memory is reclaimed for another user. You can tell if your Research Notebook kernel was killed by reviewing the list of your notebooks - running notebooks kernels are colored green. In the case that it's black (not running), you'd need to restart execution from the first cell.

It's possible that you may be negatively affected if you have very long running cells running on an unattached Notebook server. Please let us know if this is affecting you, we can certainly review the timeout we've implemented if it's causing problems.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.