Back to Community
Introducing: Memory Usage Meter in Research

If you're a heavy user of the research environment, you've probably seen it "hang" or "crash" at some point. You might even have done something as simple as open all of the lecture notebooks at once! When this "hang" happens it can be incredibly frustrating. We're giving you a new tool to help you understand and prevent most research "hang" problems.

This week we launched a new memory usage meter in the research environment. You now see this meter in the navigation of individual notebooks, and on your notebook list screen. It reflects your research memory usage across all notebooks. This will help you better understand when memory is your limiting factor - and when your research notebook is getting close to "hanging."

If you are running into memory limitations, the first thing to do is shut down any notebooks you aren't using. You can also restart a notebook to clear out the memory of that notebook and start over (run > restart in the notebook navigation).

Memory is a complicated thing to manage in the best of scenarios. Working within Python adds layers of abstraction which can make it hard to manage your memory as effectively as one would like. The goal of this meter is better help you understand when memory is an issue.

The attached notebook is a quick example to help show how the memory meter works. Clone it and execute the cells one at a time.

Loading notebook preview...
Notebook previews are currently unavailable.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

10 responses

Also is it possible to allocate additional temporary memory to users on demand when running a memory consuming backtest or notebook and perhaps paged memory.

Generally set the basic memory usage you could typically do the following:
- full price history
- 15+ pipeline factors or data (generally you might use a lesser number but this helps establish more error free threshold)
- 5 transformation on price or factors
- 5 covariance / correlation matrices
- Run of Aphan / Pyfolio afterwards

Really helpful new feature! Thanks all!

Very happy to see this got shipped!

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

I noticed my memory usage meter says 45% used but I don't have any NB running, I shut all of them down. There must be some memory we cannot directly clear in Research. Maybe some functions (pipeline?) do internal caching and we cannot clear that cache?

Also, is Research and Algorithm/Backtesting memory shared?

Anyway, this memory usage meter is very useful, thanks!

Are you sure you have shut the note book down . go to the list showing your notebooks. look for any name that are a different color then the rest. If you find any, they are running in the background. When you place your mouse over it a menu will appear, one choice will be [ A square I believe] apparent.
With %45 usage chances are you will have several running in the back ground

@alfred thanks for your help. Unfortunately I can confirm there are no NBs running. I also tried the special checkbox on the top left corner that allows me to select all running NBs, "Notebooks (On)", it didn't select anything.

Ok I suspect something is going on with your browser. Close down the port and try another browser . If that doesn't work, its probably on th server side pertaining to you. Also can you increase it ( RUN several notebooks at the same time)

The memory usage meter is so helpful, It helped me solving many problems. Are there any plans to add it to the backtester too?

Thank you very much for this!! Can't wait to learn in Quantcon.

Also profiling of where the memory and CPU is used also would be more helpful.