Back to Community
Research memory - deleting variables no effect

Hi,
what I don't understand about research memory is that when I delete variables I don't need anymore via the "del" function, the research memory doesn't fall, but stays the same. Even when I deleted all variables, I still have the same research memory, as if I didn't delete any variable.

Question 1: Shouldn't research memory update, depending on the data that is stored in the variables?
Question 2: How can I reallocate research memory without shutting down the whole notebook?

1 response

Good questions! Actually using del to manage notebook memory does work and can be useful.

Memory management is done 'under the covers' in a process called 'garbage collection'. Jupyter notebooks use Python's basic mechanism to allocate and release memory but then also adds a few 'extras' on top. Basically, deleting a variable (by using the del function) simply removes the binding of that name from the local or global namespace. It doesn't actually remove the object from memory. It just deletes this specific reference to it. In a second second step, python checks if an object has any references. If there's none, then it places that object into 'garbage'. When the amount of memory consumed by garbage exceeds a min amount python will physically re-allocate that memory. There can be cases where an object isn't referenced but the object remains in memory because the total size is under that minimum value. This isn't typically noticeable in notebooks because this min value is relatively low.

So, what could be going on? Without seeing the code, my hunch is there are multiple references to an object. Something like this:

df1 = pd.DataFrame(np.random.rand(10000, 10000))  
df2 = df1  
del(df1)  

Doing this, one won't notice any impact on memory. The del function doesn't actually delete the object but just the reference. Since the object has a second reference (ie df2) python won't delete the object at this point. However, when one deletes this second reference the object will be available to get physically deleted.

del(df2)

Doing this, one should see memory reduced.

While one can explicitly free up memory in this fashion, python does a pretty good job on its own. Normally, one doesn't need to do anything explicit. The automagic garbage collection will free memory of any objects not currently referenced. Look at the attached notebook. It demonstrates the above issue as well as highlights other best practices for managing memory such as:

  • don't create intermediate variables if not required
  • reuse variable names if it doesn't impact readability
  • enclose calculations in functions

One other comment. The memory usage indicated by 'Research Memory' is the total across all running notebooks. Notebooks continue to run even when a browser window is closed. To close a notebook it must be explicitly shutdown from the notebook list page. Another reason you may not notice any decrease in memory when variables are deleted are these other running notebooks.

Good question.

Loading notebook preview...
Notebook previews are currently unavailable.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.