Memory leakage while plotting in a loop
Contents
Issue
Memory leakage while generating python matplotlib plots in a loop on MacOS system.
I was using python 3.9 and MacOS Catalina.
I was trying to generate lots of plots for my analysis. Idea was to create them in a loop:
- render plot
- save the output
- iterate further
Simple example of the case:
|
1 2 3 4 5 6 7 |
import matplotlib.pyplot as plt for i in range(10000): fig = plt.figure(figsize=(25, 25)) plt.plot([1,2,3]) plt.savefig(f'temp.png') plt.close(fig) |
However, it turned out that this simple code quickly led to memory overflow, despite closing the figure object on each iteration.
Attempts
On the first attempt I tried cleaning up the figure object and making sure that all of the figures are indeed closed.
|
1 2 3 4 5 6 7 8 |
import matplotlib.pyplot as plt for i in range(10000): fig = plt.figure(figsize=(25, 25)) plt.plot([1,2,3]) plt.savefig(f'temp.png') plt.clf() plt.close('all') |
That did not help, so I tried to enforce garbage collection to explicitly free up the memory.
|
1 2 3 4 5 6 7 8 9 10 |
import matplotlib.pyplot as plt import gc for i in range(10000): fig = plt.figure(figsize=(25, 25)) plt.plot([1,2,3]) plt.savefig(f'temp.png') plt.clf() plt.close('all') gc.collect() |
No success either.
Solution
This time I encountered an option to change the matplotlib backend – matplotlib.use(“Agg”).
|
1 2 3 4 5 6 7 8 9 10 |
import matplotlib import matplotlib.pyplot as plt matplotlib.use('Agg') for i in range(10000): fig = plt.figure(figsize=(25, 25)) plt.plot([1,2,3]) plt.savefig(f'temp.png') plt.close(fig) |
And that helped!
What does that mean to set the plotting backend?
As quoting the documentation:There are two types of backends for matplotlib: user interface backends, also referred to as “interactive backends” and hardcopy backends to make image files, referred to as “non-interactive backends”.
That basically means that in order to save files, as in my case, it was better to use a dedicated matplotlib backend instead of the default one – which is interactive. Matplotlib should in principle automatically select proper backend based on the given use case, however as documentation states, it is not always the case.
Selected “Agg” backend refers to png file type.
Your solution did work for me. Thank you very much.
This absolutely saved my butt, thank you! had to make over 2000 plots and it would always fail by 300.
This worked for me too. My code sped up almost 100x!
This worked for me too (Python 3.9.12, macOS Ventura 13.1, M1). Thanks for posting.
I have the same problem,(Python 3.9.6, macOS Ventura 13.1, M1).
after putting in
matplotlib.use(‘Agg’)
I get:
UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
plt.show()
Works for me too (Python 3.9, macOS ventura 13.1, M1 Max). Thanks for sharing