High Memory Usage / Potential Memory Leaks Investigation Thread

[lennardw]

My memory issues have been solved.
I downloaded winrar 3.91 (from the rarlabs site). I stopped SABnzbd, removed (a lot of) files in the cache directory and now memory usage doesn't come above 250M


Update. I was wrong. The memory usage is still real high....

[shypike]

Did the cache contain a lot of files?
That specific issue (cache orphans) will be solved in 0.6.0
Surprising that it has impact on SABnzbd's memory usage.
I doubt that unrar has any bearing on this, as the program doesn't run permanently.

[bjornhallberg]

Version: 0.5.4
OS: Windows 7 x64
Install-type: Windows Installer
Skin (if applicable): Smpl
Firewall Software: Comodo 5
Are you using IPV6? No
Is the issue reproducible? Yes, in a general sense

I'm having problems with high memory consumption as well, especially after a few downloads and S3 standby cycles. In addition, I have on rare occasions come back from S3 with a constant 25% CPU drain (no downloads active).

Memory consumption goes something like this. Boot: 15 MB; Starting download of 9GB rars 30MB; 50 % done 40 MB; 95% done 49 MB ... verifying / repairing / unpacking still 50 MB. Memory is not released after download is done. After a second download, levels can be around 70-80 MB. The highest I've seen was yesterday with 140+ MB.

[shypike]

Windows is not very accurate in reporting actual memory usage.
Python (our programming language) is slow in returning memory to the OS, for efficiency reasons.
As long as the memory usage isn't steadily rising, I don't consider it to be a memory leak.
BTW: Windows will swap out any memory that isn't being used actively.

[lennardw]

Did the cache contain a lot of files?
That specific issue (cache orphans) will be solved in 0.6.0
Surprising that it has impact on SABnzbd's memory usage.
I doubt that unrar has any bearing on this, as the program doesn't run permanently.

(( You were right that winrar did not have any influence. (was a coincidence) ))

Yes the cache directory did contain a lot of files. I don't recall what types of files. ( approx 1100 files over about 75-100MB tops )

[sander]

As long as the memory usage isn't steadily rising, I don't consider it to be a memory leak.

Exactly. I believe on Linux an app will get as much memory as possible ... as long other apps are not asking more memory / memory back.

[lennardw]

As long as the memory usage isn't steadily rising, I don't consider it to be a memory leak.

Exactly. I believe on Linux an app will get as much memory as possible ... as long other apps are not asking more memory / memory back.

And this is exactly where the 'leak' is. Memory isn't returned to the OS. I don't mind SAB getting 2gig of memory, but if it causes SAB not to be able to kick off unrar or par2repair, then I do mind. (not to mention all the other stuff that happens when the OS runs out of memory).


Background:
On Linux (just like Mac OS X, Windows, Solaris, AIX and almost 100% of all other OS's) all applications will recieve all the memory they request (except when it is limited by the administrator of the OS). There is a diffferences between active and inactive memory. You can see this in /proc/meminfo on a Linux box. There is Active memory, and Inactive memory. Active memory is allocated memory, that is currently in use Inactive memory is allocated memory, cached for a specific application, but is available for the OS to reclaim on the fly when needed.

The problem here is (in my view), that the memory claimed by python is not returned to the OS for reusage (it stayes in 'active memory'). I am not a python programmer, and know very little about python. I am however aware that there is a garbage collector, that should work properly/sufficiently out of the box. I also know that it there are certain commands to force the garbage collector to run. What I really don't know is what is concidered garbage by Python. (and if unused download cache is concidered as such, by the programmer and/or the system)

[shypike]

For some reason the garbage collector works different on Windows and Linux,
or maybe rather the algorithm regarding returning memory.
On Windows we do not see serious leakage behavior.
Manipulating Python's garbage collector is mostly an illusion: it doesn't really work.
There are a number of potential true memory leaks, when dealing with
mutually linking objects. That's not the issue with SABnzbd,
because the large memory bloat only occurs on Linux.
Forcing limited memory usage of SABnzbd works, without affecting the proper operation.
This already strongly suggests that SABnzbd itself doesn't cause the leak,
because apparently it doesn't need the memory.

I'm out of options for this.
I will look into the influence of large cache folders on memory usage.
Although that issue will go away with release 0.6.0

[lennardw]

I figured as much about garbage collection from Python...

Maybe just a small thing to maybe look into. Whenever a duplicate message id is there, or messages are missing.. Maybe that isn't handled properly? (just a shot in the dark of course )

[shypike]

Still not an explanation why the situation is only so bad on Linux.

[lennardw]

That is very true. But other nntp-clients like hellanzb (also Python) do not have the same issue. Hellanzb buffers the file while it is downloading too, and releases the memory directly after it's done. (I can see the memory usage grow and shrink), so it's not Python itself. It's probably the way SAB uses the memory, and that the Linux Python handles releasing differently than Windows. (as you stated here and there before).

That said, memory leak is the wrong phraise (as it's not constantly consuming more and more) for this issue.

The problem is likely the SAB code in combination with Linux. At any rate. It it is your oppinion that you can't fix this at this moment, then I suppose the workaround would suffice...

[shypike]

I simply don't know how to fix it, because I don't know what the cause is.
Memory analysis is a nightmare with Python (no mature tools for it).
It will require a lot of time (and concentration), which I currently do not have.
Nevertheless, it's something that must be addressed.

[loopdemack]

Could we try to ask for help from some python expert somewhere out of the community.

Maybe someone is willing to help us out in this issue.

[mindflayer]

I doubt this fully accounts for the memory leaks, but SABNzbd+ leaks a lot of file handles, at least on Windows.  If you open Task Manager's 'Process' tab and add the "Handles" column, SABnzbd.exe sometimes has 20K-50K open file handles, even after all downloads are finished.  Handles seem to get leaked even during unsuccessful connection attempts.

I'm using 0.5.5RC1 on Server 2008 R2 64-bit, but it's been happening for a long time.

[loopdemack]

I doubt this fully accounts for the memory leaks, but SABNzbd+ leaks a lot of file handles, at least on Windows.  If you open Task Manager's 'Process' tab and add the "Handles" column, SABnzbd.exe sometimes has 20K-50K open file handles, even after all downloads are finished.  Handles seem to get leaked even during unsuccessful connection attempts.

I'm using 0.5.5RC1 on Server 2008 R2 64-bit, but it's been happening for a long time.

Could you check also the "user objects" and "gdi objects" for certain on Windows if you see that gdi objects start raising, than you have memory leak, but 50000 handles is for certain a leak!