SABnzbd Forums

Hi everyone

First of all: I have to say sabnzbd is an absolutely awesome piece of software. I can't really think of much to improve !

Unfortunately I have an issue with my dl speeds. I have a 100MBit/s line. When I have a "fresh" install of sabnzbd (reads: rm -rf .sabnzbd/admin ) I get the expected dlspeed (between 11 and 13MB/s). However, the speed gets slower and slower over time. It drops steadily until it's around 1MB/s (after roughly one hour). Restarting sabnzbd doesn't help, nor does pausing the queue.

The box is an Atom 1.67 dualcore with 2GB of ram. SABnzbd.py uses between 90 and 120% of CPU according to top. The box isn't running anything else really (except the md raid to store the data). I'm aware that the hardware is really sweating at full download speed, but I can't explain why the speed should steadily decrease.

Anyway, the only method to get the speed back up (temporary) is to flush out all the sqlite dbs in the admin dir and restart sabnzbd. I will try to only delete single sqlite files and see which one exactly is causing the slow speeds.

Any ideas in the meantime?

I don't see this behaviour on any of my test systems,
Including my workhorse downloader box which has similar hw specs.
You write about multiple sqlite files. SABnzbd creates and uses only one.
This could be a case of odd access rights.

root@nas2:/mnt/md0/SABnzbd/.sabnzbd/admin# ls -l
total 24
-rw------- 1 sabnzbd sabnzbd 18 Oct 30 16:02 bookmarks.sab
drwx------ 2 sabnzbd sabnzbd 4096 Oct 30 15:27 future
-rw-r--r-- 1 sabnzbd sabnzbd 2048 Oct 30 16:00 history1.db
-rw------- 1 sabnzbd sabnzbd 11 Oct 30 16:00 postproc1.sab
-rw------- 1 sabnzbd sabnzbd 101 Oct 30 15:57 queue9.sab
-rw------- 1 sabnzbd sabnzbd 180 Oct 30 16:10 totals9.sab

you're right, i guess history1.db is the only sqlite file? i'll start deleting one file after another now to see which "reset" causes the speeds to go back up.

ok, it seems that i was just stupid.

i've checked the debug log and apparently i've given the daemon "too much" ram. basically it fetches itsself a huge backlog for de-encoding yenc file. the bigger the backlog gets, the slower the dl speed becomes. giving it a 5 minute break and it's back to normal. guess i have to buy faster hardware or live with it

Only the .db file is an sqlite file, the others are just simple admin saves.
What you describe is still weird.
With 2G of RAM, you should be able to set the cache value to -1.
SABnzbd will then be able to cache all articles for one file.
The cache size should not go beyond about 150% of the largest file.
In most downloads files stay below 100M.
It could be that writing to disk is so slow that the cache fills up.
In that case you should have a cap on the cache.

finalius wrote:i've checked the debug log and apparently i've given the daemon "too much" ram. basically it fetches itsself a huge backlog for de-encoding yenc file. the bigger the backlog gets, the slower the dl speed becomes. giving it a 5 minute break and it's back to normal. guess i have to buy faster hardware or live with it

Wondering how you got to this conclusion. Have you set a memory cache, if so exactly what value is it set to? Are you downloading to a network disk or raid array? The only slowdown mechanism in sab that I know of, is triggered by yenc decoding or disk access being too slow to cope with the network speed (only visible in debug logging). Still, it shouldn't get this bad when properly configured for high speed downloading.

Well I've done some tweeking and it has become better, but still not perfect.

I've set the Memory Limit to -1 as suggested. I don't think that the disks are the problem. I have a md raid 5 with 6 disks (7200rpm), 1.5TB each. But of course we're talking about a software raid which is handled by the same CPU that handles sab - so if sab uses up all the juice, the raid will get slow eventually. Given full CPU power, the raid does something around 130mb/s sequential writing and >300mb/s sequential reading.

The biggest improvement I noticed however is by reducing the connections to the newsnet servers. I had two servers and had 10 connections to each. Reducing this to 6 each server hardly decresed peak DL speed, but the speed keeps steady. I don't know how much overhead an additional connection generates. Also, I've set enabled "Pause Downloads during Post-Processing" to give the daemon some time to catch it's breath.

All this has made the issue somewhat manageable, and DL speeds stay above 10mb/s. This is my latest test from this morning:

Completed 2011-11-01 08:01:31
Name some random movie
Size 9.0 GB
Status Completed
Category *
Path /mnt/md0/Videos/Movies/....
Download Downloaded in 12 minutes 40 seconds at an average of 12.2 MB/s
Repair [randommovie.BluRay.1080p.x264.DTS-HDChina] Quick Check OK
Unpack [randommovie.BluRay.1080p.x264.DTS-HDChina] Unpacked 1 files/folders in 2 minutes 36 seconds

Of course the queue was paused while extracting - but oh well, it doesn't really matter if it takes 25 or 30 minutes to download ~20GB, does it?

finalius wrote:I've set the Memory Limit to -1 as suggested. I don't think that the disks are the problem. I have a md raid 5 with 6 disks (7200rpm), 1.5TB each. But of course we're talking about a software raid which is handled by the same CPU that handles sab - so if sab uses up all the juice, the raid will get slow eventually. Given full CPU power, the raid does something around 130mb/s sequential writing and >300mb/s sequential reading.

These speeds don't tell us much. If you didn't have a memory cache set, every small article would have been saved to disk, quite different from sequential writes and reads. Since it's software raid 5, parity calculation would have to be done for no good reason for all those small temporary files, then everything be read back for yenc decoding, then written again when writing the segments/rars files (another parity calculation), and read and written again when extracting (the final and only useful parity calculation). Especially the small file reads and writes would likely trash the disk, as well as put extra stress on the cpu.

finalius wrote:Download Downloaded in 12 minutes 40 seconds at an average of 12.2 MB/s

If all you have is 100 mbit, theoretical max is 12.5 MB/s. That's before counting protocol overhead, other network activity, etc. 12.2 seems pretty good to me.

update: don't set memory limit to -1. The cache grew to 1.6GB, and eventually SAB couldn't unrar anymore: Error "[Errno 12] Cannot allocate memory" while unpacking RAR files I've set it to 512M now, let's see how this behaves.

I think those issues all occur because the connection is faster than what the box can handle properly. Limiting memory and pausing dls during postprocessing makes sure that the download speed is always maxed out because then SAB really only has to download (and de-encode yenc) while downloading, plus of course the md raid. If you add par2 and unrar activities into the mix it's just more than what the atom 1.67 can handle.

Even so, I don't see this behaviour on my very modest Atom-based netbook running Ubuntu server.

So far the only scenario which I have seen to cause this behaviour is this:
One article of each (or many) files is missing AND a server times out all of the time instead of telling that the article is actually missing.
This doesn't happen with the average commercial server.
It can happen if you have a dodgy server active in your setup along with reliable servers.
If the other servers don't have the article, SABnzbd will keep retrying the one that doesn't answer.
Therefore it's essential that you label such servers as "optional".
This is an old problem and part of the way SABnzbd has been designed.
There will be a solution for this in the next major release (0.7.0).

This doesn't mean this is the true cause of your problem, but only a potential scenario that could cause them.

I'm having this issue on my install. Anyone else seen this issue? I think my issues is the box is running out of ram. If i reboot the box it speeds back up for a while.

Memory leak with sab? I'm running version 0.6.14 on Ubuntu 11.04