sabnzbd performance on atom525 syno

[luttjebo]

Hoping someone can help me out.

I'm using sabnzbd with python26 on a Synology DX1511+. The cpuinfo tells me that I have 2 cores with hyperthreading. All running smooth including various other packages like Sickbeard, CouchPotato and Headphones.

I recently got my link upgraded to 100Mbit fiber and I am not able to saturate my link using the Synology. From OSX or Windows I get a full 11Mb using 40 SSL threads but the Synology only gives me about 6Mb whereas using the Download Redirector does give me the full 11Mb download speed with the same settings.

I did a bit of digging but the difference seems to be in CPU usage. If sab is only downloading (no par/rar) the CPU only shows about 20% to 25% load. If par/rar starts the performance remains 6Mb as the other tasks also appear to be single core max 25% adding up to a little over 50%.
Also. if you start a fresh download, download speed quickly starts at 7 to 8Mb for a few second untill sab need to start putting the articles together using yenc. In fact, without the yenc module performance drops to less than 4Mb as the sum of sab processes easily reaches 25% load.

So where does that leave me. It sure seems sab/python is not able to make use of the 2 cores with hyperthreading as all processes seem to max out @ 25% CPU load. I wrote a quick python script which I launched from 4 different shells and all 4 x 25% gave me fully loaded machine as it should :-)

The effect of this single core usage seems to be worse if you stop using SSL. Download performance increases but with the increased performance you also see huge fluctuations in download speeds rating from 4Mb to 8Mb as the yenc has the same parent and still shares the 25% load.

It's not just sab that is showing this. par2 and rar are both not capable of using the additional core and even worse, seem to have serious issues with the idea of hyperthreading. I wrote a small script of /proc/stat to give me the usage in percent of each CPU core as procinfo shows 4 CPU's and it seems processes are thrown from one core to another never making optimal use of the processing capacity that is available.

I was kinda hoping we have an expert here that might be able to explain this behaviour or better... tell me if it can be fixed :-)

[shypike]

Unrar is singe-core.
There are multi-core versions of par2 in the open source scene,
but compiling them for your platform will be a great challenge.

[luttjebo]

Thanks. Also read that the multi-core par2 version will introduce some other instabilities so I'll stick to current one :-)

However... sab is a whole different story I hope as Python for OSX and Windows runs multi-core right ? I managed to disable some cpu's on the synology and as I'm no specialist I just enabled and disabled all combinations to ensure I also got the hyperthreaded secondaries :-)
It seems that busybox is very optimistic in showing CPU consumption for non-aware software as the total CPU consumption simply doubles by disabling hyperthreading and the performance remains about the same.

[shypike]

OSX and Windows use multi-core par2 (with mixed resuts).
BTW: multi-core unrar would make no sense.
The majority of posts don't even use compression, they just
use RAR's convenient segmenting feature.

[luttjebo]

Right, so much for multi-core par2 and rar. But what about Python/Sab. Starting 4 processes seems to use all CPU's so is there a way to make sab spawn seperate processes for a 40 thread SSL download ? If not, can you think of any other way to have sab use multi-cores ?

[shypike]

Not without very substantial redesigns.

[TaZ1973]

Sorry for the late reply. I'm getting an average 8MB/sec on my DS1511+. After a lot of experimenting, I found that 6 connections, the cache limit set to 250M, and using stunnel for SSL (the built in SSL kills the download speed) works best for me.
I also did some experiments with NZBget and got and average of 9.5 MB/sec. So the trade of for me is 1.5MB/sec, but considering the ease of use and support from other software like couch potato and sick beard, I think I can live with it.

[shypike]

I'm surprised that SSL makes things so much slower,
since essenially all software uses the same OpenSSL libraries.

[TaZ1973]

If I remember correctly the build in SSL only utilize one process (100% on one core) while stunnel creates one for every connection.

[tg2002]

Sorry for the late reply. I'm getting an average 8MB/sec on my DS1511+. After a lot of experimenting, I found that 6 connections, the cache limit set to 250M, and using stunnel for SSL (the built in SSL kills the download speed) works best for me.
I also did some experiments with NZBget and got and average of 9.5 MB/sec. So the trade of for me is 1.5MB/sec, but considering the ease of use and support from other software like couch potato and sick beard, I think I can live with it.

Hello TaZ1973,
i have the same problem with slow DL-Speed when SSL is enabled on my Synology DS412+.
Can you please tell me, how you have implemented stunnel (installation, config, etc) and get sabnzbd+ working with it? I have ipkg running on my DS.

Thank you very much.

[bilbo_yes]

Just to confirm that it is indeed the source of the problem. I get 11.5MB/s on a desktop and couldn't go over 6-7MB/s on synology.

Started up stunnel on another computer and connected to that one from the sabnzbd/synology without SSL and voila, 11.5MB/s on the synology (DS412+).

Having a stunnel package might be a good idea (or sabnzbd could be improved to have separate ssl processing).