Download speed dropping with large NZB's; "delaying" message in DEBUG mode

[roco]

Hi y'all,

I have this problem with my download speed dropping. In the connections view, I can see that at moments there are a lot less if not no connections that the maximum configged for the newsserver.
This is something I haven't stumbled upon before, when download speeds of 8-10MB/s on average was normal, where they are now at 3-4 MB/s.
I alread applied the speedmods, like assigning 500M of article cache, and disabled the "Only Get Articles for Top of Queue" optionI changed the filesystem of the cache folder to reiserfs instead of ext3. Even pausing downloads when postprocessing does not help significantly.
When putting logmode in DEBUG mode, this is what I get: (It seems that SABnzbd is delaying itself).

2011-03-01 15:44:08,731::DEBUG::[decoder:92] Decoding
2011-03-01 15:44:08,746::DEBUG::[decoder:92] Decoding
2011-03-01 15:44:08,760::DEBUG::[downloader:345] Delaying
2011-03-01 15:44:08,762::DEBUG::[downloader:345] Delaying
2011-03-01 15:44:08,763::DEBUG::[decoder:92] Decoding
2011-03-01 15:44:08,772::DEBUG::[downloader:345] Delaying
2011-03-01 15:44:08,773::DEBUG::[downloader:345] Delaying
2011-03-01 15:44:08,774::DEBUG::[downloader:345] Delaying
2011-03-01 15:44:08,776::DEBUG::[decoder:92] Decoding
2011-03-01 15:44:08,790::DEBUG::[decoder:92] Decoding
2011-03-01 15:44:08,793::DEBUG::[downloader:345] Delaying
2011-03-01 15:44:08,799::DEBUG::[decoder:92] Decoding
2011-03-01 15:44:08,804::DEBUG::[downloader:345] Delaying
2011-03-01 15:44:08,805::DEBUG::[downloader:345] Delaying
2011-03-01 15:44:08,829::DEBUG::[decoder:92] Decoding

Version: 5.6 Final
OS: Ubuntu 8.10
Install-type: python source
Skin: Smpl (Not really relevant I guess)
Firewall Software: None (Behind NAT router, no change here)
Are you using IPV6? no
Is the issue reproducible? yes
Hardware: HP nc6120 laptop, centrino w 2Gb internal memory.

Thx in advance

[shypike]

The decoding of articles cannot keep up with the actual downloading.
Do you also see 'undelaying" messages?

Are your NZB files bigger than before?
BTW: using debug logging slows down SABnzbd also (not dramatic, but still).

[roco]

Well, yes, the undelaying message also occurs. And yes, my nzb's have been a bit larger than before.
Is there a neat way to prevent my downloads to slow down when they are this big, or is the only way to split them and repair and unpack them manually?
I tried hacking the download.py, and removed the delay argument in line 345, but that had an unwanted effect (spontaneously disappearance of the download).

About the DEBUG: I know, I only had it in DEBUG mode for a few minutes.
And thx for the quick reply..

[shypike]

I'm afraid you have to live with this until after the final release
of 0.6.0, we'll be turning to this kind of performance issues.

BTW: removing delay/undelay will explode memory usage
and make the OS run out of memory after some time.

The basic problem is that your system can download articles
faster than it can process them.
This problem is made worse by the fact that large NZBs run slower.
It is the latter problem that we will be addressing later.

[roco]

Is it a yEnc thing? I mean, I have version 3.0, would it help to use 3.1? Or is it, no offense, entirely SABnzbd's code that causes this, due to the large nzb problem?
I download quite a lot, but when downloading having up to 100GB in the queue it never is a problem with smaller nzb's. It probably takes it too far to explain why large nzb's are a problem, and large queues are not...

[shypike]

yEnc 3.0 and 3.1 are virtually identical.
The large NZBs being slower is a SABnzbd problem.
Which will be tackled later.

[Lockzi]

Hello,


EDIT: just saw the "help us help you" thread.



Version: 0.6.10
OS: Windows XP Professional N Version 2002 SP 3
Install-type: Windows Installer
Skin (if applicable): Default (blue)
Firewall Software: Dedicated hardware firewall (pfSense)
Are you using IPV6? No
Is the issue reproducible? For me yes all the time, for others - unknown?



Running SABnzbd+ 0.6.10 I also get this problem.

I've moved which has given me an upgrade in bandwidth from 8 Mbps to 200 Mbps (loving it ). I've also done many of the speed tips as the OP posted (except filesystem changes). SABnzbd is running on Win XP machine which is a virtual machine (VMWare ESXi) running on xeon processor and with loads of memory (this machine only has 4Gb due to x86 limitations).

I am seeing download speeds of about 11 Mbps but as soon as the cache (tried various settings between -1 (unlimited) and 1024M) gets full, download speed completely halts.
My first thought was that since it is a virtual machine I download to a dedicated NAS (it only has handling the downloads as a task) and the cause was that as soon that the cache gets full it starts to write to the disk at full possible speed. To check out this theory I redid the RAID volume on the NAS from RAID 5 to RAID 0 for extra speed (it copies the final download to my SAN after download and post-process is complete).

I am seeing many

Code: Select all

::DEBUG::[downloader:214] Delaying

as well as

Code: Select all

::DEBUG::[downloader:219] Undelaying

.

The CPU is not spiking on the VM nor is the RAM full (except when running -1). The downloads can vary between 412 MB to 5GB or even surpass that.
What has been noticed is that if I put cache at 128M and am downloading 412 MB it slows down significantly when cache is full.

When copying from the NAS I get speeds at about 8-11 MB/s, which should be around what SAB should slow down to, but it gets way down below 1MB/s as soon as the cache fills up.

Writing this I am thinking I should try to setup SABNZBd temporary on my workstation to see if there are any changes. I used Panic's Unison with full speed downloads all the way through on my workstation as a test, did not try SAB though.

Please let me know if you did address the problem with the v6 + updates, as I am currently running the latest.

EDIT2: I could also let you know (could be completely unrelated) that SAB now crashes constantly (3-4 times a day) without anything in sabnzbd.log except decoding articles which the log is full of obviously. This started happening after updating to the latest version.

Regards,
Lockzi

[jcfp]

When copying from the NAS I get speeds at about 8-11 MB/s, which should be around what SAB should slow down to, but it gets way down below 1MB/s as soon as the cache fills up.

No surprise there. Once the cache fills up, all subsequent small articles are written to disk as if you had no memory cache at all. The cause is your nas that obviously cannot handle any serious throughput, given that you only get 8 MB/s when doing (sequential) reads. Remember that reading from disk is usually faster than writing, and larger chunks of data transfer much faster than lots of small stuff. The overflow from the cache results in numerous small writes, severely limiting your download speed. That's before factoring in the inefficient setup of downloading on one machine, having temp stuff on a slow nas, and storing the end result on a yet another networked machine. That could also create bottlenecks in the networking department, as data are sent back and forth a number of times.

Better cut out the nas, and only store the end results of downloaded jobs on the network, keeping all temporary files, cache, processing, etc. on the downloading machine. Also don't set unlimited cache, make it large if you want no more than that: unlimited is taken literally and can bring your machine to its knees.

[Lockzi]

jcfp,

thanks for taking the time to reply.

But is as slow as 100-700 Kbits valid for writing small files?

Just tried the speed of writing to the NAS - the speed varied between 10-13 MB/s (more on the upper scale). Allthough I do acknowledge the difference between random and sequential writes.

What's interesting as well is that I could download at a constant 700 KB/s on my 8 Mbit connection, where as it now slows down way below that and even brings to a halt on occasions. The average download speed is way quicker though.

Please bare in mind that the NAS has moved from RAID 5 to 0 - so whatever problems it is having it should be helped.

Regards,
Lockzi