NZB retrieval performance get worse over time

[davidtaylornc]

I am having this odd problem that turned up in the last released version. I am downloading a lot of small NZBs. Most are under 5MB, but I process up to 30,000 per day. It's not unusual to have 5k nzbs in the queue at any time.

I was able to download about 750,000 nzbs on 0.4.x version without this issue. I clean ungraded to 0.7.20 after a hard drive failure nuked the 0.4 version a few weeks ago.

My environment is 64 bit Windows 7. I don't have any sanzbd plug ins. I do have a fairly lengthy file tossing post processing script.

When I first start sabnzbd, it runs fine and I get full download speed. As the number of processed nzbs in that session grows over 10,000, download speed start to drop. Eventually D/L speed hits 0 and nothing gets downloaded, even though I have 15 connections going.

When this bug happens, the plush web UI updates very slowly. For example, it can take 10 minutes to just shut down sanzbd. I don't have an I/O or CPU deadlock -- those run at less than 10% at all times.

My guess is that sanzbd is having trouble servicing the database when it gets over a certain size.

I've tried purging the logs from the plush, but that cannot even be accomplished when sanzbd gets into this state. I've tried deleting the log files, but it has no effect. I've re-installed several times and it hasn't solved the problem. Does anyone have any ideas?

[shypike]

Have you tried release 0.8.0Alkpha3 with the new Glitter skin?
Also, set the refresh rate of the queue much lower (default is 1 sec, which is too fast for large queues).

[davidtaylornc]

I have kept refresh rate set to to 1 minute. Otherwise the disk thrashes when adding nzbs in mass to the queue. Also I've tried increasing the memory limit and that does not affect this symptom.

I have the idea that I need to just run some database cleanup script every six hours or so to not bang some 0.7.20 limit.

[shypike]

Is there a large download queue or just a large history?

[davidtaylornc]

Since last week I've found that starting sabnzbd without the local WebUI can postpone the slowdown. The slowdown still occurs, but without the webui running it is taking 3-4 days to occur. When the slowdown hits under program only conditions, it hits like a truck into a wall instead of gradually.

To answer your questions:
In order to keep sabnzbd running, I only put 1000* nzbs in the queue directory at once. Since most of my nzbs are small, 1000 nzbs is a couple hours of processing. The log can grow to hundreds of pages in just a couple days. Since the log can't be purged from plush once sabnzbd goes into 0 byte download state, I have to stop sabnzbd, move all the backup nzbs, rename the log files, and restarts sabnzbd. Problem is: that doesn't always fix it. Sometimes I have to erase the whole database file, delete all the files in the cache directory, and requeue everything in the work directory.

--------
* if more than about 2200 folders get into the processing directory, cherrypy starts intermittently crashing with an out of memory error. There seems to be some load management built into sabnzbd to prevent that, but it doesn't handle certain corner cases. These crash events are pretty easily triggered when a big nzb gets mixed into all the tiny ones and work directories start accumulating. I use a restarter to monitor sabnzbd against those crashes.

[shypike]

The log can grow to hundreds of pages in just a couple days. Since the log can't be purged from plush once sabnzbd goes into 0 byte download state

Which log do you mean?
BTW: we haven't really tested SABnzbd with over 1000 NZB in the queue.
It's likely that SABnzbd tries to load to many jobs in memory in some circumstances.
I'll look into this, but it will take a while.

[davidtaylornc]

The log can grow to hundreds of pages in just a couple days. Since the log can't be purged from plush once sabnzbd goes into 0 byte download state

Which log do you mean?

Plush shows a list of processed nzbs with status, some number per page, below the incoming queue. So 'that log'. I guess it actually consists of database entries for processed nzb status and script file output. Anyway, that list can get to thousands of entries in just a couple days. Over some unpredictable size, sabnzbd won't clear it out with the plush buttons to do just that.

I understand if you can't look at it as a bug. Most people are just downloading a few TV shows or movies every week. I just thought that somebody might have seen the same thing.

[shypike]

The problem can only be in the download queue, not the history section.