Hello
I'm using sab on a 10gbit link and trying to max out the speeds.
However, the cache builds up very quickly, and causes both the server and sab to halt when it hits 5gb (or whatever limit I put in).
Problem seems to be that the par2 files are building up without getting downloaded/processed before a new download starts. This is not very strange since I have "Only Get Articles for Top of Queue" disabled. If I enable it the speeds will be very inconsistent as it'll only process one job at the time.
So I wonder if it's possible to make sab download the par2 files, freeing up the cache on the go, instead of getting them all once all the jobs are done? As of now, it looks like sab is just waiting for everything to finish, THEN starts downloading and processing the downloads, INSTEAD of doing it on the go. Ie; If I have 10 jobs, with 50 files in each, it'll download the 50x10 files, THEN start working on the par2.
Cache overflow choking up sab / prioritizing par2 files
Forum rules
Help us help you:
Help us help you:
- Are you using the latest stable version of SABnzbd? Downloads page.
- Tell us what system you run SABnzbd on.
- Adhere to the forum rules.
- Do you experience problems during downloading?
Check your connection in Status and Interface settings window.
Use Test Server in Config > Servers.
We will probably ask you to do a test using only basic settings. - Do you experience problems during repair or unpacking?
Enable +Debug logging in the Status and Interface settings window and share the relevant parts of the log here using [ code ] sections.
Re: Cache overflow choking up sab / prioritizing par2 files
First of all, the "top of queue" option is only for memory squeezed systems.
It should be off in all other cases.
Due to the parallel downloading parts of the secondary par2 files can be downloaded,
even if not needed.
This is because you only know the file name for sure when the first article is in.
(It's unbelievable how posters can mess up subject lines.)
Don't make your cache too large, set a limit.
When the limit is reached, SABnzbd will start dumping articles to disk.
BTW: The behaviour you see is more typical for posts that have 1 or more articles missing.
Such as Giganews posts that have been hit by a DMCA notice.
It should be off in all other cases.
Due to the parallel downloading parts of the secondary par2 files can be downloaded,
even if not needed.
This is because you only know the file name for sure when the first article is in.
(It's unbelievable how posters can mess up subject lines.)
Don't make your cache too large, set a limit.
When the limit is reached, SABnzbd will start dumping articles to disk.
BTW: The behaviour you see is more typical for posts that have 1 or more articles missing.
Such as Giganews posts that have been hit by a DMCA notice.
Re: Cache overflow choking up sab / prioritizing par2 files
Yes, I understand, but my point is that the download drops from 80MB/s to 30MB/s when the cache get's full. The server is capable of writing 1GB/s speeds to the drives, so I don't see that being the bottleneck.
Only way I found around this was to make the cache really big, so it wont fill up until all the jobs are done.
I wonder if there is a way around this.
You are also correct about that I use GN, as it's the only provider giving me speeds above 15MB/s...
Only way I found around this was to make the cache really big, so it wont fill up until all the jobs are done.
I wonder if there is a way around this.
You are also correct about that I use GN, as it's the only provider giving me speeds above 15MB/s...
Re: Cache overflow choking up sab / prioritizing par2 files
Articles aren't very big, so the disk channel will not be optimally used.
We've been looking at writing directly to the files, however this is hindered
by two issues: articles arriving out-of-order and the unpredictability of segment size.
The NZB says what the size of an article is, but not what the size of the nett data is.
So we cannot assemble a file in a random order.
BTW: missing articles are not a problem as long as all available servers say that
an article is missing. The biggest issue we have is that sometimes servers don't
say the article is missing, but just time-out, leading to endless repeats
and the delay of file assembly.
Having said that, I still think that the behaviour that you describe isn't what it should be.
But I have never seen the queue behave in the way you describe: doing all downloads before post-processing.
Unless of course each job hangs on lack of response of one server (unreliable servers should be set "optional").
The only other reason is the scheduler event "pause_post".
We've been looking at writing directly to the files, however this is hindered
by two issues: articles arriving out-of-order and the unpredictability of segment size.
The NZB says what the size of an article is, but not what the size of the nett data is.
So we cannot assemble a file in a random order.
BTW: missing articles are not a problem as long as all available servers say that
an article is missing. The biggest issue we have is that sometimes servers don't
say the article is missing, but just time-out, leading to endless repeats
and the delay of file assembly.
Having said that, I still think that the behaviour that you describe isn't what it should be.
But I have never seen the queue behave in the way you describe: doing all downloads before post-processing.
Unless of course each job hangs on lack of response of one server (unreliable servers should be set "optional").
The only other reason is the scheduler event "pause_post".