0.7.x more missing data blocks than nzbget

thepainless · Post by **thepainless** » January 3rd, 2015, 9:52 am

Version: 	0.7.x
Uptime: 	17h
Config File: 	/home/xxx/.sabnzbd/sabnzbd.ini
Used cache: 	Cached 0 articles (0 B)
Parameters: 	SABnzbd.py --config-file /home/xxx/.sabnzbd/sabnzbd.ini --logging 1 --daemon --pid /var/run/sabnzbd
Python Version: 	2.7.9 (default, Dec 25 2014, 21:45:52) [GCC 4.9.2]

i'm trying to download this nzb

the download fails with sabnzbd:

Code: Select all

Repair is required.
8 file(s) exist but are damaged.
24 file(s) are ok.
You have 1829 out of 2175 data blocks available.
You have 217 recovery blocks available.
Repair is not possible.
You need 129 more recovery blocks to be able to repair.

When I run sabnbd from terminal these errors appear:

Code: Select all

...
2015-01-03 15:08:35,082::INFO::[decoder:121] CRC Error in 54a5fecb$2$34050$b1db1813$443c6447@news.astraweb.com (6BEFF437 -> 616D3873)
2015-01-03 15:08:35,085::INFO::[decoder:230] <Article: article=54a5fecb$2$34050$b1db1813$443c6447@news.astraweb.com, bytes=793743, partnum=48, art_id=None> => missing from all servers, discarding
...

when I use the same nzb in nzbget or grabit (with wine) repair is possible:

Code: Select all

Repair is required.
8 file(s) exist but are damaged.
24 file(s) are ok.
You have 2165 out of 2175 data blocks available.
You have 216 recovery blocks available.
Repair is possible.
You have an excess of 206 recovery blocks.
10 recovery blocks will be used to repair.

missing articles cause more harm in sabnzbd than in nzbget or grabit?

Post by **sander** » January 4th, 2015, 3:08 am

Could it be time-based? Easy to check: run both downloads again.

thepainless · Post by **thepainless** » January 4th, 2015, 1:40 pm

It's not time based. I tried several times today and always the same result.
I heard some other people with sabnzbd who can succesfully download with this nzb, so it's something with my install/version.
There are 8 damaged and 24 ok files on the server. When I download with sabnzbd I only get 1829 out of 2175 data blocks available, with nzbget 2165 out of 2175 data blocks available.

sabnzbd output

nzbget log:

Code: Select all

warning	Sun Jan 04 2015 15:04:45	File 5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part25.rar has 1 bad block(s) of total 69 block(s)
warning	Sun Jan 04 2015 15:04:45	File 5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part23.rar has 1 bad block(s) of total 69 block(s)
warning	Sun Jan 04 2015 15:04:45	File 5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part19.rar has 1 bad block(s) of total 69 block(s)
warning	Sun Jan 04 2015 15:04:45	File 5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part14.rar has 2 bad block(s) of total 69 block(s)
warning	Sun Jan 04 2015 15:04:45	File 5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part12.rar has 1 bad block(s) of total 69 block(s)
warning	Sun Jan 04 2015 15:04:45	File 5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part09.rar has 1 bad block(s) of total 69 block(s)
warning	Sun Jan 04 2015 15:04:45	File 5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part07.rar has 2 bad block(s) of total 69 block(s)
warning	Sun Jan 04 2015 15:04:44	File 5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part01.rar has 1 bad block(s) of total 69 block(s)
warning	Sun Jan 04 2015 15:01:09	1 of 69 article downloads failed for "5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix/5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part25.rar"
warning	Sun Jan 04 2015 14:59:59	1 of 69 article downloads failed for "5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix/5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part23.rar"
warning	Sun Jan 04 2015 14:57:39	1 of 69 article downloads failed for "5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix/5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part19.rar"
warning	Sun Jan 04 2015 14:54:53	2 of 69 article downloads failed for "5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix/5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part14.rar"
warning	Sun Jan 04 2015 14:53:45	1 of 69 article downloads failed for "5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix/5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part12.rar"
warning	Sun Jan 04 2015 14:52:05	1 of 69 article downloads failed for "5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix/5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part09.rar"
warning	Sun Jan 04 2015 14:50:58	2 of 69 article downloads failed for "5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix/5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part07.rar"
warning	Sun Jan 04 2015 14:47:37	1 of 69 article downloads failed for "5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix/5QIhnjfY4pPpLQTPWpo3MChnvJqxC7fpYp7H08qlynXkWHTL7clIohWgJSoSBxQaix.part01.rar"

For now I made a workaround for when this happens. This post-processing script will send the nzb to nzbget when verification and unpacking fails in sabnzbd. Yes it will download everything again, luckily I have an unlimited data plan

Code: Select all

#!/bin/sh
# $7: Status of post processing. 0 = OK, 1=failed verification, 2=failed unpack, 3=1+2, -1=failed download
# $2: The original name of the NZB file

sabnzbdnzbs="/media/Seagate/downloads/nzb"
nzbgetnzbs="/media/Seagate/downloads/nzbget/nzb"

if [ $7 -eq 3 ]; then
	zcat $sabnzbdnzbs/"$2".gz >$nzbgetnzbs/"$2"
fi

Post by **shypike** » January 5th, 2015, 9:08 am

That would be a) worrisome and b) potentially easy to fix.

Can you email one or more sample NZB files?
(Please include the URL of this message and which Usenet provider(s) you are using.)

Post by **shypike** » January 5th, 2015, 10:30 am

Got your email and looked at the logging again.
Do you have multiple servers?
Is the option Config->Switches->"Fail on yenc crc Errors" on?
If so, could you try again with the option off?

The option should make sure that when a primary server delivers a malformed article,
it will be retried on a backup server.
There's a catch: when the backup server doesn't have the article,
I suspect that the original one from the primary servers isn't used.
That would be a weak design.
OTOH there may be just a plain bug here.
SABnzbd should just use the malformed article when there is no backup server at all, but maybe it doesn't.
I'll dig into this. Can you answer the above questions?

thepainless · Post by **thepainless** » January 5th, 2015, 2:15 pm

when I enable the backup block servers. part of the log:

Code: Select all

2015-01-02 18:36:15,311::INFO::[downloader:381] 7@news.dommel.be:119: Initiating connection
2015-01-02 18:36:15,314::INFO::[downloader:381] 2@news.dommel.be:119: Initiating connection
2015-01-02 18:36:15,316::INFO::[downloader:381] 6@news.dommel.be:119: Initiating connection
2015-01-02 18:36:15,318::INFO::[downloader:381] 3@news.dommel.be:119: Initiating connection
2015-01-02 18:36:15,321::INFO::[downloader:381] 4@news.dommel.be:119: Initiating connection
2015-01-02 18:36:15,323::INFO::[downloader:381] 8@news.dommel.be:119: Initiating connection
2015-01-02 18:36:15,326::INFO::[downloader:381] 1@news.dommel.be:119: Initiating connection
2015-01-02 18:36:15,328::INFO::[downloader:381] 5@news.dommel.be:119: Initiating connection
2015-01-02 18:36:16,569::INFO::[downloader:577] Connecting 5@news.dommel.be:119 finished
2015-01-02 18:36:16,576::INFO::[downloader:577] Connecting 7@news.dommel.be:119 finished
2015-01-02 18:36:16,656::INFO::[downloader:577] Connecting 1@news.dommel.be:119 finished
2015-01-02 18:36:16,662::INFO::[downloader:577] Connecting 6@news.dommel.be:119 finished
2015-01-02 18:36:16,668::INFO::[downloader:577] Connecting 4@news.dommel.be:119 finished
2015-01-02 18:36:16,672::INFO::[downloader:577] Connecting 2@news.dommel.be:119 finished
2015-01-02 18:36:16,794::INFO::[downloader:577] Connecting 8@news.dommel.be:119 finished
2015-01-02 18:36:16,805::INFO::[downloader:577] Connecting 3@news.dommel.be:119 finished
2015-01-02 18:36:56,406::INFO::[decoder:121] CRC Error in 54a5fecb$2$34050$b1db1813$443c6447@news.astraweb.com (6BEFF437 -> 616D3873)
2015-01-02 18:36:56,414::INFO::[downloader:381] 1@eunews.blocknews.net:443: Initiating connection
2015-01-02 18:36:57,197::INFO::[downloader:577] Connecting 1@eunews.blocknews.net:443 finished
2015-01-02 18:37:01,717::INFO::[decoder:121] CRC Error in 54a5fecb$2$34050$b1db1813$443c6447@news.astraweb.com (6BEFF437 -> 616D3873)
2015-01-02 18:37:01,725::INFO::[downloader:381] 1@eu.news.astraweb.com:119: Initiating connection
2015-01-02 18:37:01,975::INFO::[downloader:577] Connecting 1@eu.news.astraweb.com:119 finished
2015-01-02 18:37:04,655::INFO::[decoder:121] CRC Error in 54a5fecb$2$34050$b1db1813$443c6447@news.astraweb.com (6BEFF437 -> 616D3873)
2015-01-02 18:37:04,668::INFO::[downloader:381] 1@news.nextgennews.net:443: Initiating connection
2015-01-02 18:37:07,465::INFO::[__init__:919] Saving data for totals9.sab in /home/dweezil/.sabnzbd/admin/totals9.sab
2015-01-02 18:37:10,652::INFO::[downloader:577] Connecting 1@news.nextgennews.net:443 finished
2015-01-02 18:37:14,224::INFO::[decoder:121] CRC Error in 54a5fecb$2$34050$b1db1813$443c6447@news.astraweb.com (6BEFF437 -> 616D3873)
2015-01-02 18:37:14,227::INFO::[decoder:230] <Article: article=54a5fecb$2$34050$b1db1813$443c6447@news.astraweb.com, bytes=793743, partnum=48, art_id=None> => missing from all servers, discarding

same result when I turn off "Fail on yenc crc Errors". I'll email a debug log of the download.

Post by **shypike** » January 5th, 2015, 4:55 pm

Yes, there's a problem in SABnzbd.
Advice: disable the "CRC check" option.

Post by **shypike** » January 20th, 2015, 2:13 pm

It isn't so bad as it looked.
The only difference between enabling and disabling the CRC check
is that when enabled: 1) the other servers are tried and 2) more missing articles are logged.
However, the bad data is still always written into the file.
When the data is actually bad, you'll see par2 starting to repair the files.
So conclusion: when all servers have a CRC error in the same article(s), you will end up with exactly the same downloaded files
independent of the setting of the CRC option.
The only difference is that the "missing" count will be different.
The CRC option is useful in the (less likely) scenario where one of the servers has the correct article.

yannsab · Post by **yannsab** » March 9th, 2015, 4:12 am

Hi, I had exactly the same issue with SABnzbd 0.7.20
I upgraded to 0.7.20 and suddenly most of the download went corrupted and impossible to repair.
I used nzbget ... no pb .... tried again sabnzbd 0.7.20 same pb. So it is not time based.
I have only one server.
I revert back to 0.7.19 and the issue disappeared, could you please fix it for the next release ?
Thanks

Post by **shypike** » March 9th, 2015, 6:28 am

Can you email the NZB that you used to bugs@sabnzbd.org
Please also add the URL of this message and the name of the Usenet service that you use.

Post by **shypike** » March 9th, 2015, 6:43 am

There are no differences between 0.7.19 and 0.7.20 that can explain your observations.
On which operating system do you use SABnzbd?

yannsab · Post by **yannsab** » March 10th, 2015, 5:20 am

I am using it with windows server 2012 R2 and I am using Ngroups.NET
A colleague of mine at work noticed exactly the same issue and he is using a MAC and Ngroups.NET:
- if the post is complete : no pb at all
- but if the post needs repair SABnzbd 0.7.20 fails 100% where NZBget succeeds
I still notice it a little bit with 0.7.19 but much less frequently
Not at home this week, but will try to send you a NZB with this behavior
Thanks !

Post by **shypike** » March 10th, 2015, 7:49 am

An example NZB would be very welcome.
Hopefully it's not the provider that's a factor in this.

Support Forum

0.7.x more missing data blocks than nzbget

0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget

Re: 0.7.x more missing data blocks than nzbget