Files get corrupted

[kouze]

Hi guys,

I'm having an issue downloading files from an .nzb, which do not have any issue when I use newsleecher to download them (no need to repair them)

Here is the .nzb file: http://dbox.plouc.eu/sabnzbd/6b59e6b338 ... 9ced0f.nzb

it contains rar files + .par2 file. The rar are encrypted.

Thanks !
kouze

[shypike]

Bad naming convention.
The PAR2 group has a different name than the RAR group.
SABnzbd is more strict in handling named sets, in order to handle multiple rar/par sets in one NZB.
Future releases will be more lenient.

[kouze]

OK, will try with good par set then

[kouze]

But I'm confused. Why SABnzbd is making corrupted data when I download the files using Processing = none ?
It should then not take into account any naming convention, and just download the data, right ? But here again, the files get corrupted (I'm checking them using QuickPar, no issue with Newsleecher)

[shypike]

Just downloaded it.
It's not the naming; there are just an awful lot of articles missing.
Are you using the same Usenet server for SABnzbd and NL?

[kouze]

Yes, they got posted on giganews, and I'm using both giganews for SABnzbd and NL...

[shypike]

Hey, I just checked using nzbget on GigaNews.
The download is corrupt: each RAR file has a significant part missing.
You sent me a bad NZB or NL is able to perform magic. But I doubt that.

Code: Select all

Repair is required.
11 file(s) exist but are damaged.
You have 1574 out of 2002 data blocks available.
You have 83 recovery blocks available.
Repair is not possible.
You need 345 more recovery blocks to be able to repair.

[kouze]

I also tested on nzbget, and I'm getting the same result as you. But again, no issue with NL

the .nzb was generated by newsmangler. Does it have an incorrect format ?

[shypike]

The only odd thing is that it doesn't contain newlines,
but that's allowed in XML.
Another team member gets even worse results with Newsbin.
(Albeit with Astraweb)
For me Astraweb or Giiganews and SABnzbd or nzbget give the same results.

If I look at the byte counts of the NZB file, the files are the size they should be.
Still par2 says that about 20% valid data is missing.
Can you PM me the password, so that I can check whether the RAR files are correct after all?

[shypike]

Well, this is a very bizarre NZB.
NZB files number the parts of each file.
The yEnc encoding protocol also numbers the parts of a file.
In 99.99999% of NZB files, the numbering is identical with that of yEnc.
But not in this NZB!
It looks like NL ignores the part-numbering in the NZB, but uses only the yEnc numbering.
SABnzbd and nzbget use the NZB numbering.
Newsbin sees that something is wrong and refuses to process the files.
NL uses the yEnc numbering exclusively.
In this case, that's a good idea. Although I am a bit wary of side-effects.

BTW: it's not the posters fault, because the numbering in the subject-lines matches that of yEnc.
Newsmangler does things wrong. The NZB protocol defines that the numbering should
match the actual order of the parts.
The yEnc numbering makes it possible for NL to fix things, but a differently encoded post
would be lost this way.

Could you also complain to newsmangler?

I must see whether this issue is worth pursuing further.

[kouze]

ok I got the issue I think. It's the .nzb which is not good.

If you have a look at this .nzb http://dbox.plouc.eu/sabnzbd/6b59e6b338 ... d0f_NL.nzb generated by NL with the headers from boneless, you will see that the segments data are not matching

If you search for file 6b59e6b3382681f7f70b609a459ced0f.part01.rar for example, first segment, in the NL file you will have:
<segment number="1" bytes="784562">1349165392.13165.1@news-europe.giganews.com</segment>

in the first nzb file, generated by newsmangler, I have
<segment bytes="793091" number="1">1349165392.13168.2@news-europe.giganews.com</segment>

but NL is still able to recreate the good file...

I'll contact newsmangler's guy to try to understand that

[kouze]

Ok you replied before me
I'll contact newsmangler's developper to check if he can adapt the .nzb generation.

thanks for your time shypike !

[kouze]

I actually fixed the newsmangler issue by sorting "correctly" the nzb output.
But it's just a workaround working with giganews because the ids segment are "increasing", so easy to sort them out.

[shypike]

I've looked in the SABnzbd code.
"Fixing" it to accommodate this is no trivial matter.
The current yEnc numbering is simply ignored.
Also, it should not be necessary because the NZB format is clear that
the article numbering should be correct.
I'll make a note of this, but it won't get high priority.

[kouze]

thanks shypike, but I don't think you or your team should spend time trying to "fix" it. As you said, the nzb format is clear, then it must be respected.

newsmangler seems to be more a proof of concept than anything else (while working correctly), I don't know if the dev will fix this .nzb issue (I don't know the complexity and I don't know Python). Anyway "my" fix is working for my need

but if you know another CLI NNTP Poster under linux that can create nzb file, I will give a try

Thanks !