Empty NZBs and 404 Not Found

[ALbino]

I've been getting a fair amount of Empty NZBs lately as reported by SAB. Here's an example error line:

Code: Select all

2015-04-17 12:00:00,265	WARNING:	Empty NZB file 0123456789.nzb [https://www.indexer.com/fail/abcdefghijklmnopqrstuvwxyz/0123456789]

It's the same output in the +Debug log.

It tends to happen on fail replacements, but if you manually open the URL for the fail request it doesn't return a bad, empty, or malformed NZB. Instead it just returns: HTTP/1.1 404 Not Found.

Reading through the older posts about Empty NZBs on this forum, it seems it might be an index-specific problem. How does SABnzbd handle 404 Not Founds? Is 404 Not Found the incorrect response from the indexer? Empty NZB doesn't seem like the proper SAB error for a non-existent file, but maybe I'm not looking at it the right way. I contacted the admin of the indexer and he seems unsure what the correct response and error from both sides should be. How is SAB ending up with a request for a bad URL in the first place?

There's other reports of this happening on their forums, so it's not specific to me or my setup. If there's a better way for them to handle this then let me know and I'll gladly pass it along.

(Windows 7/SAB 0.7.20)

[jcfp]

It tends to happen on fail replacements, but if you manually open the URL for the fail request it doesn't return a bad, empty, or malformed NZB. Instead it just returns: HTTP/1.1 404 Not Found.

Sab probably expects to get a valid nzb from such an url. So when the indexer sends some html page instead, the xml structure of an nzb file is missing entirely and the response thus appears (effectively) empty when parsed as an nzb. In the past similar issues sometimes came up with indexers that were broken/overloaded or failed to embed authentication data in their urls, and responded with assorted html failure/login pages when sab tried to grab an nzb.

How is SAB ending up with a request for a bad URL in the first place?

From the indexer via X-DNZB-Failure header?

[shypike]

Often indexers require authentication info in the URL.
Where do you get the URLs from?

[ALbino]

As jcfp suggested, the URL is coming from the indexer via X-DNZB-Failure. The URLs contain my API key for authentication, and most of the time the provided URLs work, but sometimes there's nothing there and the response is 404. In the example above the API key would be "0123456789" and the NZB ID would be "abcdefghijklmnopqrstuvwxyz", though I'm not sure why SAB reports the NZB name as my API key (in this case 0123456789.nzb). The line effectively looks like this:

Code: Select all

Empty NZB file API-KEY.nzb [https://www.indexer.com/fail/NZB-ID/API-KEY]

If I go to the page for the actual NZB that failed, the URL is:

Code: Select all

https://www.indexer.com/details/NZB-ID

And that NZB exists, though obviously it fails to download, which is why SAB requests the replacement NZB in the first place. Once it fails I get the Empty NZB error associated with this URL in my log:

Code: Select all

https://www.indexer.com/fail/NZB-ID/API-KEY

Here's a more complete Debug log if it helps at all:

Code: Select all

Repair is not possible.
You need 732 more recovery blocks to be able to repair.
2015-04-18 14:32:09,367::INFO::[newsunpack:868] Par verify failed on C:\download\ExampleFileName.vol000+01.par2!
2015-04-18 14:32:09,367::DEBUG::[__init__:844] Saving data for __verified__ in C:\download\__ADMIN__
2015-04-18 14:32:09,368::INFO::[postproc:622] Par2 check finished on ExampleFileName
2015-04-18 14:32:09,377::INFO::[postproc:543] Cleaning up ExampleFileName (keep_basic=True)
2015-04-18 14:32:09,378::INFO::[misc:1329] Cannot remove folder C:\download\ExampleFileName\__ADMIN__
2015-04-18 14:32:09,378::INFO::[misc:1329] Cannot remove folder C:\download\ExampleFileName\__ADMIN__
2015-04-18 14:32:09,378::INFO::[__init__:513] Fetching https://www.indexer.com/fail/NZB-ID-abcdefghijklmnopqrstuvwxyz/API-KEY-0123456789
2015-04-18 14:32:09,378::INFO::[nzbqueue:218] Saving queue
2015-04-18 14:32:09,378::DEBUG::[__init__:844] Saving data for SABnzbd_nzo_rria_m in C:\Users\UserName\AppData\Local\sabnzbd\admin\future
2015-04-18 14:32:09,380::INFO::[__init__:919] Saving data for queue9.sab in C:\Users\UserName\AppData\Local\sabnzbd\admin\queue9.sab
2015-04-18 14:32:09,380::INFO::[postproc:85] Saving postproc queue
2015-04-18 14:32:09,380::INFO::[urlgrabber:116] Grabbing URL https://www.indexer.com/fail/NZB-ID-abcdefghijklmnopqrstuvwxyz/API-KEY-0123456789
2015-04-18 14:32:09,380::INFO::[__init__:919] Saving data for postproc1.sab in C:\Users\UserName\AppData\Local\sabnzbd\admin\postproc1.sab
2015-04-18 14:32:09,381::INFO::[downloader:246] Post-processing finished, resuming download
2015-04-18 14:32:11,079::INFO::[nzbstuff:675] Replacing spaces with underscores in API-KEY-0123456789.nzb
2015-04-18 14:32:11,079::INFO::[misc:811] Creating directories: C:\download\incomplete\API-KEY-0123456789
2015-04-18 14:32:11,095::WARNING::[nzbstuff:744] Empty NZB file API-KEY-0123456789.nzb [https://www.indexer.com/fail/NZB-ID-abcdefghijklmnopqrstuvwxyz/API-KEY-0123456789]

FWIW, if it matters, this is a very popular indexer, and not some fly by night operation. I'm guessing you have an account, and the error is still reproducible for me several days later on the same NZB, so if you want I'll send you a link to the actual NZB and you can try it yourself.

[shypike]

The weird file naming is probably due to the indexer not sending a proper name.
Instead, SABnzbd will use a derivative from the URL.
404 errors are really a problem of the indexer.
With many other error codes, SABnzbd will retry, but not with 404.
Internet protocols define 404 as a permanent error, that should not be retried.
Can you PM or email me some details about the indexer?

[ALbino]

PM sent. Thanks!