"No response from indexer" on QNAP

[haplm]

Hi,

I have a problem with sabnzbd on my QNAP (new installation).

After SickRage sends NZB URL for processing, it never gets downloaded, and the log is full of messages like:

Code: Select all

2015-09-08 17:03:32,115::INFO::[urlgrabber:116] Grabbing URL https://api.nzbgeek.info/api?t=get&id=xx&apikey=yy
2015-09-08 17:03:32,228::DEBUG::[urlgrabber:372] No response from indexer, retry after 60 sec
2015-09-08 17:03:32,228::INFO::[urlgrabber:199] Retry URL https://api.nzbgeek.info/api?t=get&id=xx&apikey=yy

I've found post/thread https://forums.sabnzbd.org/viewtopic.php?t=18509#p98171, and ran the test recommended there with this result:

Code: Select all

[~] # python test.py

header is:
Date: Tue, 08 Sep 2015 17:49:40 GMT
Server: Apache/2.4.7 (Ubuntu)
Strict-Transport-Security: max-age=63072000; includeSubDomains
Strict-Transport-Security: max-age=63072000; includeSubDomains
X-Powered-By: PHP/5.5.9-1ubuntu4.11
X-DNZB-MoreInfo: 
Content-Disposition: attachment; filename=Homeland.S04E08.720p.BluRay.X264-REWARD.nzb
Content-Type: application/x-nzb
Connection: close


First 5 lines from webserver response:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE n
[~] # 

So to me, everything seems to be working correctly. What else I can try to debug what's going on?

Thanks!

[sander]

In the code above, which URL did you use? HTTP or HTTPS? Correct API key, or not?

On your QNAP can you run this:

Code: Select all

import urllib2
f = urllib2.urlopen('https://api.nzbgeek.info/api?t=get&id=xx&apikey=yy', timeout=5)
f.read()[:100]

Post the code block here in this thread.

This is good output:

Code: Select all

$ python
Python 2.7.9 (default, Apr  2 2015, 15:33:21) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> f = urllib2.urlopen('https://api.nzbgeek.info/api?t=get&id=xx&apikey=yy', timeout=5)
>>> f.read()[:100]
'<error code="100" description="Invalid API Key"/>'
>>> 

[haplm]

Https URL, correct API key.

>>> import urllib2
>>> f = urllib2.urlopen('https://api.nzbgeek.info/api?t=get&id=xxx&apikey=yyy', timeout=5)
>>> f.read()[:100]
'<?xml version="1.0" encoding="UTF-8"?>\n<!DOCTYPE nzb PUBLIC "-//newzBin//DTD NZB 1.1//EN" .....

All this seem correct, right?

[haplm]

This is content of the script I ran originally:

Code: Select all

import urllib

url = 'https://api.nzbgeek.info/api?t=get&id=xxx&apikey=yyy'

opener = urllib.FancyURLopener({})
fn, header = opener.retrieve(url)
print "\nheader is:\n", header
print "\nFirst 5 lines from webserver response:"
with open(fn) as myfile:
   print myfile.read(50)

[sander]

Yes, looks correct. What if you use HTTP instead of HTTPS in the SABnzbd / SickRage RSS URL ?


And can you put this into SABnzbd RSS settings: https://nzbindex.com/rss/?q=hello+nothi ... esc&max=25, and activate it?

[haplm]

Unsurprisingly, the situation is the same:

Code: Select all

2015-09-09 16:46:11,121::INFO::[urlgrabber:116] Grabbing URL http://api.nzbgeek.info/api?t=get&id=xxx&apikey=yyy
2015-09-09 16:46:11,300::DEBUG::[urlgrabber:372] No response from indexer, retry after 60 sec

However, when I try to test connection to a news server, it works, so there has to be connectivity...

Anything else I can try?

[haplm]

Strange, I'm getting SSL error while trying to get the RSS feed:

Code: Select all

Failed to retrieve RSS from https://nzbindex.com/rss/?q=hello+nothing&sort=agedesc&max=25: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

However, this doesn't explain why I can't get a simple Http URL either...

[haplm]

Http for RSS works:

Code: Select all

RSS Feed http://nzbindex.com/rss/?q=hello+nothing&sort=agedesc&max=25 was empty

[haplm]

Sorry for so many replies, but now I see why Http doesn't work either:

Code: Select all

MartinH:~ haplm$ wget http://api.nzbgeek.info/api?t=get&id=xxx&apikey=yyy
[1] 10046
[2] 10047
MartinH:~ haplm$ --2015-09-09 19:16:18--  http://api.nzbgeek.info/api?t=get
Resolving api.nzbgeek.info... 212.71.250.176, 46.38.48.29, 109.236.89.135
Connecting to api.nzbgeek.info|212.71.250.176|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://api.nzbgeek.info/api?t=get [following]
https://api.nzbgeek.info/api?t=get: [b]HTTPS[/b] support not compiled in.

There's a redirect to Https...

[sander]

Thanks for the analysis: useful.

We now have two options:
1) Further analysis, leading to the root cause
2) Assume your QNAP SSL setup is corrupt (all analysis points in that direction). Read my Jan 2015 post https://forums.sabnzbd.org/viewtopic.ph ... 565#p98483 and do that, meaning adding a few lines to sabnzbd as a workaround for the incorrect setup on your QNAP (FYI: The same QNAP problem at user @mogodon http://forums.sabnzbd.org/viewtopic.php ... 15#p101719 was solved with work around)

[haplm]

We can continue with further investigation if you want to find the root cause, no problem.

[sander]

Further analysis: Get the code from https://raw.githubusercontent.com/sande ... -tester.py and run it on your QNAP. Post the output here in a code block.

[haplm]

Output:

Code: Select all

[~] # python http-https-tester.py 
Python version 2.7.3 (default, Aug 18 2015, 01:08:10) 
[GCC 4.9.1]
URL  http://www.google.com/ OK, with result <!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="cs"><head><meta content
URL  https://www.google.com/ OK, with result <!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="cs"><head><meta content
URL  http://api.nzbgeek.info/api?t=get&id=xxx&apikey=yy OK, with result <error code="100" description="Invalid API Key"/>
URL  https://api.nzbgeek.info/api?t=get&id=xxx&apikey=yy OK, with result <error code="100" description="Invalid API Key"/>
URL  https://tv.eurosport.nl/ not OK, with error  <class 'ssl.SSLError'>
URL  http://does.not.exist/ not OK, with error  <class 'urllib2.URLError'>
Finished

Interesting is what happened when I tried to download the script:

Code: Select all

[~] # wget https://raw.githubusercontent.com/sanderjo/stuff/master/http-https-tester.py
--2015-09-10 18:40:02--  https://raw.githubusercontent.com/sanderjo/stuff/master/http-https-tester.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 23.235.43.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|23.235.43.133|:443... connected.
ERROR: cannot verify raw.githubusercontent.com's certificate, issued by ‘/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA’:
  Unable to locally verify the issuer's authority.
To connect to raw.githubusercontent.com insecurely, use `--no-check-certificate'.

[sander]

Hmmm, not as expected:

- python 2.7.3. I had expected at least 2.7.9.
- no problems with https://api.nzbgeek.info/. I had expected problems there

Strange. You're running that on the QNAP, right? And there is only one python executable on the QNAP?

And the error with "wget https://raw.githubusercontent.com/sande ... -tester.py" points in the same directions; problems with SSL certificates.

Can you re-download and re-run the script? I've added the github-URL into it. Let's see what happens.

[haplm]

OK, so this is getting messy.

These are all python links on the system:

Code: Select all

[/] # find -type l -name python
./mnt/ext/opt/Python/bin/python
./share/CACHEDEV1_DATA/.qpkg/container-station/bin/python
./share/CACHEDEV1_DATA/.qpkg/container-station/usr/local/python/bin/python
./share/CACHEDEV1_DATA/.qpkg/SABnzbdplus/Repository/lib/python
./share/CACHEDEV1_DATA/.qpkg/SABnzbdplus/x86/bin-utils/python
./share/CACHEDEV1_DATA/.qpkg/SickBeard-TVRage/Repository/lib/python
./share/CACHEDEV1_DATA/.qpkg/XBMC/bin/python
./share/CACHEDEV1_DATA/.qpkg/GoogleCloudStorage/python/bin/python
./share/CACHEDEV1_DATA/.qpkg/CloudLink/python/bin/python
./share/CACHEDEV1_DATA/.qpkg/CloudDrive/python/bin/python
find: File system loop detected; ‘./share/CACHEDEV1_DATA/.qpkg/HD_Station/share/CACHEDEV1_DATA’ is part of the same file system loop as ‘./share/CACHEDEV1_DATA’.
./share/CACHEDEV1_DATA/.qpkg/HD_Station/usr/bin/python
./usr/local/python
./usr/local/bin/python
./usr/bin/python

And files:

Code: Select all

[/] # find -type f -name python
./share/CACHEDEV1_DATA/.qpkg/Python/bin/python
./share/CACHEDEV1_DATA/.qpkg/QKVM/opt/bin/python
find: File system loop detected; ‘./share/CACHEDEV1_DATA/.qpkg/HD_Station/share/CACHEDEV1_DATA’ is part of the same file system loop as ‘./share/CACHEDEV1_DATA’.
./share/CACHEDEV1_DATA/.qpkg/HD_Station/usr/share/bash-completion/completions/python
./share/CACHEDEV1_DATA/.qpkg/HD_Station/usr/share/lintian/overrides/python

Then:

[/] # which python
/share/CACHEDEV1_DATA/.qpkg/container-station/bin/python

[/] # /share/CACHEDEV1_DATA/.qpkg/container-station/bin/python
Python 2.7.3 (default, Aug 18 2015, 01:08:10)
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.

The one visible in the UI (and installed as a dependency of Sabnzbd QNAP "official" package) claims that it is 2.7.3 in the name of the package, but that's apparently not true:

[/] # /share/CACHEDEV1_DATA/.qpkg/Python/bin/python
Python 2.7.9 (default, Mar 24 2015, 00:42:24)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Which explains why I see this in Sabnzbd setup page:

Code: Select all

Python Version:	2.7.9 (default, Mar 24 2015, 00:42:24) [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)]

So finally your script.

2.7.9:

[~] # /share/CACHEDEV1_DATA/.qpkg/Python/bin/python http-https-tester.py
Python version 2.7.9 (default, Mar 24 2015, 00:42:24)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)]
URL http://www.google.com/ OK, with result <!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="cs"><head><meta content
URL https://www.google.com/ not OK, with error <class 'urllib2.URLError'>
URL http://api.nzbgeek.info/api?t=get&id=xxx&apikey=yy not OK, with error <class 'urllib2.URLError'>
URL https://api.nzbgeek.info/api?t=get&id=xxx&apikey=yy not OK, with error <class 'urllib2.URLError'>
URL https://tv.eurosport.nl/ not OK, with error <class 'urllib2.URLError'>
URL http://does.not.exist/ not OK, with error <class 'urllib2.URLError'>
URL https://nzbindex.com/ not OK, with error <class 'urllib2.URLError'>
URL https://raw.githubusercontent.com/ not OK, with error <class 'urllib2.URLError'>
Finished

2.7.3

[~] # /share/CACHEDEV1_DATA/.qpkg/container-station/bin/python http-https-tester.py
Python version 2.7.3 (default, Aug 18 2015, 01:08:10)
[GCC 4.9.1]
URL http://www.google.com/ OK, with result <!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="cs"><head><meta content
URL https://www.google.com/ OK, with result <!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="cs"><head><meta content
URL http://api.nzbgeek.info/api?t=get&id=xxx&apikey=yy OK, with result <error code="100" description="Invalid API Key"/>
URL https://api.nzbgeek.info/api?t=get&id=xxx&apikey=yy OK, with result <error code="100" description="Invalid API Key"/>
URL https://tv.eurosport.nl/ not OK, with error <class 'ssl.SSLError'>
URL http://does.not.exist/ not OK, with error <class 'urllib2.URLError'>
URL https://nzbindex.com/ OK, with result <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd
URL https://raw.githubusercontent.com/ OK, with result <!DOCTYPE html> <html lang="en" class=""> <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns
Finished

So the problem is obvious - the Python package installed as Sabnzbd dependency doesn't handle certificates (and my system has too many pythons to my taste, however all is result of installing relatively modest amount of packages through UI).

Install page for QNAP: http://forum.qnap.com/viewtopic.php?f=133&t=86644