[problem] Downloading from RSS feed

[john3voltas]

[removed similar post from mytvnzb thread]
I have 2 "servers" running sabnzbd, 1 at home and another 1 at work.
I have tried to download from popular tvnzb/mytvnzb/nzbs.org.
At home it all works fine every time but at work it fails every time.

At home: dsl thomson 585 router pc running sabnzbd+
pc specs: intel coreduo on English Windows XP Pro SP3

At work: dsl zyxel modem pfSense firewall pc running sabnzbd+
pc specs: intel pentium 4 Xeon on English Windows 2003 R2 SP2

Both pcs are running sabnzbd 0.5.0a8 (link provided by impheaux) but both have already been running 0.4.12stable and the issue was present on that release too.

Here's a sabnzbd log from the malfunctioning pc:

Code: Select all

2009-10-31 16:56:14,765::INFO::[sabnzbd:939] --------------------------------
2009-10-31 16:56:14,765::INFO::[sabnzbd:940] SABnzbd.exe-0.5.0Alpha8 (rev=2914)
2009-10-31 16:56:14,765::INFO::[sabnzbd:948] Platform=Windows-2003Server-5.2.3790-SP2 Class=nt
2009-10-31 16:56:14,765::INFO::[sabnzbd:951] Python-version = 2.5.4 (r254:67916, Jan 29 2009, 12:02:11) [MSC v.1310 32 bit (Intel)]
2009-10-31 16:56:14,765::INFO::[sabnzbd:954] Test release, setting maximum logging levels
2009-10-31 16:56:14,765::DEBUG::[lang:125] Language file E:\ptbl_tools\sabnzb\language\us-en.txt, encoding=latin-1, country=United States, language=English
2009-10-31 16:56:14,796::INFO::[__init__:638] Loading data for bookmarks.sab from E:\news\cache\bookmarks.sab
2009-10-31 16:56:14,796::INFO::[__init__:668] E:\news\cache\bookmarks.sab removed
2009-10-31 16:56:14,796::INFO::[__init__:638] Loading data for rss_data.sab from E:\news\cache\rss_data.sab
2009-10-31 16:56:14,796::DEBUG::[scheduler:118] Scheduling RSS interval task every 15 min (delay=5)
2009-10-31 16:56:14,796::DEBUG::[scheduler:129] Scheduling VersionCheck on day 3 at 11:20
2009-10-31 16:56:14,796::INFO::[__init__:638] Loading data for bytes9.sab from E:\news\cache\bytes9.sab
2009-10-31 16:56:14,796::INFO::[__init__:638] Loading data for queue9.sab from E:\news\cache\queue9.sab
2009-10-31 16:56:14,796::INFO::[postproc:141] Loading postproc queue
2009-10-31 16:56:14,796::INFO::[__init__:638] Loading data for postproc1.sab from E:\news\cache\postproc1.sab
2009-10-31 16:56:14,796::INFO::[__init__:668] E:\news\cache\postproc1.sab removed
2009-10-31 16:56:14,796::DEBUG::[downloader:227] Initializing downloader/decoder
2009-10-31 16:56:14,796::INFO::[__init__:638] Loading data for watched_data.sab from E:\news\cache\watched_data.sab
2009-10-31 16:56:14,812::INFO::[downloader:313] Resuming
2009-10-31 16:56:14,812::DEBUG::[__init__:382] PAUSED_ALL inactive
2009-10-31 16:56:14,812::INFO::[__init__:239] All processes started
2009-10-31 16:56:14,812::INFO::[sabnzbd:257] Web dir is E:\ptbl_tools\sabnzb\interfaces\smpl
2009-10-31 16:56:14,812::DEBUG::[lang:125] Language file E:\ptbl_tools\sabnzb\interfaces\smpl\language\us-en.txt, encoding=latin-1, country=United States, language=English
2009-10-31 16:56:14,812::INFO::[sabnzbd:257] Web dir is E:\ptbl_tools\sabnzb\interfaces\Classic
2009-10-31 16:56:14,828::DEBUG::[lang:125] Language file E:\ptbl_tools\sabnzb\interfaces\wizard\language\us-en.txt, encoding=latin-1, country=United States, language=English
2009-10-31 16:56:14,828::INFO::[sabnzbd:1008] Starting SABnzbd.exe-0.5.0Alpha8
2009-10-31 16:56:14,828::DEBUG::[__init__:250] Starting postprocessor
2009-10-31 16:56:14,828::DEBUG::[__init__:253] Starting assembler
2009-10-31 16:56:14,828::INFO::[nzbqueue:95] Saving queue
2009-10-31 16:56:14,828::DEBUG::[__init__:256] Starting downloader
2009-10-31 16:56:14,842::INFO::[__init__:617] Saving data for queue9.sab in E:\news\cache\queue9.sab
2009-10-31 16:56:14,842::DEBUG::[scheduler:154] Starting scheduler
2009-10-31 16:56:14,842::DEBUG::[__init__:261] Starting dirscanner
2009-10-31 16:56:14,842::DEBUG::[newzbin:101] Starting msgidgrabber
2009-10-31 16:56:14,842::INFO::[dirscanner:372] Dirscanner starting up
2009-10-31 16:56:14,842::DEBUG::[__init__:266] Starting urlgrabber
2009-10-31 16:56:14,842::INFO::[urlgrabber:104] URLGrabber starting up
2009-10-31 16:56:14,842::INFO::[sabnzbd:374] _yenc module... found!
2009-10-31 16:56:14,842::INFO::[sabnzbd:382] par2 binary... found (E:\ptbl_tools\sabnzb\win\par2\par2.exe)
2009-10-31 16:56:14,842::INFO::[sabnzbd:387] par2-classic binary... found (E:\ptbl_tools\sabnzb\win\par2\par2-classic.exe)
2009-10-31 16:56:14,842::INFO::[sabnzbd:390] unrar binary... found (E:\ptbl_tools\sabnzb\win\unrar\UnRAR.exe)
2009-10-31 16:56:14,842::INFO::[sabnzbd:395] unzip binary... found (E:\ptbl_tools\sabnzb\win\unzip\unzip.exe)
2009-10-31 16:56:14,842::INFO::[sabnzbd:410] pyOpenSSL... found (True)
2009-10-31 16:56:14,842::INFO::[sabnzbd:1102] Starting web-interface on 0.0.0.0:8080
2009-10-31 16:56:14,842::INFO::[__init__:617] Saving data for bytes9.sab in E:\news\cache\bytes9.sab
2009-10-31 16:56:14,842::INFO::[__init__:617] Saving data for rss_data.sab in E:\news\cache\rss_data.sab
2009-10-31 16:56:14,858::INFO::[__init__:617] Saving data for bookmarks.sab in E:\news\cache\bookmarks.sab
2009-10-31 16:56:14,858::INFO::[__init__:617] Saving data for watched_data.sab in E:\news\cache\watched_data.sab
2009-10-31 16:56:14,875::INFO::[postproc:136] Saving postproc queue
2009-10-31 16:56:14,875::INFO::[__init__:617] Saving data for postproc1.sab in E:\news\cache\postproc1.sab
2009-10-31 16:56:16,828::INFO::[_cplogging:55] [31/Oct/2009:16:56:16] ENGINE Bus STARTING
2009-10-31 16:56:16,828::INFO::[_cplogging:55] [31/Oct/2009:16:56:16] ENGINE Started monitor thread '_TimeoutMonitor'.
2009-10-31 16:56:17,046::INFO::[_cplogging:55] [31/Oct/2009:16:56:17] ENGINE Serving on 0.0.0.0:8585
2009-10-31 16:56:17,265::INFO::[_cplogging:55] [31/Oct/2009:16:56:17] ENGINE Serving on 0.0.0.0:8080
2009-10-31 16:56:17,265::INFO::[_cplogging:55] [31/Oct/2009:16:56:17] ENGINE Bus STARTED
2009-10-31 16:56:17,265::INFO::[misc:434] Lauching browser with https://localhost:8585/sabnzbd
2009-10-31 16:56:18,703::DEBUG::[misc:544] Checked for a new release, cur= 50008, latest= 0 (on )
2009-10-31 16:56:29,796::INFO::[__init__:617] Saving data for rss_data.sab in E:\news\cache\rss_data.sab
2009-10-31 17:01:14,796::INFO::[__init__:617] Saving data for rss_data.sab in E:\news\cache\rss_data.sab
2009-10-31 17:08:47,108::DEBUG::[rss:244] Running feedparser on http://mytvnzb.foechoer.be/feed/id/72615470692.rss
2009-10-31 17:08:53,125::DEBUG::[rss:246] Done parsing http://mytvnzb.foechoer.be/feed/id/72615470692.rss
2009-10-31 17:08:53,125::WARNING::[rss:253] Failed to retrieve RSS from http://mytvnzb.foechoer.be/feed/id/72615470692.rss: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Found

Here's the part I find relevant:

2009-10-31 16:56:17,265::INFO::[misc:434] Lauching browser with https://localhost:8585/sabnzbd
2009-10-31 16:56:18,703::DEBUG::[misc:544] Checked for a new release, cur= 50008, latest= 0 (on )
2009-10-31 16:56:29,796::INFO::[__init__:617] Saving data for rss_data.sab in E:\news\cache\rss_data.sab
2009-10-31 17:01:14,796::INFO::[__init__:617] Saving data for rss_data.sab in E:\news\cache\rss_data.sab
2009-10-31 17:08:47,108::DEBUG::[rss:244] Running feedparser on http://mytvnzb.foechoer.be/feed/id/72615470692.rss
2009-10-31 17:08:53,125::DEBUG::[rss:246] Done parsing http://mytvnzb.foechoer.be/feed/id/72615470692.rss
2009-10-31 17:08:53,125::WARNING::[rss:253] Failed to retrieve RSS from http://mytvnzb.foechoer.be/feed/id/72615470692.rss: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was: Found

It says done 'parsing'.
By that I would imagine that the feed had been downloaded and parsed.
Bu then there's a 'failed to retrieve' message.
I am lost and I don't know what else I can do to solve this mystery.
My only hope is that someone here with a similar problem can give me advise.
Thanks in advance.
Cheers

[shypike]

Both pcs are running sabnzbd 0.5.0a8 (link provided by impheaux) but both have already been running 0.4.12stable and the issue was present on that release too.

It says done 'parsing'.
By that I would imagine that the feed had been downloaded and parsed.
Bu then there's a 'failed to retrieve' message.
I am lost and I don't know what else I can do to solve this mystery.
My only hope is that someone here with a similar problem can give me advise.
Thanks in advance.
Cheers

I checked with your exact URL in 0.5.0A8 and 0.4.12.
In both cases it just worked (giving a single NZB file as result).

You are sure it's not a temporary problem at the mytvnzb.foechoer.be side?
When it went wrong, did you immediately test the URL with the browser?
The pfSense firewall doesn't interfere?

[john3voltas]

Hi Shypike.
Great to see that I captured some of your attention .
As said, at home I can use 0.4.12 or 0.5.0a8 because the rss feed always works.
On the contrary, from work I can use 0.4.12 or 0.5.0a8 because it doesn't matter: it always fails.
At work (where it fails) using Firefox 3.5.4 a few seconds after an error message from SAB, I can open any RSS feed that I want, make it Mytvnzb or binsearch.info or nzbs.org and it never fails me.
That's using the browser to look at the feed which seems to get the servers out of the equation.
SAB doesn't look to be the bad guy either because it works from home.
I even deleted the sabnzbd.ini and started a fresh new config just with an RSS feed and yet it fails at work using SAB (0.4 or 0.5 it doesn't matter).
pfSense? Well, I honestly don't see why since the browser can go through pfS and get the feeds...
Can I launch a simple python script by hand and look at some logs to see what's going on behind the curtains of the browser gui?
I'm quite positive that SAB is not at fault but it could help me realize who's at fault here.
Thanks in advance for all your help.
Cheers

[shypike]

Are you using a proxy at work (this is quite normal in companies) ?
We do not support any proxy.

[john3voltas]

No, pfSense is not set up for proxy and there isn't any other server set up for that.
Please remember that from the very same PC where SAB is installed I can launch firefox and read those RSS feeds without any problems.
TIA
Cheers

[shypike]

Supposing the connections at work are slower then at home (maybe due to the firewall).
This could be a problem if the standard library functions we use are too impatient (short timeouts).
Must discuss this with the team.

[john3voltas]

Hi shypike,
Thanks for all your help on this.
Today I made a test with a 3rd computer at workplace and it worked out fine.
So I guess this has nothing to do with SAB or Mytvnzb or with pfSense.
This sounds like an issue with the PCs I'm using, more likely a network issue.
I'll keep on looking for what is causing this but I believe this is more my issue than SAB's issue.
I'll let you know if I find anything else relevant.
Keep on the good work ;-).
Cheers

[john3voltas]

Ok, I've got it.
We also have a M$ ISA server at work but it is not the default gateway of the PC running SAB.
When I said that
SAB pc > switch > pfSense (no proxy) > DSL modem
I was right.
ISA is there for some other PCs and SAB's PC default gateway is pfSense and after that I only have the DSL modem.
Let me remind you that I am using FireFox 3.5.4 on SAB's PC.
But...some months ago SAB's PC had been behind the ISA server and it's IE8 had a proxy setting enabled.
Again, I am not using IE8 for anything.
As soon as I removed IE8 proxy setting, SAB started receiving RSS feeds from all providers.
Can you check your code and see if there's something that uses IE8?
To me the best thing would be for SAB to be completely IE independent if possible.
Regards.

[shypike]

SABnzbd is not depending on IE8 at all.
It will just start up IE8 when that browser is the default browser.
You can choose not to have SABnzbd start up the browser.

[john3voltas]

There's something strange then.
When I check the "use proxy" in IE8, SAB cannot get the RSS feeds.
On the contrary when I leave that option unchecked, SAB can get all RSS feeds that I throw at it.
There's a cause and effect.
All I can assure you is that SAB's PC default gateway is not ISA server, it is pfSense server.
This means that SAB's PC doesn't use ISA as a proxy.
Never mind. All that matters is that SAB is working perfectly with RSS feeds.
This case should only be noted in case someone else runs into RSS feed issues in the future.
Regards

[shypike]

When I check the "use proxy" in IE8, SAB cannot get the RSS feeds.
On the contrary when I leave that option unchecked, SAB can get all RSS feeds that I throw at it.
There's a cause and effect.

Could be that somehow the Python runtime library finds out that the default browser
has a proxy server set and decides to use it.
For obvious reasons I do not test this situation.
The only place I have a proxy is at work and I do not use SABnzbd at work

[john3voltas]

Then I'd ask you to do me a favor.
Let's say that you run SAB only at home on your own computer and that your computer runs on Win32.
Could you please make IE your default browser and set a phony proxy server (say 10.10.10.10:9898) and then run SAB and throw an RSS feed into it and check if it works?
I'll do it at home too but I'd like to have more than my own experience .
Cheers

[shypike]

It looks like your right about this.
The feedparser and the nzb-fetcher both try to use the proxy configured in IE.
And IE isn't even my default browser.

It seems like you uncovered a serious problem in the Python HTTP library!

I don't know if we can/will solve this problem.
At least we could file a complaint at www.pyhton.org. ActiveState.com.
It's a known problem, see:
http://bytes.com/topic/python/answers/8 ... ls-connect

[john3voltas]

Woohoo! At least this is a generic issue and not my own particular issue.
Since they already have a ticket open, let's hope those guys fix it in the near future.
In the meantime I'll remove IE proxy settings on that PC even because it's not really needed.
Then again, I just looked again at that forum thread from bytes.com forum and it seems that thread is over one year old.
And it doesn't look as if they're worried about that issue...
Having said that, I don't believe there would be a way to circumvent that issue in SAB, would there?
Nevertheless thanks a lot shypike.
Cheers

[shypike]

We could circumvent it but not without re-wrting part of the library.
The problem affects too few people to do that.