My ISP has blocked newzbin2.com newzbin2.es - proxy / tunnel

[nickb834]

Hi all,

I've been using sabnzdb for a number of years now and have hit a problem totally not Sabnzbds fault - but I'd like suggestions of how to get round it.

My ISP (Sky UK) has blocked newzbin2.com and newzbin2.es via the well known ip addresses (those available via public DNS).

For various reasons I can't switch provider (I like my Sky Anytime Plus into HD box - need to stay "on net" for now to get it plus other "stuff", - so assume switching not an option).

I have hacked about with "Pies automatic downloader" for a few years also that IIRC was written for Hellanzb, such that it scrapes myepisodes.com for the tv eps I'm interested in, then using the appropriate API key posts the newzbin2.es message id into Sabnzbd and off Sabnzbd goes to grab it and so on.

Now the catch is - my isp has put a block on the ip's - such that I can still resolve newzbin2.es but can't connect (telnet to IP on 80 / 443 never connects - tcpdump shows RST's -ergo firewalled).

I've set up an SSH tunnel to a VPS (virt private server) that I have in the US for proxying access to Hulu - now - given that Sabnzbd and my Pads script are both Python AND I can't use socksipy (having read a few posts on here regards proxies), I've put a hosts file entry to point http://www.newzbin2.es at my localhost so I can send it over an ssh tunnel:

ssh -f -N -q -L 5150:www.newzbin2.es:80 user@proxy.somedomain.com

This sends any traffic that hits my local host (newzbin2.es is set to 127.0.0.1 in my hosts remember) on port 5150 (nowt special just a port regularly used in Firefox socks proxy over ssh guides) to http://www.newzbin2.es on port 80 via my proxy in the US.

This gets my message id's into Sabnzbd quite nicely, but - Sabnzbd still won't download / connect to http://www.newzbin2.es (presumably as it's hitting it on port 80?).

As I see it my choices are:

1) wait for proxy support (loooooow priority I hear)
2) hack away at the code to add in a port variable (I'm no kind of python guy) so I can point it at http://www.newzbin2.es:5150 and get it over my tunnel
3) try and configure a transparent proxy on the host sabnzbd runs on - and rewrite the requests to point where I want them
4) try an SSH VPN and route all outbound traffic via my us proxy - don't like, extra interfaces / NAT / iptables / routing - takes all fun out of it :-)

I think option 2) is most likely - but before I have at it, does anyone else have a proven method for running a Sabnzbd installation on an ISPs connection that blocks access to newzbin2.es?

FYI - my install is thus:

10.04 Ubuntu LTS (I only upgrade to LTS versions so 12.04 is next in a couple of months)
Sabnzbdplus 0.6.15 (Upgraded this AM as it happens to a PPA version a kind soul set up - this got me pointed at newzbin2.es)
Sky UK ISP

Cheers

Nick

[nickb834]

BTW - this is what's in my logs:

Code: Select all

 File "/usr/share/sabnzbdplus/sabnzbd/newzbin.py", line 170, in _grabnzb
    conn.request('POST', fetchurl, postdata, headers)

error: [Errno 111] Connection refused

So I guess I'm hacking away at /usr/share/sabnzbdplus/sabnzbd/newzbin.py then :-)

[nickb834]

Actually - looks like it's this file

cfg.py
courtesy of this line in newzbin.py

Code: Select all

(cfg.newzbin_url())

And specifically this bit in cfg.py:

Code: Select all

newzbin_url = OptionStr('newzbin', 'url', 'www.newzbin2.es')

[nickb834]

so - tried:

Code: Select all

newzbin_url = OptionStr('newzbin', 'url', 'www.newzbin2.es:5150')

Which didn't work - I pycompiled cfg.py after I changed it but that didn't help (restarted sabnzd after compiled btw)

[sander]

Interesting.

But what if instead of

Code: Select all

ssh -f -N -q -L 5150:www.newzbin2.es:80 user@proxy.somedomain.com

you use

Code: Select all

sudo ssh -f -N -q -L 80:www.newzbin2.es:80 user@proxy.somedomain.com

Would the ssh-proxy-forward then listen on port 80? If so, would any plain HTTP / 80 request to your localhost then forwarded to http://www.newzbin2.es:80 ?

Before doing this, make sure nothing else is running port 80.

EDIT:
I tried this on my own VPS (Ubuntu 11.10) with port 443 because port 80 is already in use, and I have to type the password of root, not of my own account. Strange.
Anyway: it works.

Code: Select all

sander@toverdoos:~$ lynx --dump http://localhost:443/ | head -10
   #[1]Atom feed for this page [2]RSS feed for this page [3]CSV feed for
   this page [4]Opera Widget for this page

[5]Newzbin

     * [6]Home
     * [7]News
     * [8]Browse
     * [9]Search
     * [10]Account
sander@toverdoos:~$

[sander]

Ah:

Code: Select all

 sudo ssh -g -f -N -q -L 443:www.newzbin2.es:80 localhost

is also great: the "-g" (global?) allows remote connections: "Allows remote hosts to connect to local forwarded ports.".

So my VPS/URL:443 is now a transparant proxy to www.newzbin2.es ...

I have not tested whether I can login etc ...

[jcfp]

I would just use a vpn, and route all blocked address space (hulu/newbzin/whatever gets censored tomorrow) through there. Future proof, no proxy support needed, and program independent so no need for weird mods to individual programs in a attempt to fix a system level problem.

[nickb834]

I would just use a vpn, and route all blocked address space (hulu/newbzin/whatever gets censored tomorrow) through there. Future proof, no proxy support needed, and program independent so no need for weird mods to individual programs in a attempt to fix a system level problem.

Well I looked into it and I'd really rather not force all traffic this way (I could split horizon obv / different routes etc)

I do see your point tho - and I'd prob implement all of this were it not for the fact that the sabnzbd host is primarily a web server (so thats 80 and 443 in use), but also an asterisk, pxe, dnla, upnp, file server, print server etc - I have a fetish for consolidation (perhaps as I'm fed up supporting an unecissarily large number of servers at work :-) )

So in this case I think it simpler to deal with this edge case with Newzbin2.es, tho like I said your points are very valid -and thanks for them

[nickb834]

Ah:

Code: Select all

 sudo ssh -g -f -N -q -L 443:www.newzbin2.es:80 localhost

is also great: the "-g" (global?) allows remote connections: "Allows remote hosts to connect to local forwarded ports.".

So my VPS/URL:443 is now a transparant proxy to http://www.newzbin2.es ...

I have not tested whether I can login etc ...

Thanks for this, however the big catch is that I've already got services running on 443 and 80 so they're out of the question (can't be moved)

Also in our case I don't believe '-g' is needed - as remote hosts aren't connecting to the local forwarded ports on the server where the tunnel was initiated, if you wanted to allow some remote host access to the local ports on the client then yeah - but we're the other way round. Without the g I already have my pads client working over the tunnel.

I think if I change my apache hosts listening on 80 and 443 to glom onto a different IP address I can change my tunnel to use 80 and 443 and the problem is solved.

I'll get back to you with what I come up with - it's good to see two different minds came up with the same answer tho so I'm not a million miles off - thanks! :-)

[nickb834]

Ah:

Code: Select all

 sudo ssh -g -f -N -q -L 443:www.newzbin2.es:80 localhost

is also great: the "-g" (global?) allows remote connections: "Allows remote hosts to connect to local forwarded ports.".

So my VPS/URL:443 is now a transparant proxy to http://www.newzbin2.es ...

I have not tested whether I can login etc ...

So - I've moved things around on my server - Apache now gloms onto a diff IP address - leaving me with 127.x.x.x to play with - I've changed ssh thus:

Code: Select all

ssh -f -N -q -L 80:www.newzbin2.es:80 root@proxy.somedomain.com

links http://www.newzbin.com works - I can browse the site, login and see my details etc so that's all good. Sabnzbdplus however won't work - this in the logs:

Code: Select all

2012-02-15 17:19:51,404::WARNING::[newzbin:58] Problem accessing Newzbin server, wait 1 min.
2012-02-15 17:19:51,404::INFO::[newzbin:178] Traceback:
Traceback (most recent call last):
  File "/usr/share/sabnzbdplus/sabnzbd/newzbin.py", line 170, in _grabnzb
    conn.request('POST', fetchurl, postdata, headers)
  File "/usr/lib/python2.6/httplib.py", line 910, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.6/httplib.py", line 947, in _send_request
    self.endheaders()
  File "/usr/lib/python2.6/httplib.py", line 904, in endheaders
    self._send_output()
  File "/usr/lib/python2.6/httplib.py", line 776, in _send_output
    self.send(msg)
  File "/usr/lib/python2.6/httplib.py", line 735, in send
    self.connect()
  File "/usr/lib/python2.6/httplib.py", line 1108, in connect
    sock = socket.create_connection((self.host, self.port), self.timeout)
  File "/usr/lib/python2.6/socket.py", line 514, in create_connection
    raise error, msg
error: [Errno 111] Connection refused

Possibly due to how Python is wanting to open a socket but can't as there's already one in place?

[sander]

You say "links http://www.newzbin.com works ", but I hope you mean http://www.newzbin2.es/ ?

And if so, what happens with https://www.newzbin2.es/ ?

[nickb834]

Whoops - spotted my mistake - yes I meant to say newzbin2.es - and yes it browses just fine in links, I can login - see my details etc etc etc:

Code: Select all

root@nas:/media/nas/Incoming/TV# grep newzbin2.es /etc/hosts
127.0.0.1       www.newzbin2.es         newzbin2.es
root@nas:/media/nas/Incoming/TV# lynx --dump http://www.newzbin2.es/ | head -10
   #[1]Atom feed for this page [2]RSS feed for this page [3]CSV feed for
   this page [4]Opera Widget for this page

[5]Newzbin

     * [6]Home
     * [7]News
     * [8]Browse
     * [9]Search
     * [10]Account

I opened up another tunnel for port 443 and I get the same:

Code: Select all

root@nas:/media/nas/Incoming/TV# lynx --dump https://www.newzbin2.es | head -10
   #[1]Atom feed for this page [2]RSS feed for this page [3]CSV feed for
   this page [4]Opera Widget for this page

[5]Newzbin

     * [6]Home
     * [7]News
     * [8]Browse
     * [9]Search
     * [10]Account

The problem I got now is with Python and the way it opens sockets it would seem - it no likey ssh tunnels :-(

[nickb834]

ignore that - it seems as soon as I opened up the tunnel for 443 - it fired into life and started working :-)

I never knew that sabnzbd hit's newzbin over ssl, so in summary to get this working (for anyone coming along after)

You'll need access to a box you can ssh to, that is allowed to forward (sshd config) and that can itself resolve http://www.newzbin2.es, and can more importantly access ports 80 and 443 on whatever IP http://www.newzbin2.es resolves to. If you have all that then two simple steps and you're done:

Edit your hosts file and point http://www.newzbin2.es at localhost (127.0.0.1 for eg)
setup an ssh tunnel thus: ssh -f -N -q -L 80:www.newzbin2.es:80 -L 443:www.newzbin2.es:443 user@proxy.somedomain.com

You could also add this tunnel to ~/.ssh/config which once I've got the syntax right I'll post back here.

Cheers Sander - much appreciated!

[sander]

Cool that it works!

"443:www.newzbin2.es:80" ... so you reroute an incoming HTTPS connection into a plain outgoing HTTP, and that works ... ?

Another setup is this:
you local system that points (www.)newzbin2.es to the IP of your VPS with /etc/hosts
your VPS has the SSH-tunnel-proxy running with options as above, with "-g" added and just a "localhost" at the end.

This way you could share your proxy with other blocked victims. ;-)
And: by pointing your webbroser to the name / IP of your VPS, you can also browse newzbin2.es . And by changing the URL in SABnzbd, you can even skip the /etc/hosts stuff.

[nickb834]

Cool that it works!

"443:www.newzbin2.es:80" ... so you reroute an incoming HTTPS connection into a plain outgoing HTTP, and that works ... ?

Woops no - another typo of mine which I have now fixed with an edit!

Another setup is this:
you local system that points (www.)newzbin2.es to the IP of your VPS with /etc/hosts
your VPS has the SSH-tunnel-proxy running with options as above, with "-g" added and just a "localhost" at the end.

Hmmmm, I think I understand this - and could prob make it work as there's not much other than squid running....

This way you could share your proxy with other blocked victims. ;-)

I maybe could - after all it's just to browse newzbin, not like it's nntp traffic!

But then again - it's my sneaky proxy and it's locked down to my ip :-)

And: by pointing your webbrowser to the name / IP of your VPS, you can also browse newzbin2.es . And by changing the URL in SABnzbd, you can even skip the /etc/hosts stuff.

That's tempting but I like my squid proxy on the vps for that sort of thing - I have a proxy.pac for the browser that automatically sends traffic via the proxy for "blocked" sites and direct for other sites.

Once again - thanks for your help Sander :-)