precheck and connection count

[stouty]

Hi

Does anyone know if the precheck article STAT connections to your news server are included in the thread count? And the thread count must be less than the max connections set in the server config - I guess?

The reason I ask is that I've just recently switched on pre-checking and have since noticed these errors in my logs:

Code: Select all

2012-10-28 20:49:08,108::DEBUG::[downloader:504] Server login problem: 481, 481 (remote) gn;gnXXXXXX: max number of simultaneous IP addresses reached
2012-10-28 20:49:08,109::ERROR::[downloader:520] Probable account sharing (news.giganews.com:563)

I know I'm not sharing or running more than one instance.

I have one server set up, with 20 connections, no backup server, Disconnect on Empty Queue is on, SSL on port 563. I just recently switched on prechecking and saw these errors. So I thought I'd try to look at the code before asking, but it's all a bit new to me - Python, NZB file structure and NNTP!

Sooooo .... trying to get my head around the code. I see the precheck bool is set on an nzo object. I think an nzo is an object that encapsulates the NZB file plus other info/methods. I see a queue, nzo added to the queue... newswrapper that just gets the header not the full article if precheck is set on the nzo.A decoder that does a special check if it' a precheck - sees if the file exists on the server, or looks for clues as to why not if it's missing. Then some maths - foreach nzbfile obj, see how much is left, see if there are any pars, see what we need, ratios, need, short, pars, enough ... it's enough for me ... it's 5am here!

Well a bit more ... A singleton downloader that tracks the active threads(connections) ... ah, the downloader sets the newswrapper body up with the precheck bool. So it should count a precheck in the active threads right? I haven't the energy to check out how the busy and idle threads arrays are managed...

The giganews 481 error mentions "simultaneous IP" so the clues_too_many_ip() method returns true. however if the text was different and mentioned threads, you'd have the same error, 481, but the if/elseif code at this point raises the too_many_connections error first, then, if not that then the clues_too_many_ip() .... and the account sharing warning.

I guess other news providers give 481 errors with 'connections', and 'threads' in the error text? Or am I just confused max sim IPs is 1 and I'm barking up the wrong tree with threads/connections? Ah, no, I spoke to a giganews support guy about this error and he told me to increase my max connections from 20 to 40. So he thought the error was to do with threads/connections and not different IPs. Not sure how increasing my max connections helps with this issue though.

Jeez 6am and I'm reading out-dated NNTP RFCs http://tools.ietf.org/html/rfc4643#section-4

Anyway, any ideas why I'm getting the errors?

Thanks
James

BTW: I'm very impressed with the code. And I'm liking Python. What's a good editor for reading Python projects. I was mainly grepping the code just now. Would be nice to be able to jump to method/class defs etc.

[shypike]

Pre-check uses the same method as actual download.
The only difference is in sending a "stat" instead of a "body" request.
The GigaNews message signals that you are sharing your subscription with someone else.
Either that or you often get a new IP address assigned by your ISP.

[stouty]

Thanks for the reply. I'm pretty sure I'm not sharing, or someone has my login details without my knowledge. I'll change my password to be sure.

Is there somewhere in the code that records your external IP? I'd be interested in adding some code to watch my IP to see if it changes.

I'm behind a router however.

thanks

[shypike]

The detection of IP addresses is done by Giganews.
SABnzbd knows nothing about this.
In fact, SABnzbd on its own cannot even cause this issue,
because it doesn't actually do anything with your external IP.
Behind one router you can have multiple instances of SABnzbd (or other Usenet clients)
and at the most you should get a "too many connections" message from Giganews.
You may need to check with Giganews' customer service.

[stouty]

I added a quick and dirty IP checker to downloader.py, importing/borrowing the timing code from BPSMeter, to send a Growl notification if my IP changes:

Code: Select all

def check_ip(self):
    t = time.time()

    if t > self.end_of_day:
        # current day passed. get new end of day
        self.end_of_day = tomorrow(t) - 1.0

        if t > self.end_of_week:
            self.end_of_week = next_week(t) - 1.0

        if t > self.end_of_month:
            self.end_of_month = next_month(t) - 1.0

    check_time = t - (MIN * 5)

    if self.log_time < check_time:
        self.log_time = t
        url = 'http://automation.whatismyip.com/n09230945.asp'
        req = Request(url)

        try:
            response = urlopen(req) #urlopen(req, None, 10) for a 10s timeout

        except URLError, e:
            if hasattr(e, 'reason'):
                logging.debug('Failed to reach: %s', url)
                logging.debug('Reason: %s', e.reason)
            elif hasattr(e, 'code'):
                logging.debug('The server couldn\'t fulfill the request for %s', url)
                logging.debug('Error code: %s', e.code)
        else:
            new_ext_ip = response.read()

            if check_valid_ip(new_ext_ip):
                fqn = os.uname()[1]
                logging.info("Host %s, Checking in from IP#: %s ", fqn, new_ext_ip)

                if self.ext_ip == new_ext_ip:
                    logging.debug("IP address is UNCHANGED")

                else:
                    if self.ext_ip == "":
                        self.ext_ip = new_ext_ip
                    else:    
                        logging.info("IP address HAS CHANGED")
                        tmpStr = "IP address changed from %s to %s" % (self.ext_ip, new_ext_ip)
                        growler.send_notification("SABnzbd", tmpStr, 'other')
                        self.ext_ip = new_ext_ip



def check_valid_ip(ext_ip):
    valid = False
  
    if not ext_ip or ext_ip == "0.0.0.0" or string.count(ext_ip, ".") != 3:
        logging.info('Invalid IP (blank, or 0.0.0.0 or doesn\'t have 3 dots): [%s]', ext_ip)
    else:
        logging.debug('Checking octets for [%s]', ext_ip)

        octets = string.split(ext_ip, ".")
        ip = [0,0,0,0]
        try:
          for i in range(4):
            ip[i] = int(octets[i])
        except:
          ip = [0,0,0,0]

        # 0-255 in first three allowed, 1-255 in last also
        # code below allows 0 in last octect
        if ip[0] < 0 or ip[0] > 255 \
        or ip[1] < 0 or ip[1] > 255 \
        or ip[2] < 0 or ip[2] > 255 \
        or ip[3] < 0 or ip[3] > 255:
            logging.info('Invalid IP (octets out of range): %s', ext_ip)
        else:
            logging.debug('IP octects in range - we have a VALID IP')
            valid = True

        return valid  

If I then restart my router to get a new external IP, (after setting MIN=1 for a quicker check), I get a Growl notification: http://cl.ly/image/1Y022q3O2t35

Not sure how valid this approach is, but I'll leave it going for a few days, see what happens.