I've been testing a server configuration with the latest version 0.4.5
The primary server has spotty completion and I have configured 1 connection for it.
The fill server has excellent completion and allows unlimited connections, or at least I haven't found the connection limit.
On a 430 error the download task is passed to the fill server. When that task is completed, the connection remains open for thread #1. When there is another 430 error on the primary server, and thread #1 on the fill server is idle, instead of re-using the idle thread #1 (Feature Request) SABnzbd opens thread #2.
I configured 64 connections on the fill server, and there were occasional 430's on the primary server. SABnzbd gradually opened all 64 connections on the fill server and used them in round-robin order. Only after thread #64 had been used, did thread #1 finally get re-used.
So 64 connections are kept open on the fill server, when only a few are needed at any one time. If I estimate that I won't need more than 3 simultaneous connectons on the fill server, configure it for 3 and guess wrong because it needs more, then fill tasks can stack up waiting for unused connections.
I can't think of any advantage to open a new connection instead of re-using an idle connection. If that isn't a bug, then it's close to being a bug. SABnzbd can be smarter, and not open any more simultaneous connections to the fill server than needed.
Reuse of idle threads on fill server
Reuse of idle threads on fill server
Last edited by pobox on December 2nd, 2008, 2:29 pm, edited 1 time in total.
Re: Reuse of idle threads on fill server
That's the way it's designed. It would be a lot of work to change it.
Also many servers don't respond very well to closing connections.
(Some time later they claim you have too many connections).
Why do you set 63 connections for the backup server anyway?
Also many servers don't respond very well to closing connections.
(Some time later they claim you have too many connections).
Why do you set 63 connections for the backup server anyway?
Last edited by shypike on December 3rd, 2008, 2:36 am, edited 1 time in total.
Re: Reuse of idle threads on fill server
I expected that a change would require a lot of re-coding but thought I'd ask anyway.
I should have explained that the client wouldn't need to close unused connections, because they'll timeout by the fill server; usually after 300 seconds of inactivity.
After the primary server passes a task to the fill server, the primary thread doesn't wait for the fill task to complete, it immediately tries the next article in the queue. I do actually prefer it that way; I have some programs where the primary server thread always waits for the task passed to the fill server to complete, but if you changed SABnzbd to wait like that too, I really wouldn't mind.
However, if the primary doesn't wait, then theoretically the fill server can fall way behind over time. Assembly of a complete file might be delayed long after the first article in the file is downloaded. I don't know how SABnzbd would handle such a backlog.
To make sure there would always be enough fill threads to handle a potential flood of 430's received from the primary server, I configured 64 connections as a test to see if SABnzbd would gradually open all of them. It did.
If I configured 3 connections on the fill server, it would just be a guess. There might not be a need for more than 3 simultaneous fill tasks, or there might be a sporadic demand for a lot more and then a backlog starts. I can't predict how many it would need to avoid a backlog building up.
If SABnzbd's priority was to re-use an idle fill thread instead of opening a new connection, I wouldn't make a wild guess like that; I could just configure some maximum number I thought would be safe, and SABnzbd would probably never have to open simultaneous connections approaching that configured number during the whole download.
I should have explained that the client wouldn't need to close unused connections, because they'll timeout by the fill server; usually after 300 seconds of inactivity.
After the primary server passes a task to the fill server, the primary thread doesn't wait for the fill task to complete, it immediately tries the next article in the queue. I do actually prefer it that way; I have some programs where the primary server thread always waits for the task passed to the fill server to complete, but if you changed SABnzbd to wait like that too, I really wouldn't mind.
However, if the primary doesn't wait, then theoretically the fill server can fall way behind over time. Assembly of a complete file might be delayed long after the first article in the file is downloaded. I don't know how SABnzbd would handle such a backlog.
To make sure there would always be enough fill threads to handle a potential flood of 430's received from the primary server, I configured 64 connections as a test to see if SABnzbd would gradually open all of them. It did.
If I configured 3 connections on the fill server, it would just be a guess. There might not be a need for more than 3 simultaneous fill tasks, or there might be a sporadic demand for a lot more and then a backlog starts. I can't predict how many it would need to avoid a backlog building up.
If SABnzbd's priority was to re-use an idle fill thread instead of opening a new connection, I wouldn't make a wild guess like that; I could just configure some maximum number I thought would be safe, and SABnzbd would probably never have to open simultaneous connections approaching that configured number during the whole download.
Re: Reuse of idle threads on fill server
Would it be a lot of work to change it so a primary thread waits for the fill thread to complete?
Re: Reuse of idle threads on fill server
To be quite honest, the connection handler part of SABnzbd is the part we least enjoy maintaining.
Too much black magic going on (to get some decent performance out of the not-so-strong Python socket library).
If you're interested...
Too much black magic going on (to get some decent performance out of the not-so-strong Python socket library).
If you're interested...