How easy: More than one active download at a time.

[Chatsubo]

Hi,

I'm not posting this as a bug or feature request, etc, as I'm well aware sabnzbd doesn't support this, and I'm not implying I want this implemented. My question/requirement is about an edge case. I merely want to ask some questions and think out loud. I am a senior software dev with python experience but haven't taken a look at the sabnzbd+ code quite yet, so I want to ask some naive questions to give me an indication of feasibility.

My ISP allows 4 concurrent connections, I am is also not in a "well connected" country, so the speed difference between locally cached and non-cached (by the ISP's news server) is huge. Locally cached files max out my connection, ones that are not cached (older content) come down about 1/4 of the speed. This means that I could max my pipe with 1 thread downloading on a cached file, but 4 running on a non-cached one goes very slowly, increasing the total time for my entire queue.

So in my situation I could want to automatically have more than one download to be active at a time, with say, a limit set on connections per active download. If I set that to 1, I would have 4 downloads running concurrently with 1 connection each. This would allow the "slow" ones to come down at their own pace whilst still allowing, to a degree, the "fast" ones to clear out in the meantime, with TCP hopefully enforcing fairness. I could (and do) just change the order of the queue around manually to put newer stuff first, but it takes some guessing and is onerous: I use sabnzbd+ exactly because it's hands-off.

So, the question becomes: Is the code structured in a way that the above could be done without ripping up large sections of the codebase? I guess it's really down to how deep some assumptions run. Is it all nice and segregated and I could achieve this without extremely deep knowledge of the code/design or rewriting half the application?

If it is not that bad a thing, I could do it myself on a private branch, or merge it at some point, who knows... or, of course, just request it as a feature if I think I'd be messing with your chi too much.

[jcfp]

Locally cached files max out my connection, ones that are not cached (older content) come down about 1/4 of the speed.

Sab does concurrent downloading in case you have multiple servers but the job at the top of the queue is beyond the age limit of at least one defined server, as set in sab's server configuration. The age limited server will then start working on the next job (if any) that lies within its age limits.

If the only difference between locally cached or not is age, you could configure your isp server twice in sab, with one of those bound by an age limit and allowed just a single connection. That way, a newer (and thus cached) job at the top of the queue will use "both" servers, while older jobs in that position will only be assigned to the server without age limit (with the remaining three connections). That leaves the age limited instance free to download any locally cached data further down in the queue.

Only con is that speed may suffer when you have a queue that consists exclusively of old stuff (only the unlimited server will be active, with its three connections vs. the permitted total of four), although that still depends mostly on how your isp limits (total "long distance" bandwidth vs. bandwith per thread times the number of threads). You may also have to double the memory cache, if you use that.

[shypike]

So, the question becomes: Is the code structured in a way that the above could be done without ripping up large sections of the codebase? I guess it's really down to how deep some assumptions run. Is it all nice and segregated and I could achieve this without extremely deep knowledge of the code/design or rewriting half the application?

By all means checkout the code.
You'll find that the core downloader consists of rather weird and painful code (downloader.py + newswrapper.py).
Mostly to get any usable speed out of Python, that doesn't exactly have a brilliant socket implementation.
I inherited this when I took over many years ago and the code was both improved and made worse
by me and sw1tch (former team member).
I certainly would appreciate it if an experienced Pythonista had a look at it.

Regarding your question.
I think it would require quite a bit of redesign to implement your wish.

One of my goals is to implement proper server priority instead of the crude primary/backup system.
I've been putting it off because it will require long and concentrated bouts of time, which I currently do not have available.

[Chatsubo]

If the only difference between locally cached or not is age, you could configure your isp server twice in sab, with one of those bound by an age limit and allowed just a single connection. That way, a newer (and thus cached) job at the top of the queue will use "both" servers, while older jobs in that position will only be assigned to the server without age limit (with the remaining three connections). That leaves the age limited instance free to download any locally cached data further down in the queue.

Not a bad idea, I'll definitely give it a whirl.

I certainly would appreciate it if an experienced Pythonista had a look at it.

Hmm, misunderstanding I think, I'm a senior developer who knows Python, not a Senior Python developer. My forte is actually C++, in the words of Jeremy Clarkson: "How hard can it be?"

I've been putting it off because it will require long and concentrated bouts of time, which I currently do not have available.

As you can probably tell from the delay: I feel your pain. Thanks for the info though.