Download queue

[Handyman1984]

Hello,

First of all, I've been running sab for about 2 years now and I still love it.
I've been running it on Debian (wheezy for the last 4 months). Last month I've installed (0.6.15) through the apt repository since my version was slightly outdated.
So far for the background information.

My 'problem' has everything to do with me being impatient.
Lately I've been searching for some older stuff ( > 900 days). I've got 2 excellent payservers, however when things get that old it tends to slow down.

I know sab has the option to "Only Get Articles for Top of Queue" which I have "Disable to prevent slow jobs from blocking the queue.".
Unfortunately sab and I have a different understanding of the word 'slow'.
Right now I have a download running at about 1MB/s, sab thinks this is fast enough and does not consider it to be blocking.
Changing the 38 threads over 2 servers to 15 threads on 1 server gives me the same speed.
On less obscure articles I can fill my line doing 6.9MB/s.

I would rather see stuff in the queue thats that slow flagged with low priority automatically so that my queue will be smaller sooner.
I do understand that it would still take the same time to complete the whole queue, but to me it's a big difference if 90% of the downloads would be done at 75% of the total time instead of 10% @ 75%.

So now I wonder, is there some setting that allows me to tell sab what a blocking speed should actually be?

[sander]

I have no experience with old posts being slow, and I can't 100% follow what you're saying, but how about this:

How about running two instances of SABnzbd: a "fast lane" SABnzbd on port 8080, and a "slow lane" SABnzbd on port 8090. Each with 15 threads (or the fast one a bit more than the slow one). Put the old posts in the "slow lane" SABnzbd, and they will not block / slow down the download of recent posts.

[Handyman1984]

Thanks for the fast response.

That option would work perfectly for me. But then again I could just manually toggle the priority as I did now.
I have sab set up to scan an nzb dir that is shared over the network, allowing quite a lot of people to dump nzb's in there which automagically returns their download in another dir. For obvious reasons I don't want everyone to have access to the job-control, even if they would understand what to do.

[sander]

I've never used the prio stuff. Can you automatically set it ... ?

Another possible solution: Automate your manual priority toggling. Via the API, read the age of the items in the queue, and if old, give it a low prio.

Get queue including age via http://wiki.sabnzbd.org/api#toc7
Set prio via http://wiki.sabnzbd.org/api#toc36

Does this help?

[Handyman1984]

I'm half way down that road now.
I'll post my dirty code when I'm done. It will be perl though, I'm not a python hero.

[Handyman1984]

This works for me:

http://pastebin.com/Xu3BjxSj

I might have overlooked some details though...

[sander]

On my Ubuntu:

"Can't locate Proc/Daemon.pm in @INC" ... solved with "sudo apt-get install libproc-daemon-perl"
"Can't locate HTTP/Tiny.pm in @INC" ... solved with"sudo apt-get install libhttp-tiny-perl"
"Can't locate JSON.pm in @INC" ... solved with " sudo apt-get install libjson-perl"

... and now it's running, however without any logging?

[Handyman1984]

Indeed it doesn't output anything. The example just works as long as you give the right settings.
And that's exactly why I called it dirty code.

I've made it publicly available so that someone with more experience with python might make something nice out of it if there are people out there who are as impatient as I am.

edit:
It just occurred to me that you might actually want to use it.
If you have any specific questions or anything like that I'd be happy to assist.

[sander]

I have sab set up to scan an nzb dir that is shared over the network, allowing quite a lot of people to dump nzb's in there which automagically returns their download in another dir. For obvious reasons I don't want everyone to have access to the job-control, even if they would understand what to do.

Back to the high-level: have you created a kind of multi-user SABnzbd? How do you know to move which download to which user directory? Do you attach some tag / earmark to .NZBs you find in the scan directory, for example based on the user name that put it there?
And you use a directory structure? No web interface?

EDIT:

My guess:
Users can dump their .NZB into a shared directory.
A script periodically checks for new files. If there is one, it also notes the user name it put there. It then adds the .NZB to the SABnzbd, combined with the user name as category name.
Via the standard Category settings, the final download lands in the specific user directory.

Something like that?

[Handyman1984]

Not that complicated at all, I just toss around with files and filenames mostly.
You're right about the shared directory (ies), I've got folders emulating the categories, (which should be automated in time).
The parent folder is on a CIFS (samba) share, writable to anyone with a valid IP.
A small script scans the dirs for nzbs every minute, copies them to the watched folder, with the category prepended to the name.
Sabs kickass preprocess option helps me assign the right category by splitting the filename again.
The download ends up in the folder where it belongs category-wise and is again shared over CIFS.

The webUI is good for me, but not for occasional users. Especially not if you have more than 1 user on it.
What I noticed was that people were putting their downloads on high priority or pausing other downloads.
Also the occasional 'shutdown' was triggered. So I decided that my friends and family may profit from my line/disks/info/coverart and other tools but they're not allowed to touch couchpotato, sickbeard (remove show anyone?) or sab.

[sander]

Cool. And do you use this only across a LAN, or also across Internet?

Is this the directory structue:

\share\user1\input-for-NZB\
\share\user1\resulting-downloads\


\share\user2\input-for-NZB\
\share\user2\resulting-downloads\

[Handyman1984]

I have:
/var/network/share < which is rsynced periodically over my servers, most folders here are writable.
/var/network/share/nzb < which has a symlink in dropbox folder
/var/network/share/nzb/movies && /var/network/share/nzb/music etc etc to act as categories.

The output generally goes to:
/var/network/media/[category] < which is a CIFS share, read only.
I don't control access per user, but per mac/ip. It's easier to say pc #4 can access the porn folder and pc #5 can't for example.
Setting up your own DHCP server is way easier than you might think.

Also: /var/network/share actually is an iscsi target ( but I'm not using it as one, yet).
On top of that I have a VPN, so all of this is also accessible from the net, eg stream music.