New .NZB format spec: inline files + API access to downloads

[noice]

Hello,

I know what I'm asking is much and disruptive, but that's what an true innovative product offers, disruptions.

So my two ideas are:

1. Extend the NZB indexers and search engines with previews (cover and a streaming trailer) downloaded through SABnzbd

For some articles on the newsgroup, you often find a cover .jpg file and a short trailer or movie proof.

I want to be able to see this cover + trailer immediately WITHIN the search engine or indexer web page, I don't want to loose my time downloading the .nzb, monitor SABnzbd for completion, open the download dir, open the file, and then delete everything.

I can easily modify the search engine/indexer with GreaseMonkey or a firefox addon. What I can't do right now is retrieving from the SABnzbd API the files I instructed SABnzbd to download.

>> FEATURE REQUEST: Get from the SABnzbd API the completed files.


2. Include movie link + cover in the download folder, even when it is not available in the newsgroups !

Do you know the "What's that movie and why did I download it?" effect when you look at your SABnzbd download folder and you're unable to remember what the movie is about since you forgot the trailer ? I do.

So I want to be able to have, in the download folder, two additional files:
- A link to the movie description on the web (eg. IMDB...)
- The movie cover

I am able through e.g. Greasemonkey to transfer these information to SABnzbd, ideally within the .NZB file.
So now I would like SABnzbd to understand it.

The issue is that .NZB specs are pretty... simple. We need to enhance it to allow the .NZB files to store, within them, complete .jpg or .url files, that will be put in the SABnzbd download directory.

>> FEATURE REQUEST: Copies "inline files" (blobs) from the .NZB file to the download dir

This will require binary XML support and a new NZB specs. There is some binary XML features bundled in Python: http://stackoverflow.com/questions/2088 ... -in-python
I propose to use a new <blob> tag instead of <file> tag for .NZB inline files.


As you can understand, I'm more of a Greasemonkey coder than a Python coder, so I'm asking for your help to develop the sabnzbd-part features

[shypike]

Interesting idea.
Have you tried selling it to any indexing site?
Be aware that these sites may be wary of including possibly copyrighted material in their NZB files.

[noice]

Forget about my 2nd feature request, I'm actively working on it.

[noice]

Hello,

For those interested I've developed the 2nd feature, on Github: https://github.com/kriswebdev/sabnzbd/t ... inlinefile

I've had a long talk with inpheaux and he thinks it should be developer as a post-processing script (using DirectNZB headers or sort of), not as data within NZB or data transmitted through API calls so there's no probability of merge.

That's true the two features I talked about can be viewed as out of SABnzbd scope, still as I find this convenient for user-scripts I'll keep using it for my personal use.

[inpheaux]

Yeah, we discussed this at length on IRC friday.

As I see it, the two features as-described are pretty out-there.

The first request is to have SABnzbd download arbitrary files from arbitrary web URLs passed in via the API. To me that's clearly out of scope for SABnzbd. We download stuff from Usenet, and provide post-processing script functionality so if you want to build a script to handle downloading metadata, go for it.

The second request is to extend the NZB spec, so that's right out as well. If you really want to go down the route of modifying the NZB spec, get indexers to implement it first, then we'll consider using the new data. But even then, extending the NZB spec to include blob metadata is sort of crazy, since NZBs are huge enough as-is, without adding straight binary encoded files right in the NZB.

The general use-case of getting metadata for files downloaded is typically handled by a third party app like Sickbeard / CouchPotato / Headphones, etc, but your specific use case dictates that you want to be able to pass some kind of metadata through solely the API, so you can bundle it into a userscript. That, I think, can be at least partially covered by exposing access to nzo_info['more_info'] via the various add-nzb api calls. If you did that, you could then along with a URL to an NZB, add a link to IMDB, or TVDB, or wherever, and it would end up associated with the completed job in SABnzbd's history.

This option wouldn't require getting SABnzbd to download arbitrary files, or extending any specs, it'd just be exposing an existing variable as a new optional parameter to an existing API call, which I think would be perfectly reasonable.

[noice]

OK. By the way, there's one thing we didn't discussed concerning your proposal:
How do you get access back to nzo['more_info'] from the post-procesing script? Does the post-processing script have to make an API call to SABnzbd to read this variable when the job is done or is there another convenient way to access it?

User scripts only provide access to:

1 The final directory of the job (full path)
2 The original name of the NZB file
3 Clean version of the job name (no path info and ".nzb" removed)
4 Indexer's report number (if supported)
5 User-defined category
6 Group that the NZB was posted in e.g. alt.binaries.x
7 Status of post processing. 0 = OK, 1=failed verification, 2=failed unpack, 3=1+21

Another minor point is that more_info has to be stored in memory during all the NZB download process, but it's ok if the data is limited to some MB.
Is nzo['more_info'] backed up? So that it won't be lost if SABnzbd bis restarted.

[inpheaux]

OK. By the way, there's one thing we didn't discussed concerning your proposal:
How do you get access back to nzo['more_info'] from the post-procesing script? Does the post-processing script have to make an API call to SABnzbd to read this variable when the job is done or is there another convenient way to access it?

I'm not saying anything about a post processing script anymore. I'm saying add it to the addurl api call (and maybe the other two) as an extra parameter. So along with URL, Nicename, Category, etc, you also add "MoreInfo" and supply a URL. Not incredibly difficult from an API perspective, super easy for you to implement in your userscript, no need for post-processing anything . . . unless you're 100% dead-set on downloading crap after the fact.

[noice]

Thanks for your reply inpheaux.

Regarding the second feature, I've implemented two modes and I think what you propose is quite similar to my implementation.

The implementation (Github link in my post above) provides the following functionalities:

Feature 2 mode "extended NZB specs": (out of scope)
Extended .NZB file with <blob> data => API:addfile(extended .NZB, etc) => [SABnzbd] => Result: Data within <blob> is written in the NZB download folder

Feature 2 mode "form file API handling":
Form file => API:addfile(regular .NZB, form files, etc) => [SABnzbd] => Result: Data within form file is written in the NZB download folder

I think the implementation mode you propose is similar to this last mode, but limited to a URL:
URL link => API:addfile(regular .NZB, URL link as nzo_info['more_info'], etc) => [SABnzbd] => Result: Data (URL link) is written in the NZB download folder (as a "moreinfo.URL" file)

Do I understand well?

Should the file creation be managed in API:addfile or somewhere else?
Somewhere else would allow this functionality to be also compatible with DirectNZB... note that in this latter case we mey need a config switch to enable/disable the feature.

Can we extend your proposed implementation mode to filetypes different than .url or is it out of scope?

[inpheaux]

You're getting hung up on downloading files from arbitrary sources. Anything involving SABnzbd downloading stuff from outside of Usenet is going to almost certainly be considered out of scope. I'm trying to give you an alternative option that doesn't require vastly extending the functionality of SABnzbd and doesn't require extending the NZB spec, where you can pass in a moreinfo link via the API and have it show up in the SABnzbd UI.

No file downloading. Just adding an extra field to the API to display in the SABnzbd UI.

That at least solves your originally stated goal of "have some way to send extra metadata to SABnzbd so if I forget what something is that I downloaded, I'll have a way to easily find what it was". Just not in a way that involves downloading extra files.

Again, if you're dead-set on downloading extra stuff, go for it. Have fun adding that to a personal fork we probably won't integrate into mainline SABnzbd. Implement it however you want.

[noice]

Finally implemented in personnal fork with limited single-article jpg file support, for whoever's interrested.
https://github.com/kriswebdev/sabnzbd/c ... 4b42b7ac7c