HOWTO: Increase performance when using SSL

[quixote]

Hi All,

I just started playing around with this great package but discovered I was only getting 4.4MB/s over my 100Mbps broadband connection!
This was after tweaking things like article caching - which helped, but not enough!

Looking at top, I noticed that SABnzbd was at 100% on one of my cores (sometimes exceeding but rarely), so I figured the major bottleneck is CPU, but I have a spare core
(on my old Core2 Duo). So how do I workaround the Python GIL - which effectively forces the single core nature of the interpreter...

As I'm using SSL, I figured that even though I have pyOpenSSL install (so it should be using compiled code for SSL rather than interpreted), maybe offloading the SSL handling to a separate process would help...enter stunnel. This little utility enables you to configure a local port to be forwarded over SSL to a remote host/port. Thus a quick reconfigure of SABnzbd to use a local ntp server without SSL, but stunnel to relay that connection to my news server, and additionally provide the SSL.

Now I'm seeing about 70-80% of one core used by stunnel and 70-80% of the other core is used by SABnzbd, but more importantly, I'm seeing d/l speeds of 12.5MB/s!

Hope this helps anyone else using limited or old h/w.

[shypike]

So how do I workaround the Python GIL - which effectively forces the single core nature of the interpreter...

You cannot.
Indeed, your stunnel approach is a good solution.
Unfortunately not one we can integrate into SABnzbd, given all OS dependencies.
For more powerful platforms it's not really needed.

[quixote]

Hmm, maybe I should knock together a Python implementation of stunnel using pyOpenSSL - would still provide the core offload capabilities

Fair point on stunnel not being one to integrate in this package (I wasn't expecting it to be) - I just wanted to suggest a workaround for people in a similar position to me.

Anyway, great work on this - I am really impressed by the ease of use and flexibility - plus i like Python
Now we just need some enhancements to pypy as that would give me all the performance i need

[shypike]

I've looked at alternative Python implementations.
Problem number one: they all break important libraries that SABnzbd depends on.
A multi-processing approach may work, but requires very substantial redesigns.