Mystery performance bottleneck: speed 30 -> 15 MB/s, SSD, HDD

[certain-grip]

I'm trying to figure out why Sabnzdb downloads are slower than expected on my HTPC: 15MB/s sustained, whereas a desktop computer on the same network can get 40MB/s using the same server and configuration. I know that many people would kill for 15MB/s, and obviously I'm not complaining about it in that sense. But if it _can_ go faster, I sure wish it would.

Both have an SSD and gigabit ethernet/internet. The desktop is certainly beefier in terms of CPU and RAM, but as near as I can tell there is no bottleneck on the HTPC. top reveals moderate CPU usage by python (30%), low usage by unrar (7%), plenty of free memory (>1G), and iotop reveals hardly any disk usage (and anyway even a slow hdd wouldn't cap out at 15MB/s).

What else can I do to diagnose the bottleneck on the HTPC?

My best theory at this point is that the usenet servers have a somewhat high ping (75ms), and Sabnzbd is doing some sort of optimization on the HTPC that requires more connections, so the ping is more of a factor than it would be otherwise. I dunno, kind of a weak theory I think

[sander]

What are the hardware specs of the HTPC?

On the HTPC, in SABnzbd's upper right corner, click on the wrench symbol ("Status and interface options"), then click on 'refresh' symbol at the lower specs. Can you post those lower spec (pystone etc) here?

And:
On the HTPC, in SABnzbd's upper right corner, click on the wrench symbol ("Status and interface options"), then click on first tab Status, and there click on "Test download" 100MB or 1000MB.

What is the result?

[certain-grip]

Thanks for replying! It's an Intel NUC D54250WYK: Core i5-4250U, 4G RAM, SSD temp dir + HDD completed storage.

Specs:

Code: Select all

System performance (Pystone): 73879
Download folder speed: 106 MB/
Complete folder speed: 36.8 MB/s

Test downloads look exactly the same as ordinary downloads, they start fast (30MB/s), and over the course of about 10 seconds drop to a sustained 15MB/s.

The RAM is obviously low by any modern standard, but if I'm reading this right it seems like there's still plenty available. SABnzbd is pretty much the only thing running on the box.

Code: Select all

$ free -m
total        used        free      shared  buff/cache   available
Mem:           3837        1698         112          19        2026        1840
Swap:           979          48         931

[sander]

i5 with 4GB is a very nice download box.

However:
Your Pystone is low. My i3's score is higher. That is strange. Redo that measurement a few times, and check the highest.
Your Complete / HDD is quite slow. What if you put Comple on your SSD and do a test download.

I'm guessing you're using Linux. If so, and you know how to use git, let me know ...

[certain-grip]

Hah, yeah re-running Pystone gave a much better result, no idea why: 194685. I re-ran it a few times to be sure, similar results.

I agree HDD is slow, not sure why, I'll look into that separately. But I don't think it's the issue here. I changed the completed downloads folder to the SSD and nothing changed performance-wise.

I'm quite comfortable with Git Thanks again for the responsiveness.

[sander]

https://github.com/sabnzbd/sabnzbd/tree ... cache-skip is a branch (two days old) that solves "start at high speed, then go lower": it gives a more steady download speed. So you could give it a try.

Oh, and it's based on SAB 3.0.0 alpha2. So disclaimer for SAB-3 are in place.

Having said that:
I've used SAB3 for over 6 months now
I'm using that cache-skip on my download box since this weekend.

[certain-grip]

Thanks lot, I'm pretty sure that solved it. I say pretty sure because at first it behaved a little inconsistently. After installing from source and switching to that branch, the first job started very fast and then slowed to 15MB/s. All subsequent jobs topped out at 15MB/s. But after restarting the server again, jobs are now capped at about 40MB/s, which is to be expected given the completed directory HDD speed.

Thanks again for the help, I'll monitor the performance of real nzbs over the next few days.

[certain-grip]

Ugh... there is definitely something weird going on here. That feature branch is better, like I said the first download goes super fast until the very end where it trails off, but every subsequent download is as slow as ever. Although they don't show the spike-and-tail-off behavior of before, they are just consistently slow.

It's just weird because after restarting the daemon a few times, at least one time performance was rock solid over a dozen tests.

[safihre]

Your HDD or Python just cannot keep up with the writing. So the cache-to-write fills up.
We have improved it in this new branch, but we can't write faster than the core-system allows us.
Could you check in the Status and Interface-settings what happens if you perform the disk-speed test? How much do you get?
Will see if there is also a bottleneck in Python-code that I can maybe remove!

[sander]

@certain-grip: for high speed, use your SSD. Certainly if you have DirectUnpack on

[certain-grip]

I don't think it's a disk bottleneck. Even with all folders (temporary/completed) set to the SSD, I get the same behavior. And I know that it _can_ write faster, it just doesn't, because the first job after restarting the server downloads at full speed (>40MB/s), and all subsequent jobs top out at 15MB/s.

The performance test shows that my SSD writes >115MB/s.

[certain-grip]

It sure seems suspicious that the first download on a fresh daemon start works great, and all subsequent downloads slow down. Could this be a symptom of a memory leak or some volatile configuration issue?

[safihre]

In your task manager, does the memory usage increase and never goes down?
What is shown in the Status window of Sabnzbd? It also shows the number of cached articles of the cache limit.

[certain-grip]

I guess a memory leak is unlikely since memory usage returns to normal after downloads complete. The cache usage never goes above ~20 MB.

Still, it seems like there must be some problem related to memory / caching / internal configuration if the first download after a restart works fine, and all subsequent downloads perform poorly.

[sander]

@certain-grip

I did some tests with SAB-2 and SAB-3-cache-skip on my Celeron with SSD-only on my 750/750 Mbps fiber, on Ubuntu 18.04: steady 80 MB/s with 100MB, 1GB and 10GB downloads, also on next downloads.

Proof: "Downloaded in 32 seconds at an average of 86.0 MB/s"

Things to check:

all on SSD, and their speed. SABnzbd reports my SSD Incomplete and Complete above 150 MB/s
the sabnzbd.log, especially Tracebacks (very bad), and "delaying" (bad)
I've turned on "Pause Downloading During Post-Processing" to get highest download speed
I've turned off DirectUnpack for the same reason
I've turned off Unwanted Extension, because that triggers a bug in SAB-3-cache-skip
I've set the download to "Download" (no Repair, no Unpack), also to get highest download speed