Slow download speeds, can't find the cause [VM, docker]

[flonkyg]

Hey, 

I have VM, which handles most of my Docker containers, and SABnzbd is one of the containers. I've noticed some slow download speeds, and I can't figure out what's wrong. The speed bounces around 3–9 MB/s. I have a 1000/100 connection at home. I'm not sure if this Docker issue or SABnzbd, but I'll try my luck.

The slowness affects all Linux ISO downloads from NZBgeek and even the build it test files in SABnzbd.

The VM has 16 vCPU's (E5-2699 v3) and 15GB of RAM. Neither are even close to being fully utilized.

I did some testing on the VM that is hosting the Docker containers:

dd write

Code: Select all

manager@docker-host:/media/windowsshare/usenet/complete$ dd if=/dev/zero of=/media/windowsshare/usenet/complete/testfile bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 9.99246 s, 107 MB/

dd read

Code: Select all

manager@docker-host:/media/windowsshare/usenet/complete$ dd if=/media/windowsshare/usenet/complete/testfile of=/dev/null bs=1M
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 9.20755 s, 117 MB/s

fio write

Code: Select all

fio --name=write_test --ioengine=sync --rw=write --bs=1M --size=5G --numjobs=1 --time_based --runtime=60 --group_reporting --directory=/media/windowsshare/usenet/complete/
write_test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1
fio-3.28
Starting 1 process
write_test: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [f(1)][100.0%][eta 00m:00s]
write_test: (groupid=0, jobs=1): err= 0: pid=653740: Sun Sep 22 19:40:12 2024
  write: IOPS=140, BW=141MiB/s (148MB/s)(8450MiB/60007msec); 0 zone resets
    clat (usec): min=256, max=20145, avg=5091.54, stdev=6718.67
     lat (usec): min=272, max=20172, avg=5109.34, stdev=6718.59
    clat percentiles (usec):
     |  1.00th=[  285],  5.00th=[  302], 10.00th=[  314], 20.00th=[  334],
     | 30.00th=[  351], 40.00th=[  375], 50.00th=[  408], 60.00th=[  652],
     | 70.00th=[11338], 80.00th=[15401], 90.00th=[15795], 95.00th=[15795],
     | 99.00th=[16188], 99.50th=[16188], 99.90th=[19792], 99.95th=[19792],
     | 99.99th=[20055]
   bw (  KiB/s): min= 8192, max=2682880, per=100.00%, avg=197690.85, stdev=387059.76, samples=87
   iops        : min=    8, max= 2620, avg=193.06, stdev=377.99, samples=87
  lat (usec)   : 500=55.94%, 750=9.93%, 1000=0.65%
  lat (msec)   : 2=0.05%, 10=0.20%, 20=33.22%, 50=0.01%
  cpu          : usr=0.25%, sys=8.30%, ctx=4651, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,8450,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=141MiB/s (148MB/s), 141MiB/s-141MiB/s (148MB/s-148MB/s), io=8450MiB (8860MB), run=60007-60007msec

fio read

Code: Select all

manager@docker-host:/media/windowsshare/usenet/complete$ fio --name=read_test --ioengine=sync --rw=read --bs=1M --size=5G --numjobs=1 --time_based --runtime=60 --group_reporting --directory=/media/windowsshare/usenet/complete/
read_test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1
fio-3.28
Starting 1 process
read_test: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=112MiB/s][r=112 IOPS][eta 00m:00s]
read_test: (groupid=0, jobs=1): err= 0: pid=654704: Sun Sep 22 19:44:10 2024
  read: IOPS=110, BW=111MiB/s (116MB/s)(6635MiB/60022msec)
    clat (usec): min=84, max=71584, avg=8984.45, stdev=15372.52
     lat (usec): min=84, max=71584, avg=8984.60, stdev=15372.48
    clat percentiles (usec):
     |  1.00th=[   97],  5.00th=[  111], 10.00th=[  115], 20.00th=[  120],
     | 30.00th=[  125], 40.00th=[  131], 50.00th=[  137], 60.00th=[  147],
     | 70.00th=[  165], 80.00th=[35390], 90.00th=[35390], 95.00th=[35914],
     | 99.00th=[35914], 99.50th=[35914], 99.90th=[35914], 99.95th=[70779],
     | 99.99th=[71828]
   bw (  KiB/s): min=28672, max=122880, per=100.00%, avg=113276.77, stdev=8173.53, samples=119
   iops        : min=   28, max=  120, avg=110.62, stdev= 7.98, samples=119
  lat (usec)   : 100=1.67%, 250=73.31%, 500=0.02%, 1000=0.02%
  lat (msec)   : 2=0.02%, 20=0.02%, 50=24.90%, 100=0.06%
  cpu          : usr=0.02%, sys=5.00%, ctx=1994, majf=0, minf=269
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6635,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=111MiB/s (116MB/s), 111MiB/s-111MiB/s (116MB/s-116MB/s), io=6635MiB (6957MB), run=60022-60022msec

Network speedtests

Code: Select all

manager@docker-host:/media/windowsshare/usenet/complete$ speedtest -s Secret

   Speedtest by Ookla

      Server: Secret
         ISP: Secret
Idle Latency:    12.44 ms   (jitter: 0.76ms, low: 12.04ms, high: 13.25ms)
    Download:   840.71 Mbps (data used: 591.4 MB)
                 12.78 ms   (jitter: 3.62ms, low: 6.43ms, high: 41.77ms)
      Upload:    87.96 Mbps (data used: 153.7 MB)
                 30.07 ms   (jitter: 6.39ms, low: 11.76ms, high: 176.56ms)
 Packet Loss:     0.0%
  Result URL: 
manager@docker-host:/media/windowsshare/usenet/complete$ speedtest -s Secret

   Speedtest by Ookla

      Server: Secret
         ISP: Secret
Idle Latency:    13.70 ms   (jitter: 0.67ms, low: 12.70ms, high: 13.97ms)
    Download:   823.46 Mbps (data used: 1.4 GB)                                                 
                 12.52 ms   (jitter: 3.44ms, low: 6.70ms, high: 36.41ms)
      Upload:    96.20 Mbps (data used: 103.8 MB)
                 22.20 ms   (jitter: 12.58ms, low: 11.31ms, high: 232.13ms)
 Packet Loss:     0.0%
  Result URL: 

SABnzbd built-in test:

Code: Select all

Used cache  
0 B (0 articles)
System load  
0.01 | 0.05 | 0.09 | V=151M R=93M
System performance (Pystone)  
346384 Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz AVX2 Docker
Download folder speed  
7.3 MB/s /media/usenet/incomplete
Complete folder speed  
5.7 MB/s /media/usenet/complete
Internet Bandwidth  
108.16 MB/s 865.28 Mbps 

This is my docker-compose configuration:

Code: Select all

---
version: "2.1"
services:
  radarr:
    image: lscr io/linuxserver/radarr:latest
    container_name: radarr
    environment:
      - PUID=0
      - PGID=0
      - TZ=Moon
    volumes:
      - ./radarr/data:/config
      - /media/windowsshare:/media
    ports:
      - 7878:7878
    restart: unless-stopped

  sonarr:
    image: lscr io/linuxserver/sonarr:latest
    container_name: sonarr
    environment:
      - PUID=0
      - PGID=0
      - TZ=Moon
    volumes:
      - ./sonarr/data:/config
      - /media/windowsshare:/media
    ports:
      - 8989:8989
    restart: unless-stopped

  sabnzbd:
    image: lscr io/linuxserver/sabnzbd:latest
    container_name: sabnzbd
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Moon
    volumes:
      - ./sabnzbd/data:/config
      - /media/windowsshare:/media
    ports:
      - 8080:8080
    restart: unless-stopped
  jellyfin:
    image: lscr io/linuxserver/jellyfin:latest
    container_name: jellyfin
    environment:
      - PUID=0
      - PGID=0
      - TZ=Moon
      - JELLYFIN_PublishedServerUrl=192.168.50.115 #optional
    volumes:
      - ./jellyfin/config:/config
      - /media/windowsshare:/media
    ports:
      - 8096:8096
      - 8920:8920 #optional
      - 7359:7359/udp #optional
      - 1900:1900/udp #optional
    restart: unless-stopped
  bazarr:
    image: lscr io/linuxserver/bazarr:latest
    container_name: bazarr
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Moon
    volumes:
      - ./bazarr/config:/config
      - /media/windowsshare:/media
    ports:
      - 6767:6767
    restart: unless-stopped

Any ideas what could be wrong?

[sander]

Download folder speed
7.3 MB/s /media/usenet/incomplete
Complete folder speed
5.7 MB/s /media/usenet/complete

... so there you have it: the experienced disk speed is low, and the bottleneck. And SABnzbd will tell you "Download speed limited by" under the Wrench

You could do two things:
Run SABnzbd on the VM, and not inside docker
or ... inside docker, let SABnzbd download to a non-mapped directory, so inside the docker. Only useful for testing.

Internet Bandwidth
108.16 MB/s 865.28 Mbps

... and that's good.


> did some testing on the VM that is hosting the Docker containers:

... you must go inside the docker, and test the disk speed

[flonkyg]

My bad for not realizing that I needed to do the tests inside the Docker container... Now I did test the speeds inside the container, and I'm still confused.

Code: Select all

manager@docker-host:~$ sudo docker ps
[sudo] password for manager:
CONTAINER ID   IMAGE                                                        COMMAND                  CREATED        STATUS                PORTS                                                                                                                                                                        NAMES
7f387af2c7c3   lscr io/linuxserver/sabnzbd:latest                           "/init"                  2 days ago     Up 2 days             0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                                                                                                                    sabnzbd

Tests inside the container:

Code: Select all

root@7f387af2c7c3:/# dd if=/dev/zero of=/media/testfile bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.7718 s, 99.7 MB/s
root@7f387af2c7c3:/# dd if=/dev/zero of=/media/testfile bs=1M count=5120 conv=fdatasync
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 47.1676 s, 114 MB/s
root@7f387af2c7c3:/# dd if=/media/testfile of=/dev/null bs=1M
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 45.9734 s, 117 MB/s
root@7f387af2c7c3:/# dd if=/dev/zero of=/media/testfile bs=1M count=20480 conv=fdatasync
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 183.263 s, 117 MB/s

The "~/media/" is the same share that is mounted on the host and then passed to the Docker container

Code: Select all

  sabnzbd:
    image: lscr io/linuxserver/sabnzbd:latest
    container_name: sabnzbd
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Moon
    volumes:
      - ./sabnzbd/data:/config
     - /media/windowsshare:/media
    ports:
      - 8080:8080
    restart: unless-stopped

[sander]

And SABnzbd will tell you "Download speed limited by" under the Wrench

... do you see that?

[flonkyg]

I'm sorry I don't see "Download speed limited by" anywhere. According to Google, it should be in the same place as the benchmarks, but I can't find it. Also forgot to mention I'm running "4.3.3 [f24b3ce]" version of SAB.

This is all I can see in the "Status" tab:

Code: Select all

Local IPv4 address  
172.25.0.3
Public IPv4 address  

IPv6 address  
None
Nameserver / DNS Lookup  
OK
Used cache  
38.8 MB (53 articles)
System load  
1.16 | 0.86 | 0.38 | V=164M R=107M
System performance (Pystone)  
320616 Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz AVX2 Docker
Download folder speed  
7.3 MB/s  /media/usenet/incomplete
Complete folder speed  
3.5 MB/s /media/usenet/complete
Internet Bandwidth  
100.39 MB/s 803.12 Mbps
Test download  

[sander]

Weird.

Did you do this: "inside docker, let SABnzbd download to a non-mapped directory, so inside the docker. Only useful for testing." ... what is the disk speed, and what is the resulting download speed of the 1GB test file?

[flonkyg]

Created a folder inside the Docker by the name "downloadtest" and added two folders inside it: "incomplete" and "complete". I changed the settings in SAB to use these folders.

The 1GB test download was getting speeds of about 45 MB/s.

I now can also see the "Download speed limited by" field:

Code: Select all

Used cache  
0 B (0 articles)
System load  
0.31 | 0.33 | 0.13 | V=150M R=93M
Download speed limited by  
Disk speed (245x)
System performance (Pystone)  
398306 Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz AVX2 Docker
Download folder speed  
282.4 MB/s /downloadtest/incomplete
Complete folder speed  
217.6 MB/s /downloadtest/complete
Internet Bandwidth  
110.79 MB/s 886.32 Mbps

Based on this, the problem I'm encountering has to do something with the mounted path on the host. What confuses me is that I didn't have these problems before, and other hosts on my network don't have any problems writing to the same share with speeds close to 100 MB/s. I have not changed any settings or components prior to these issues.

[sander]

As expected. Good. With these internal folders, after a restart of SABnzbd, and then a few downloads, you should NOT see "Download speed limited by Disk speed (245x)". Can you confirm?

This has always been a problem with SABnzbd in docker and mapped drives. My advice: always keep Incomplete inside docker. Of course Complete must on a mapped drive. So ... it might help ... can you test?

> I didn't have these problems before

Maybe docker volumes could improve speed? See https://docs.docker.com/engine/storage/ ... ws%20hosts.

> other hosts on my network don't have any problems writing to the same share with speeds close to 100 MB/s

You could try what nzbget-in-docker achieves?

[flonkyg]

Okay, thank you for your advice!

I'll be leaving for a holiday tomorrow, but once I'm back I'll test stuff out.

[sander]

On my Ubuntu Linux, I tried two docker parameters: without and with "--security-opt seccomp:unconfined --privileged"


... but alas no differences: in both cases 600.000 pystone and around 600 MB/s disk speed (including the mapped drive). Pity.

Note: the mapped drive is on the local Linux host. No VM involved.

[flonkyg]

I actually had some time to fiddle with this before leaving. I read somewhere that the SAB's internal tests are somewhat unreliable, so I decided to dig in to the settings to see what can I find. 

It turns out I was only using 8 of my 30 available connections... After changing it to 30, the speeds are around 80-100MB/s.

[sander]

LOL

[flonkyg]

I know right?

Thanks for your efforts either way!