Page 1 of 1
linux rar 7zip support for multithreading
Posted: March 28th, 2016, 1:25 pm
by nonenix
hi i wondered about multithreaded rar archive file extraction on linux and after some googleing it seems that 7zip seems support multhithrading also also at least reading rars.
so I wondered if it would be possible to add an option to choose 7zip instead of unrar for linux so small server with multicore can benefit.
Especially since its the slowest bottleneck so far
(at least on my small server with raid / 16GB ram and atom quad core rar takes by far longest since it uses only one core and those atom boards don't have a big GHrz per CPU amount... multithreaded par runs fine so far so it would be nice and since the 16GB ram alsmot cache 100% of the download and the raid write speed isn't maxed out only the single core power is slowing the process)
Re: linux rar 7zip support for multithreading
Posted: March 28th, 2016, 2:06 pm
by sander
Did you test the speed differences?
I did:
Code: Select all
$ time unrar x *01.rar
UNRAR 5.00 beta 8 freeware Copyright (c) 1993-2013 Alexander Roshal
Extracting from bigrandomfile.part01.rar
<...>
All OK
real 0m13.926s
user 0m0.608s
sys 0m0.916s
versus
Code: Select all
$ time 7z x *01.rar
7-Zip 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,1 CPU)
Processing archive: bigrandomfile.part01.rar
Extracting random-300MB.bin
Everything is Ok
Size: 314572800
Compressed: 314575528
real 0m13.484s
user 0m1.200s
sys 0m0.708s
So no difference in real time.
Re: linux rar 7zip support for multithreading
Posted: March 28th, 2016, 2:19 pm
by safihre
Sander: in the log 7zip says '1 CPU'.
Maybe it does improve with bigger file and that switch to enable more CPU?
Re: linux rar 7zip support for multithreading
Posted: March 28th, 2016, 2:26 pm
by sander
safihre wrote:Sander: in the log 7zip says '1 CPU'.
Maybe it does improve with bigger file and that switch to enable more CPU?
Ah, good point; I was running it on my VPS. I'll find a multi-core system, with enough diskspace.
On a dual-core Atom:
First create a 300MB random file, then create the rar files:
Code: Select all
dd if=/dev/urandom of=random-300MB.bin bs=1M count=300
rar a bigrandomfile -v10m -m0 random-300MB.bin
Then unrar the files:
Code: Select all
sander@netbook:~/rar-kul$ time unrar x *01.rar
UNRAR 5.00 beta 8 freeware Copyright (c) 1993-2013 Alexander Roshal
Extracting from bigrandomfile.part01.rar
... random-300MB.bin OK
All OK
real 0m6.192s
user 0m1.940s
sys 0m1.932s
and
Code: Select all
sander@netbook:~/rar-kul$ time 7z x *01.rar
7-Zip 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
p7zip Version 9.20 (locale=C,Utf16=off,HugeFiles=on,2 CPUs)
Processing archive: bigrandomfile.part01.rar
Extracting random-300MB.bin
Everything is Ok
Size: 314572800
Compressed: 314575528
real 0m7.043s
user 0m2.628s
sys 0m1.852s
So ... 7z is slower?
Re: linux rar 7zip support for multithreading
Posted: August 30th, 2016, 9:25 pm
by fysa
sander wrote:
Code: Select all
real 0m6.192s
user 0m1.940s
sys 0m1.932s
vs
Code: Select all
real 0m7.043s
user 0m2.628s
sys 0m1.852s
So ... 7z is slower?
It's actually slightly faster in processing time -- compare the sys values. But the user time and remaining difference (wait time) is slower. This is likely because all threads were in contention for the handle for that single file.
Test this on a multi-part rar and/or a rar with many files and multiple parts and you will likely see faster values for 7z, at least on the second test.