Long 'Repair > Starting Repair' phase for large nzb's
Posted: June 10th, 2011, 4:58 am
Version 0.6.4 on Ubuntu 11.04 'Natty'.
After downloading a nzb consisting of 20 50 file rarsets, I noticed the 'Repair > Staring Repair' message lingering for multiple minutes.
Some checking of the log files showed this to be related to the sabnzbd.newsunpack.build_filelists method. While there is a check_rar parameter, that only seems to apply to the joinables array, with every candidate for the rars array still being verified.
This would create a small but significant delay on a single 1000 file rar set, but there being 20 results in this operation being performed an additional 19 times.
Replacing (line 1217)
with
solved the problem for me, but I haven't spent enough time with the sabnzbd code to know if this change would break anything.
A more solid solution would probably be to change the is_rarfile method or create a wrapper method to add some elementary caching of results.
As I'm running some unholy VMware ESXi + Solaris + ZFS + NFS + Ubuntu hybrid, I realize this might not be a problem in most other use cases, but I hope you can still take a look at it.
After downloading a nzb consisting of 20 50 file rarsets, I noticed the 'Repair > Staring Repair' message lingering for multiple minutes.
Some checking of the log files showed this to be related to the sabnzbd.newsunpack.build_filelists method. While there is a check_rar parameter, that only seems to apply to the joinables array, with every candidate for the rars array still being verified.
This would create a small but significant delay on a single 1000 file rar set, but there being 20 results in this operation being performed an additional 19 times.
Replacing (line 1217)
Code: Select all
rars = [f for f in filelist if RAR_RE.search(f) and is_rarfile(f)]Code: Select all
if check_rar:
rars = [f for f in filelist if RAR_RE.search(f) and is_rarfile(f)]
else:
rars = [f for f in filelist if RAR_RE.search(f)]
A more solid solution would probably be to change the is_rarfile method or create a wrapper method to add some elementary caching of results.
As I'm running some unholy VMware ESXi + Solaris + ZFS + NFS + Ubuntu hybrid, I realize this might not be a problem in most other use cases, but I hope you can still take a look at it.