Random crashes during saving file to cache folder

[habrys]

Here is my environment:

Version: 0.4.6, 0.4.11 - both tested
OS: Debian Sarge, kernel 2.6.16.16, installed on arm NAS (armv4tl)
Install-type: python source (python 2.5 itself and modules also built/installed from source)
Skin: Default
Firewall Software: None, just router's firewall
Are you using IPV6? no
Is the issue reproducible? no, it seems to happen randomly

I might add, that I installed SABnzb on a Linux server / NAS called bubba. It runs under Debian Sarge and on ARM processor. Since there is no Python 2.5 package in Debian Sarge repositories, I had to compile Python from sources and install all the modules manually. I used this howto to do it: http://www.thibernet.com/442-install-sa ... adynas-duo

Basically all works as expected. Download, postprocessing, access from another computer, webgui. But... sometimes it just crashes. All python2.5 processes crash once a day or so. The last line I can see in logfile before crash is something like that:

Code: Select all

2009-06-03 01:26:42,522::INFO::[sabnzbd] Saving data for SABnzbd_nzo_I2atcA in /home/storage/nzb/cache/SABnzbd_nzo_I2atcA

After starting SABnzb daemon again I get following error message:

Code: Select all

2009-06-03 01:58:14,155::ERROR::[sabnzbd] Loading /home/storage/nzb/cache/SABnzbd_nzo_I2atcA failed

and the currently processed job disappears from the queue.

The file itself exists:

Code: Select all

-rw-------  1 root root 61440 Jun  3 01:26 SABnzbd_nzo_I2atcA

so I assume it gets corrupted during crash and cannot be loaded anymore.

Any ideas what could cause the problem and how to solve it?

And the second question... The files of the partially downloaded job stay in the downloads/incomplete/ folder after crash. Is there a way to resume the job from where it crashed? The data is still there after all... When I simply start the same nzb via webgui again, it starts to download again from scratch in a separate folder called something.1. It's very annoying considering a 13GB file, which I downloaded lately takes 1-2 days to download on my 1MB connection...

Some additional information here.
The problem is not related to some particular nzb file(s).
I tried both SABnzb versions: 0.4.6 and 0.4.11

First I used python 2.5.2 with following versions of modules (as described in howto I referenced to earlier):
Cheetah-2.0.1
yenc-0.3
feedparser-4.1
pyOpenSSL-0.8

and then the latest 2.5 python (2.5.4) with latest versions of the modules:
Cheetah-2.2.0
yenc-0.3
feedparser-4.1
pyOpenSSL-0.9

There is no difference, random crashes still occur.
I tried also non-SSL connection. No difference.

Now I use the 0.4.9 version from Ubuntu repositories on my "big" PC - to test if I can see the same errors there (I don't think so) and as backup solution till I solve the problem in my little bubba server...

I'd greatly appreciate any help. I fight with that annoying problem since days with no effect...

[switch]

Probably a cPickle issue. Either not compiled at all, or is unstable.

I would try opening __init__.py in the sabnzbd folder and changing

Code: Select all

import cPickle

on line 27 and changing it to the following:

Code: Select all

import pickle as cPickle

As for resuming crashed data, no their is no solution for it. Crashes like these are so rare it isn't worth programming in support for.

[shypike]

Do not use CherryPy module, but the CherryPy delivered with the distribution package.
We use pyOpenSSL 0.7

If it consistently crashes on saving article files,
it could very well be the problem switch mentioned (cPickle).

Crashes need not be fatal, SABnzbd should pick up where it left.
There might be some duplicate downloading though.
The problem is that what is saved on disk may be inconsistent because
the dump to files crashed halfway.

[habrys]

Thanks a lot for very fast answer. I changed the line as you suggested, switch and started the 13GB job again. We'll see if it helps after a few days, so I'll just sit with crossed fingers and pray for now

Yes, shypike it crashes only on saving article files, so I didn't change anything to CherryPy or pyOpenSSL for now.

One more thing.
SABnzb is simply great, just what I needed for my tiny bubba server. Much better, than hellanzb I used earlier. Now if only these random crashes stopped... Keep up good work!

[habrys]

Alas your solution with cPickle didn't help, switch. I,ve had another crash, very similar to the previous ones. Here messages from logfile.

Last 2 lines before crash:

Code: Select all

2009-06-04 06:11:59,171::INFO::[sabnzbd] Saving data for SABnzbd_nzo_XTji-t in /home/storage/nzb/cache/SABnzbd_nzo_XTji-t
2009-06-04 06:12:03,375::INFO::[downloader] Thread 5@ssl-eu.astraweb.com:563: part2of391.L3PBfsGSECwufDAjUIsq@postin.things.org done

...and the error message on the startup after crash:

Code: Select all

2009-06-04 07:58:20,055::ERROR::[sabnzbd] Loading /home/storage/nzb/cache/SABnzbd_nzo_XTji-t failed

The file exists again:

Code: Select all

/home/storage/nzb/cache/SABnzbd_nzo_XTji-t

The job disappeared from the queue, as before. The only difference I see this time is the very last line about done thread, which wasn't there before.

In the meantime I downloaded the 13GB file on my Ubuntu PC using SABnzbplus 0.4.9 installed from repositories without any problems.

I'm going to look into CherryPy I use and change pyOpenSSL to 0.7 now, as shypike suggested. But I don't really think it'll help much. CherryPy is responsible for webgui, right? These crashes happen when I don't look, so they shouldn't have anything to do with CherryPy. And pyOpenSSL... well... I tried to disable SSL and it didn't help, so I don't think the problem lies in version of pyOpenSSL.

But I try it anyway.

Any more suggestions? Maybe I could switch on some debug or trace mode if you are interested in pinpointing the problem? Or even add some debug/dump lines to the python code?

[switch]

It depends what type of crash it is. Python is very good about handling and reporting tracebacks of python code exceptions and should display them in the terminal window if you don't start sabnzbd with the -d flag.

Some of these crashes aren't logged to the sabnzbd.log in 0.4.x, so I would look in your terminal window - if that is not what you are doing already - for any exception logging.

If it is an internal python crash, then there is little we can suggest as these types of crashes are rare and hard to debug and not supposed to happen with python code. They are pretty much always not the fault of sabnzbd's and is some misconfiguration of python or hardware/software stability issues.

[habrys]

I used sabnzbd in daemon mode so far - on bubba NAS I mean. That's the whole point: to be able to run it all the time without the big PC on. Saves money (bubba needs max. 10W to operate), reduces noise etc.

All right, I'll run sabnzb on bubba without -d flag and leave the pc with terminal window on - for test purposes. Thanks for your advice.

I'm also starting to think, that it's some problem with python itself or some python modules.

One more question. I noticed following warnings in the sabnzbd.error.log file:

Code: Select all

 *** WARNING *** 
 The following keys are members of the Template class and will result in NameMapper collisions! 
  > session 

 Please change the key's name or use the compiler setting "prioritizeSearchListOverSelf=True" to prevent the NameMapper from using 
 the Template member in place of your searchList variable 
 *************** 

Could it be related to the crashes?

[switch]

That looks more like a Cheetah error, which could cause crashing, however not in the manner you are experiencing.

What drive are you saving it to, and what format is it formatted in? Is it possible to try another drive - external?

I still think this is an error in how python was compiled. Maybe try going back to python 2.4.

[habrys]

It's a 750GB PATA Seagate Barracuda drive, which came preinstalled with the bubba server. It should be suitable for long-term 24/7 operation. The filesystem is ext3.

Yes, it is possible to use an external USB drive, but it's not easy and I don't have one right now. I don't think it's a problem with the drive. I used it constantly with a torrent client before without any problems... If nothing helps, I'd rather buy a new version of bubba (Bubba | Two), which comes with Debian Etch and python 2.5 preinstalled.

As for python 2.4, yes I thought about it and will try it out eventually. Unfortunately the last python in Debian Sarge repositories is 2.3, so 2.4 needs compiling anyway. But I'll try it out.

For now I just started another job in terminal, without -d option. I hope the error message from python gives me some clues.

[shypike]

I used sabnzbd in daemon mode so far - on bubba NAS I mean. That's the whole point: to be able to run it all the time without the big PC on. Saves

Code: Select all

 *** WARNING *** 
 The following keys are members of the Template class and will result in NameMapper collisions! 
  > session 

 Please change the key's name or use the compiler setting "prioritizeSearchListOverSelf=True" to prevent the NameMapper from using 
 the Template member in place of your searchList variable 
 *************** 

Could it be related to the crashes?

No this will not cause crashes, it will affect the Web-UI session-key security fix that is in 0.4.11.
It means that your Cheetah is too recent. The latest Cheetah (after 2 years of silence)
contains some incompatible changes.
If you grab the latest 0.4.x interface.py from the SVN archive, this problem will be fixed.
(Or go back to Cheetah 2.01)

[habrys]

Bad news.

Downgrading to Python 2.4.6 didn't help, I just got the same crash as usually. This time I used the recommended versions of modules too: pyOpenSSL 0.7 and Cheetah 2.0.1.

I didn't see any error message from Python itself. I started without -d flag, but with nohup, like this:

Code: Select all

nohup /usr/local/bin/python2.4 /home/habrys/src/SABnzbd-0.4.11/SABnzbd.py -f /home/storage/nzb/sabnzbd.ini &

As far as I know, this way all messages (also these, which aren't logged to sabnzbd.log file) should land in nohup.out file, without having to keep open terminal for days. Or am I mistaken?

Anyway, I'm starting to think, that it could be some problem with synchronization between threads, so I just started sabnzbd again with only one connection, instead of 8 or 12. These things (thread racing conditions etc.) tend to happen more often on slower systems, so there can be a bug which is pretty much invisible on normal machines. Bubba runs on a 200 MHz CPU and sabnzbd uses 30-40% of it's resources all the time. Perfect environment to provoke threading synchronization problem...

[shypike]

Normally you should always use the -d option.
It's there for a reason.
The only things that your solution may grab is panic crash messages (from Python itself).
So maybe it's useful for an experiment.
There are more people using SABnzbd on slow systems, without similar complaints

The general problem with SABnzbd on Linux systems is that we rely
on so many third-party modules that troubleshooting someones setup is
almost impossible.
For Windows and OSX we distribute binaries where we control all components.

I look forward to your testing results!

[jubei]

I used sabnzbd in daemon mode so far - on bubba NAS I mean. That's the whole point: to be able to run it all the time without the big PC on. Saves

Code: Select all

 *** WARNING *** 
 The following keys are members of the Template class and will result in NameMapper collisions! 
  > session 

 Please change the key's name or use the compiler setting "prioritizeSearchListOverSelf=True" to prevent the NameMapper from using 
 the Template member in place of your searchList variable 
 *************** 

Could it be related to the crashes?

No this will not cause crashes, it will affect the Web-UI session-key security fix that is in 0.4.11.
It means that your Cheetah is too recent. The latest Cheetah (after 2 years of silence)
contains some incompatible changes.
If you grab the latest 0.4.x interface.py from the SVN archive, this problem will be fixed.
(Or go back to Cheetah 2.01)

Okay I'm getting the same exact error on my FreeNAS (FreeBSD 7.2) box....no settings will save because of it, invalid session key.  Any clue as to where I could get py26-cheetah-2.0.1?  I could install that via pkg_add before I install sab...  Thanks!

[jubei]

Okay I got interface.py from here

http://trac2.assembla.com/SABnzbd/brows ... c1/sabnzbd

I uploaded it with WinSCP to:

Code: Select all

/usr/local/lib/python2.6/site-packages/sabnzbd

Now I get errors when I start up SABnzbd.py:

Code: Select all

/usr/local/lib/python2.6/site-packages/cherrypy/lib/cptools.py:3: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5
/usr/local/lib/python2.6/site-packages/cherrypy/filters/sessionfilter.py:33: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha
Traceback (most recent call last):
  File "/usr/local/bin/SABnzbd.py", line 44, in <module>
    import sabnzbd
  File "/usr/local/lib/python2.6/site-packages/sabnzbd/__init__.py", line 52, in <module>
    from sabnzbd.assembler import Assembler
  File "/usr/local/lib/python2.6/site-packages/sabnzbd/assembler.py", line 34, in <module>
    from sabnzbd.interface import CheckFreeSpace
  File "/usr/local/lib/python2.6/site-packages/sabnzbd/interface.py", line 54, in <module>
    from sabnzbd.codecs import TRANS, xml_name, LatinFilter
ImportError: cannot import name LatinFilter

[shypike]

SABnzbd 0.4.11 needs Python 2.4 or 2.5, it's not compatible with 2.6!