Crashing while downloading

[tslayer]

When using SAB packaged in openelec (0.6.14), it would always crash while downloading.

I finally decided to run this inside of gdb to get the backtrace.

Code: Select all

Thread 4 (LWP 7099):
#0  0xb7789174 in PyType_IsSubtype () from /usr/lib/libpython2.7.so.1.0
#1  0xb72e2fd2 in ?? () from /usr/lib/python2.7/lib-dynload/cPickle.so
#2  0xb72e499a in ?? () from /usr/lib/python2.7/lib-dynload/cPickle.so
#3  0xb72e510e in ?? () from /usr/lib/python2.7/lib-dynload/cPickle.so
#4  0xb72e3eeb in ?? () from /usr/lib/python2.7/lib-dynload/cPickle.so
#5  0xb72e4cce in ?? () from /usr/lib/python2.7/lib-dynload/cPickle.so
#6  0xb72e4d82 in ?? () from /usr/lib/python2.7/lib-dynload/cPickle.so
#7  0xb776ab3f in PyCFunction_Call () from /usr/lib/libpython2.7.so.1.0
#8  0xb77d23bf in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#9  0xb77d4272 in PyEval_EvalCodeEx () from /usr/lib/libpython2.7.so.1.0
#10 0xb77533f7 in ?? () from /usr/lib/libpython2.7.so.1.0
#11 0xb7728d65 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
#12 0xb77cf5e0 in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#13 0xb77d4272 in PyEval_EvalCodeEx () from /usr/lib/libpython2.7.so.1.0
#14 0xb77d2b16 in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#15 0xb77d2bf9 in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#16 0xb77d4272 in PyEval_EvalCodeEx () from /usr/lib/libpython2.7.so.1.0
#17 0xb77533f7 in ?? () from /usr/lib/libpython2.7.so.1.0
#18 0xb7728d65 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
#19 0xb77cf5e0 in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#20 0xb77d4272 in PyEval_EvalCodeEx () from /usr/lib/libpython2.7.so.1.0
#21 0xb77d2b16 in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#22 0xb77d2bf9 in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#23 0xb77d2bf9 in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#24 0xb77d4272 in PyEval_EvalCodeEx () from /usr/lib/libpython2.7.so.1.0
#25 0xb77532fd in ?? () from /usr/lib/libpython2.7.so.1.0
#26 0xb7728d65 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
#27 0xb7736f3c in ?? () from /usr/lib/libpython2.7.so.1.0
---Type <return> to continue, or q <return> to quit---
#28 0xb7728d65 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
#29 0xb77ccb7c in PyEval_CallObjectWithKeywords () from /usr/lib/libpython2.7.so.1.0
#30 0xb7807281 in ?? () from /usr/lib/libpython2.7.so.1.0
#31 0xb76e6d31 in ?? () from /usr/lib/libpthread.so.0
#32 0xb760897e in clone () from /lib/libc.so.6

I then modified:
/storage/.xbmc/addons/service.downloadmanager.SABnzbd-Suite/SABnzbd/sabnzbd/__init__.py

Instead of : import cPickle
I changed it to: import pickle as cPickle

To not use the native c pickle version and that solved the issue.

You can find other folks running into the same issue here: http://openelec.tv/forum/16-supplementa ... ince-r8688

Thanks.

[shypike]

Good detective work!
The cPickle module works on all (most?) other platforms.
The pure-Python Pickle works a lot slower, but if it does the job better...

[tslayer]

Yeah..

And for what it's worth, this crashing did not occur on somewhat older versions. I am tempted to think this bug was introduced with your following commit here: https://github.com/sabnzbd/sabnzbd/comm ... 0195007a3e

Thanks.

[shypike]

Interesting.
This change makes SABnzbd more stable when downloading very large items (>20G) on the popular platforms.

It needs further investigation.
What kind of thing is this "OpenElec"?

[tslayer]

OpenElec is a stripped down linux version running XBMC.

Description from their website: "Open Embedded Linux Entertainment Center, or OpenELEC for short, is a small Linux distribution built from scratch as a platform to turn your computer into a complete XBMC media center. OpenELEC is designed to make your system boot as fast as possible and the install is so easy that anyone can turn a blank PC into a media machine in less than 15 minutes."

Thanks!

[shypike]

The strange thing is that both the old (<0.6.10) and the new code (>=0.6.10) should
work fine on any Python implementation, whether using Pickle or cPickle.
The reality is that the behaviour differs per Python version and platform port.
Even more worrying is that we're talking about fairly low level basic Python functionality.
It will take quite a while to investigate this.

[edgard]

This fix works, but on large files (30G+), pickle slows the machine to a crawl.

Seems to be a bug in cPickle + 64bits + Python 2.7 (wich Openelec was upgraded from 2.6 around r8688) that is being worked on:

http://bugs.python.org/issue13555

[shypike]

With larger NZBs, the administration to be saved is larger too.
There's room for improvement here...

BTW: did the problems occur on a 64bit version of OpenElec?