TypeError: character mapping must return integer, None or un

[pyro42]

I'm attempting to do a fresh install of Alpha3 on a clean install of Windows Server 2012 Essentials R2. I am attempted to set it up as a server following the direction from http://wiki.sabnzbd.org/how-to-install-on-whs & http://wiki.sabnzbd.org/sabnzbd-as-a-windows-service.

The install goes well, I configure the service, get the firewall to allow it though. And now I am getting:

http://Servernamehere:8080/wizard/

Code: Select all

500 Internal Server Error

The server encountered an unexpected condition which prevented it from fulfilling the request.

Traceback (most recent call last):
  File "cherrypy\_cprequest.pyo", line 656, in respond
  File "cherrypy\lib\encoding.pyo", line 188, in __call__
  File "cherrypy\_cpdispatch.pyo", line 34, in __call__
  File "sabnzbd\wizard.pyo", line 71, in index
  File "Cheetah\Template.pyo", line 1259, in __init__
  File "Cheetah\Template.pyo", line 1553, in _compile
  File "Cheetah\Template.pyo", line 637, in compile
  File "Cheetah\convertTmplPathToModuleName.pyo", line 20, in convertTmplPathToModuleName
TypeError: character mapping must return integer, None or unicode

http://forums.sabnzbd.org/viewtopic.php?f=11&t=18976
and
http://forums.sabnzbd.org/viewtopic.php?f=11&t=18977

Seem to describe the same problem and a possible solution.

"c:\program files (x86)\sabnzbd\sabnzbd-console.exe" -t smpl
launches a browser that displays the same error. However if I stop the SABnzbd-service and run the command, it launches a browser with the configuration wizard.

User nate1280 was seemingly able to get around it by compiling some python files and inserting them, but I'm not quite comfortable going that far. I would be more than happy to test any changes you need though.




Powered by CherryPy 3.2.2

[shypike]

The cause is a registry entry with a non-standard name.
Solving it is hard to do with Alpha3.
Alpha4, which should come out this week, has a work-around for this.

[pyro42]

any idea what? like i said, this is a clean install of WSE 2012..

off the top of my head, the only things installed before sab were...

SATA PCI card drivers
Search Everything
Chrome
WinDirStat
Stablebit Drivepool
Stablebit Scanner

[shypike]

IIRC it was about odd names in Mime types.
Chrome sets MIME types, but cannot the the sole cause, otherwise more people would have this issue.

[sander]

@Shypike: just checking & FWIW:

I'm able to reproduce the error by feeding convertTmplPathToModuleName() with a unicode value:

Code: Select all

>>> from Cheetah.convertTmplPathToModuleName import convertTmplPathToModuleName

>>> convertTmplPathToModuleName('/tmp/hello.txt')
'_tmp_hello_txt'

>>> convertTmplPathToModuleName(u'/tmp/hello.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/Cheetah/convertTmplPathToModuleName.py", line 20, in convertTmplPathToModuleName
    return splitdrive(tmplPath)[1].translate(_pathNameTransChars)
TypeError: character mapping must return integer, None or unicode
>>>

So ... is this what you are referring to and has been worked around in Alpha4?

S

[shypike]

I'm able to reproduce the error by feeding convertTmplPathToModuleName() with a unicode value:

Hmm, this is indeed something else.
Like Cheetah not being Unicode-compliant
But then again, why doesn't this happen on all Windows systems)?

Looking at the suggestions in the referred topics, a Cheetah patch is needed,
which isn't a solution because for Linux/Unix systems we rely on an already installed Cheetah version.
That means the Cheetah paths shouldn't be Unicode, but that would give problems for people
with user names outside of the Latin-1 character set.
I'll investigate further.

[sander]

The problem is in string.translate(), not in Cheetah:

Code: Select all

>>> import string
>>> tab = string.maketrans('ab','XY')
>>> 'baldadig'.translate(tab)
'YXldXdig'
>>> u'baldadig'.translate(tab)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: character mapping must return integer, None or unicode
>>> 

I'll google around to find more information; I'm quite sure string.translate() can handle unicode.

EDIT:

http://stackoverflow.com/questions/1324 ... de-strings

Code: Select all

>>> u'baldadig'.translate(tab.decode("latin-1"))
u'YXldXdig'

So ... first put the the translate table through .decode("latin-1") ?

Or:

Code: Select all

>>> str(u'balbadig').translate(tab)
'YXlYXdig'

... first put the to-be-translated string through str() ?

For reference:

Code: Select all

import os.path
import string

letters = None
try:
    letters = string.ascii_letters
except AttributeError:
    letters = string.letters

l = ['_'] * 256
for c in string.digits + letters:
    l[ord(c)] = c
_pathNameTransChars = ''.join(l)
del l, c

def convertTmplPathToModuleName(tmplPath,
                                _pathNameTransChars=_pathNameTransChars,
                                splitdrive=os.path.splitdrive,
                                ):
    print "SJXXX a unicode", u'Hello', isinstance(u'Hello', unicode)
    print "SJXXX tmplPath", tmplPath, isinstance(tmplPath, unicode)
    print "SJXXX _pathNameTransChars", _pathNameTransChars, isinstance(_pathNameTransChars, unicode)
    print "SJXXX splitdrive", splitdrive, isinstance(splitdrive, unicode)
    print "SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars)", splitdrive(tmplPath)[1].translate(_pathNameTransChars), isinstance(splitdrive(tmplPath)[1].translate(_pathNameTransChars), unicode)

    return splitdrive(tmplPath)[1].translate(_pathNameTransChars)

[sander]

I put logging into /usr/lib/python2.7/dist-packages/Cheetah/convertTmplPathToModuleName.py, and the result (below) only shows plain strings, no unicode. Strange (assuming my logging method is correct). Oh wait: that does explain why I don't get errors, so at least that's consistent.

Code: Select all

SJXXX tmplPath /home/sander/git/sabnzbd/interfaces/Plush/templates/main.tmpl
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________
SJXXX splitdrive <function splitdrive at 0x7fe6df2191b8>
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git_sabnzbd_interfaces_Plush_templates_main_tmpl
SJXXX tmplPath /home/sander/git/sabnzbd/interfaces/Plush/templates/_inc_header.tmpl
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________
SJXXX splitdrive <function splitdrive at 0x7fe6df2191b8>
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git_sabnzbd_interfaces_Plush_templates__inc_header_tmpl
SJXXX tmplPath /home/sander/git/sabnzbd/interfaces/Plush/templates/_inc_footer.tmpl
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________
SJXXX splitdrive <function splitdrive at 0x7fe6df2191b8>
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git_sabnzbd_interfaces_Plush_templates__inc_footer_tmpl
SJXXX tmplPath /home/sander/git/sabnzbd/interfaces/Plush/templates/_inc_modals.tmpl
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________
SJXXX splitdrive <function splitdrive at 0x7fe6df2191b8>
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git_sabnzbd_interfaces_Plush_templates__inc_modals_tmpl
SJXXX tmplPath /home/sander/git/sabnzbd/interfaces/Plush/templates/queue.tmpl
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________
SJXXX splitdrive <function splitdrive at 0x7fe6df2191b8>
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git_sabnzbd_interfaces_Plush_templates_queue_tmpl
SJXXX tmplPath /home/sander/git/sabnzbd/interfaces/Plush/templates/history.tmpl
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________
SJXXX splitdrive <function splitdrive at 0x7fe6df2191b8>
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git_sabnzbd_interfaces_Plush_templates_history_tmpl
SJXXX tmplPath /home/sander/git/sabnzbd/interfaces/Plush/templates/queue.tmpl
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________
SJXXX splitdrive <function splitdrive at 0x7fe6df2191b8>
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git_sabnzbd_interfaces_Plush_templates_queue_tmpl
SJXXX tmplPath /home/sander/git/sabnzbd/interfaces/Plush/templates/history.tmpl
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________
SJXXX splitdrive <function splitdrive at 0x7fe6df2191b8>
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git_sabnzbd_interfaces_Plush_templates_history_tmpl
SJXXX tmplPath /home/sander/git/sabnzbd/interfaces/Plush/templates/queue.tmpl
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________
SJXXX splitdrive <function splitdrive at 0x7fe6df2191b8>
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git_sabnzbd_interfaces_Plush_templates_queue_tmpl
SJXXX tmplPath /home/sander/git/sabnzbd/interfaces/Plush/templates/history.tmpl
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________
SJXXX splitdrive <function splitdrive at 0x7fe6df2191b8>
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git_sabnzbd_interfaces_Plush_templates_history_tmpl

Extra logging: isinstance(..., unicode) at the end of each line:

Code: Select all

SJXXX a unicode Hello True
SJXXX tmplPath /home/sander/git/sabnzbd/interfaces/Plush/templates/queue.tmpl False
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________ False
SJXXX splitdrive <function splitdrive at 0x7f9ec9fcc1b8> False
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git_sabnzbd_interfaces_Plush_templates_queue_tmpl False

So, yes, all parameters or non-unicode, thus plain strings.

TBC

[sander]

EDIT:

Ignore the below as

Code: Select all

>>> isinstance('/home/sander/git/Ελλάδα/sabnzbd', unicode)
False

... meaning my testing method is incorrect.


IGNORE:

Strange: I moved sabnzb/ (and thus the templates) into /home/sander/git/Ελλάδα/sabnzbd (thus Unicode), but there is NO unicode in the logging:

Code: Select all

SJXXX a unicode Hello True
SJXXX tmplPath /home/sander/git/Ελλάδα/sabnzbd/interfaces/Plush/templates/main.tmpl False
SJXXX _pathNameTransChars ________________________________________________0123456789_______ABCDEFGHIJKLMNOPQRSTUVWXYZ______abcdefghijklmnopqrstuvwxyz_____________________________________________________________________________________________________________________________________ False
SJXXX splitdrive <function splitdrive at 0x7fa6874481b8> False
SJXXX splitdrive(tmplPath)[1].translate(_pathNameTransChars) _home_sander_git______________sabnzbd_interfaces_Plush_templates_main_tmpl False

[shypike]

I think I'm going to monkey-patch Cheetah.
The module in question is stand-alone and backwards-compatible with previous versions.
The problem is in fact the bizarre way Python handles mixes of ASCII and Unicode.

The solution is this one, already from 2012, but there are so many older versions of Cheetah floating around.
https://github.com/cheetahtemplate/chee ... 89d9993eb7

[shypike]

The weird thing is that is most cases the path name fed to Cheetah is just plain old ASCII.
But it seems that in some cases it is not.
What is the right thing to do?
Force the path to ASCII or force Cheetah to do it right (with a monkey patch)?

@Sander: I can confirm your test for OSX. The path is UTF-8 (which counts as ASCII for Python).
Possibly it only wrong on Windows, in which case I can simply use a correct Cheetah version for the binary distribution.

[shypike]

I have not been able to reproduce the error.
What I will do for Alpha4 is simply patch the Cheetah module of the build.
That should at least solve the problem for the people who have reported this issue.

[sander]

So ... the problem is known in Cheetah, it was patched in Jan 2012, but in Sept 2015 (3.5 years later) that patched version of Cheetah is not used everywhere?

I checked my Ubuntu 15.04 and it is not patched and it says "VersionTuple = (2, 4, 4, 'development', 0)" (not 1). So indeed not patched pre-Jan-2012. :-(

And the monkey patch 'just' means expiditing the known patch on your Windows build setup? That sounds safe and pragmatic.

One remark as we can't reproduce it: Pity. I was thinking about putting more logging in SAB to find it out, but probably that is just curiosity and not really necessary with putting in the patch.

[pyro42]

Well this has certainly sparked more than I expected lol. Glad to hear there is a solution coming.

So why does it only happen when you try to run sabnzbd as a Service and not when you run it normally?

[sander]

Well this has certainly sparked more than I expected lol. Glad to hear there is a solution coming.

So why does it only happen when you try to run sabnzbd as a Service and not when you run it normally?

I have no Windows, so I can't verify with you.
Question:
Can you run SABnzbd from the terminal (so not as a service), with the same settings as the Service? Does that work?
If so, as soon as you SABnzbd as a service, you get the mentioned error?

If Yes and Yes, then I see something weird in the traceback: 'File "sabnzbd\wizard.pyo",', meaning SAB is starting the wizard. SAB only starts the wizard if it can't find an INI file...