Porting to Python3: ‘bytes’ in sys.path gotchya!

I’m busy hacking away trying to get my embedded scripting engine ported to python3.

The biggest pain in the ass is the new bytes VS str type. Today I ran into this error, raised once my import hook called the original builtins.__import__.

TypeError: must be string, not bytes

Oh geeze, thanks! What the hell does that mean? Want to see how rediculously long the path to debugging this problem was? Read on…

First thing was try to find the old builtin_import from import.c. No dice, it’s gone.

Next thing was find_frozen_module, since I had frozen the stdlib and thought there was a problem demarhsalling the code. No dice.

Then it dawned on me to put a breakpoint in PyErr_Format. No dice. That function is called more times just for Py_Initialize() than I have to restart my machine with m-audio hardware.

Then I set a breakpoint in PyErr_SetObject for exception == PyExc_TypeError. Bingo – the error is raised from PyArg_ParseTuple, as called by zipimporter_init(). Oh Boy.

Then I debugged vagetargsl() for a while, and finally found convertsimple(). Ahh, it’s getting a bytes object and not a string, as the “s:zipimporter” format string indicated. Finally, square one.

A look into the module docs say the first value should be a path. Debugging further up the call stack into import.c showed that the ‘args’ object passed to zipimporter_init() was created waaaay up in find_module(), and was generated from the sys.path object. I have a object in my sys.path instead of all objects. But why?

Because my application is embedded and the std lib is statically linked as frozen C source, I reset sys.path using “sys.path = [myinternalstuff.SCRIPTS_PATH]” in my app startup code. SCRIPTS_PATH is set in my internal module’s init function like this:

PyObject *scriptsdir = PyBytes_FromString(getScriptsDirPath());
PyModule_AddObject(m, “SCRIPTS_PATH”, scriptsdir);

THE GLORIOUS SOLUTION!!!

Reading the porting docs, I understood that PyString_FromString() has been replaced with PyBytes_FromString() instead of some PyUnicode_*() equivalent. For crying out loud!! I did a direct API swap-out, which led to this mega debugging fiasco due to poor error reporting, and an entire morning wasted.

Lesson learned, I guess? Not quite sure…

By | 2010-08-06T19:09:00+00:00 August 6th, 2010|Uncategorized|0 Comments

Leave A Comment

+ 42 = 52