Consider this scenario: The CPython interpreter is embedded in a C++ app. Users can use a built-in editor to define a script or two with pre-defined callbacks, and can make calls on the application’s engine. Users can write extension modules and packages, and can import and communicate other scripts written right in the editor. A standard library is included as frozen source in the application executable to be imported by the user’s scripts.

This poses a very interesting problem with regards to global data stored in the interpreter. Specifically, how would you handle importing modules with respect to the cache in sys.modules? How do you handle editing script A in the editor then recompiling it? What happens when you’ve written script B in the editor and it imports script A, and you edit and re-compile script A?

Importing modules like ‘random’ and ‘math don’t post much of a problem, since they don’t store local data. But, how do you expose the engine’s objects as globals? If our app is loaded as a plugin then there is more than one engine, but still only one set of __builtins__, one sys.modules, etc.

What I have so far:

Our scripts have the ability to manipulate the application engine, and can add gui controls to a little window for controlling run-time parameters. Access to these objects is provided by adding “engine” and “interface” objects directly to the module’s __dict__ when the script is compiled.

But, how are these objects added when importing a frozen module in the application’s standard library using the built-in dotted-named import? I’ve implemented an import hook, but the dotted name import doesn’t need it when importing frozen modules.

Currently I define an import hook that removes every module from sys.modules right after it’s imported by the original import mechanism. This causes the module to be reloaded from frozen or disk storage whenever it’s imported. This currently works well for us.

But what about that dotted-name import?

from myutils import This, That, Other

When This, That, or Other is imported using the above scheme, your import hook (set using __builtin__.__import__ = myimport) doesn’t get called for This, That, Other, so you can’t add the “engine” and “interface” objects. I’m sure there’s another way to do this, but what is the other way to do this?

Getting into the CPython code is a little intimidating, to say the least. I have no idea if I’m doing any of this import stuff the right way. Is there a right way?

How have other people handled this? Any similar or slightly different situations out there?