One idea for GIL contention

One thing that would eliminate the problem of the single GIL from our app would be to load and reload the libpython dll into a new address space within the same process. Our app is a plugin and the instances never have to talk to each other, so if we could simply load libpython into a new, duplicated address space each time we created an instance of our plugin, we would not have to worry about the GIL.

Has anyone ever heard of this sort of thing? The same thing could effectively be achieved by creating libpython31_1.dylib, libpython31_2.dylib, libpython31_3.dylib, libpython31_4.dylib, etc, and then loading them in order, but this seems kind of excessive.

What do you think? Stupid? Genius?

Please don’t post comments like “Use multiprocessing module”, or “Use process migration”. Believe me, I’ve looked at it and they won’t work for our app. If you want to know why, read this post:

http://vedanamedia.com/?p=81

By | 2010-09-08T21:04:00+00:00 September 8th, 2010|Uncategorized|4 Comments

4 Comments

  1. jcalderone September 9, 2010 at 1:55 am - Reply

    At the very least, you’ll have to make sure you do the same thing for each extension module you load. I suspect you’ll run into other problems beyond that, as well, though perhaps not intractable ones. This is a somewhat limited solution though, as most applications would like to be able to interact between the units running on various extra hardware resources.

  2. Patricio September 9, 2010 at 2:26 am - Reply

    As described in the linked post, we don’t use extension modules. We simply want to execute python code from an embedded interpreter and make calls to our engine api.

  3. illume September 9, 2010 at 9:02 am - Reply

    See here: http://docs.python.org/c-api/init.html#Py_NewInterpreter

    I’ve had success mmap’ing buffers too. Then you can run your separate python script – which gets given audio buffers, or you can use message passing (through midi, or OSC). I imagine some of your users might want midi/osc anyway.

    cu.

  4. Patricio September 9, 2010 at 3:06 pm - Reply

    illume: Using separate interpreters does not afford you true multithreading, which is the goal here. What buffers have you mmap’d? dll buffers? audio buffers? Also, as of Python 2.5.1 (and I imagine little has changed in Python 3.1.2) multiple interpreters doesn’t actually work very well. The correct way to do this is still to use process migration.

    I am not aware of any dlopen hook or equivalent on windows or mac for loading a dll from memory. In fact, I’m sure there isn’t one, and if anyone has heard of anything similar that would be great!

Leave A Comment

− 6 = 4