It’s time to get serious about processes

No matter how hard it may be to say or accept it, I think that the real answer for this whole GIL/process thing is to just buckle down and start getting better at using processes.

Opening the Chrome beta reminded me that using more processes should not be a scary thing. Before we got used to async javascript requests, we never would have thought to develop full-on applications using a web browser. But, the basic technology was there, someone just had to do it. No one would have thought that streaming video to a phone would work, but now they do it, and now we know that only 3% of AT&T’s users are causing the majority of their bandwidth issues.

We just need more practice using processes, and we need better tools to use them. The multiprocessing module is a TERRIFIC example of this. Now it’s easy to write tools from the process-domain. Other than the technical difficulty of writing code to manage child processes and use shared memory, I can’t think of a single reason why we shouldn’t just be using processes instead of threads.

My motivation for this comes completely from the annoyance of the gil.

Unfortunately, I have no idea how to reliably start and stop a child process on both windows and mac, and I have never used shared memory. It seems like a really great library to do just that and to map objects in shared memory as if they were in the same processes would be the real killer “app” here.

I mean seriously…

By | 2009-12-11T20:08:00+00:00 December 11th, 2009|Uncategorized|3 Comments

3 Comments

  1. Lucian December 11, 2009 at 9:46 pm - Reply

    Threads explicitly share memory, which is the main problem with them. Sharing should be an implementation detail and so should threads for that matter.

    So until there’s a green-thread library that can multiplex over OS threads for SMP scalability for python (like Erlang’s runtime), go ahead and use multiprocessing. It’s great for easier process management.

  2. Peter December 11, 2009 at 11:47 pm - Reply

    Have you looked into PSI? (http://psychofx.com/psi/) It supports a cross platform mechanism for fetching process information such as child process and state and termination.

  3. redmoxie December 12, 2009 at 1:35 pm - Reply

    There are a number of ways to use multiple processes that don’t involve a single master process. Using AMQP producers and consumers in place of greenlet channels allows you to create large async systems with as many processes as you like.

Leave A Comment

+ 46 = 56