Consider this problem:
Your C++ project includes libpython and special multiprocessing constraints require you to move the CPython interpreter code and GIL out of the main process. No, you can’t use multiprocessing or any other python-based code because it requires the GIL. The GIL must move. The GIL will move.
You can use your nifty CS degree to easily abstract your use of the python interpreter into an abstract interface. Communication with the interface is easily serializable. You’re well on your way.
But wait, there’s one problem. Clean, tight process management is something you’ve never worked with before – not to mention on Windows and Mac. Your requirements are simple, but the challenge is hard.
“All I want to do is start a process, get a read and a write file descriptor, and kill the process.” Sorry, but it’s not that simple. Unix has sockets, Windows does not. They both use radically different calls to manage processes. And what about shared memory? Sheesh…scary.
You won’t be firing off new processes very often, but you can’t afford hung processes. You need 100% reliability. You can’t afford much time to waste time debugging runaway processes. You just need the basic process management code to work – and all in C++.
How simple can this be? Surely there is a library for this? is it just a little code to copy and maintain myself? Seems like it should be.
If we are going to leave the GIL as is, shouldn’t it’ be a no-brainer to attack a problem like this? Is there a C++ version of multiprocessing, or at least something close? If we are going to accept process migration as a viable solution to true python multiprocessing, shouldn’t we remove problems like this from the equation?
Can we simply do this with better tools? Should we just do this with other tools? I think so.
The CPython interpreter is fast. It’s like, really super fast, and we use it in a real-time audio application without so much as a blip on the profiler. It is such a shame that such a fast, clean language can’t be used by our several threads without overcoming this crazy process migration hurdle. I’d love to move it to a new process, but first I have to find out how!