A truely concurrent python interpreter would allow the language to be used in a realtime dsp engine while simultaneously used for the user interface. This has been brought up before and is a point of contention for the python project since the GIL is really really tough to remove. I suppose if I was able to provide a patch with a build option to at least provide a concurrently capable interpreter the language could be used in very specialized multithreaded cases. This is a pretty specialized requirement though, so maybe that’s good enough? We just wanted the language itself, and I could have easily patched the sip and PyQt extensions to accommodate the change.
Having moved to a C++ project with strict performance requirements, we’ve spent a ton of time on build config and memory related issues. Not only is the majority of our effort not spent on writing features, but this “wasted” time also happens to be the most agonizing. Being a hopeless idealist and having nothing but subjective and difficult to argue cases to present to my boss, I thought I’d just try to put it all on paper to remember for the next go around.
Developing a modern audio plugin with good platform support provides plenty of code management challenges. The plugin formats we support are AU/VST/RTAS, the platforms are Windows, Mac/ppc/intel, and all 32/64 bit. No one has ever bothered to count the total, but I do know that it amounts to a big pain. We use XCode 2.5 and Visual Studio 2005, and have incorporated qmake to use qt for the gui into both. Our main targets are a standalone app or a plugin, which in turn loads a separate plugin for each of our products that you’ve purchased. This means you have to maintain several build configurations over several platforms, and they all behave differently.
Lesson 1: Separate features and config
Clearly define your problem domain, and try to separate your build and deployment environment from the development environment as much as possible. For example, our problem domain is to implement a high-performance sample playback engine and accompanying UI, so adding some new graphical classes and menus should not affect the build config, and visa-versa. When you write in C/C++, you will typically define config macros on the command line, add and remove files to be compiled, etc, and all of this stuff will change on each platform. For example, when I add a QWidget subclass, I have to add the files to XCode, add an entry in the qmake project to generate the meta source, add the meta source to the XCode, and do it all over again for Visual Studio.
Conflicting headers has also caused us plenty of problems. Our primary third-party libraries are juce, Qt, and python. Python and Qt conflict over the “check” symbol, and juce and Qt conflict over the T macro and others. The juce library is nice enough to include “using namespace juce” in it’s headers, so we get conflicts with just about everything, including system frameworks like Carbon. To solve this problem I had to find the correct order of #include and #undef statements to get all the libraries to play nice together. Don’t even bring up windows.h. Finding a solid scheme for including the fundamentals is very important, and you should never have to worry about it in your feature code.
In a nutshell, You should write features with 0% brain power spent on how your features will affect the build config.
Lesson 2: Unify your build environment
Managing your build configuration is a pain in the butt. qmake does a really good job of flattening the tool chain, but as long as you are using more than one compiler you are going to have to have conditional variables built into the build config. This is almost always the case when you support more than one platform.
If you can manage to flatten the config to a point where you do not need to separate changes for your different platforms, you’ve taken a huge step. Adding a new library to your code will cause a change to the config, so the cleaner the libraries you choose, the better. Qt, sip, and PyQt are examples of libraries that add little to no compilation and linkage config overhead to a project because their lack of dependencies allow them to be relatively self-contained. The sip code actually compiles with no special flags.
Lesson 3: Write modular code
I know all of this stuff sounds elementary, but it’s all very important. The more black boxes you write, the less they will change. Our project consists of a few key components; the audio engine, the gui, the product plugins, and the target exe/plugin (of which there are many). Trying to meet our stupidly tight initial deadlines, our project manager originally had everything compiled into a single target for each exe/plugin. Granted, this would have fixed the symbol visibility problem that we are having right now, but it added significant compile time and the code behaved differently everywhere.
It doesn’t really matter how much intertwangling there is within the bounds of each component (your boss is going to assign those bugs to you anyway and you’ll just go fix them), but it does matter that your major components are well separated. We aaaalmooooost nailed it in our project by creating engine, gui, and product libraries, but we unnecessarily mixed the headers and instantiated objects from each in the target exe’s. A better approach is to provide a single header for each library that includes only pure virtual classes and some global factory functions. These headers should not define any types, and should not include each other.
I like using our project as an example for project management topics because it has some relatively tough requirements. The audio engine must be written in C++ because performance is top priority, and the gui code could be written in python. Given that the engine is as small as possible and 100% independent of the other components, we could have written a small layer to move all of the application code into python. That way we could have added and tested new features with little worry to causing crashes and compilation/linking related issues.
As a result, the two most interesting problems that I would like to solve are:
1) A 100% self contained AU/VST/RTAS wrapper with PyQt support.
For the plugin wrapper, a flexible engine => gui interface would be key for component communication, but also the build config for the compiled elements would need to require as few compiler options as necessary to make it easy to be compiled as a local target as opposed to linked as a library. This way, the developers would have total access to the code once it was included in their project and would not have to deal with any overhead related to integrating it into their projects. Including updates to said code could be tricky, but I’m convinced that the changes could be made small enough to remain manageable.
2) A concurrent python interpreter.