Archive for March, 2009

Sometimes Using Less Abstraction is Better

March 24, 2009

Last week I was working on getting the Apache Qpid unit tests running on Windows. The unit tests are arranged to take advantage of the fact that the bulk of the Qpid client and broker is built as shared/dynamic libraries. The unit tests invoke capabilities directly in the shared libraries, making it easier to test. Most of the work needed to get these tests built on Windows was taken care of by the effort to build DLLs on Windows. However, there was a small but important piece remaining that posed a challenge.

Being a networked system, Qpid tests need to be sure it correctly handles situations where the network or the network peer fails or acts in some unexpected way. The Qpid unit tests have a useful little class named SocketProxy which sits between the client and broker. SocketProxy relays network traffic in each direction but can also be told to drop pieces of traffic in one or both directions, and can be instructed to drop the socket in one or both directions. Getting this SocketProxy class to run on Windows was a challenge. SocketProxy uses the Qpid common Poller class to know when network data is available in one or both directions, then directly performs the socket recv() and send() as needed. This use of Poller, ironically, was what caused me problems. Although the Windows port includes an implementation of Poller, it doesn’t work in the same fashion as the Linux implementation.

In Qpid proper, the Poller class is designed to work in concert with the AsynchIO class; Poller detects and multiplexes events and AsynchIO performs I/O. The upper level frame handling in Qpid interacts primarily with the AsynchIO class. Below that interface there’s a bit of difference from Linux to Windows. On Linux, Poller indicates when a socket is ready, then AsynchIO performs theĀ  I/O and hands the data up to the next layer. However, the Windows port uses overlapped I/O and an I/O completion port; AsynchIO initiates I/O, Poller indicates completions (rather than I/O ready-to-start), and AsynchIO gets control to hand the resulting data to the next layer. So, the interface between the frame handling and I/O layers in Qpid is the same for all platforms, but the way that Poller and AsynchIO interact can vary between platforms as needed.

My initial plan for SocketProxy was to take it up a level, abstraction-wise. After all, abstracting away behavior is often a good way to make better use of existing, known-to-work code, and avoid complexities. So my first approach was to replace SocketProxy’s direct event-handling code and socket send/recv operations with use of the AsynchIO and Poller combination that is used in Qpid proper.

The AsynchIO-Poller arrangement’s design and place in Qpid involves some dynamic allocation and release of memory related to sockets, and a nice mechanism to do orderly cleanup of sockets regardless of which end initiates the socket close. Ironically, it is this nice cleanup arrangement which tanked its use in the SocketProxy case. Recall that SocketProxy’s usefulness is its ability to interrupt sockets in messy ways, but not be messy itself in terms of leaking handles and memory. My efforts to get AsynchIO and Poller going in SocketProxy resulted in memory leaks, sockets not getting interrupted as abruptly as needed for the test, and connections not getting closed properly. It was a mess.

The solution? Rather than go up a level of abstraction, go down. Use the least common denominator for what’s needed in a very limited use case. I used select() and fd_set. This is just what I advise customers not to do. Did I lose my mind? Sell out to time pressure? No. In this case, using less abstraction was the correct approach – I just didn’t recognize it immediately.

So what made this situation different from “normal”? Why was it a proper place to use less abstraction?

  • The use case is odd. Poller and AsynchIO are very well designed for running the I/O activities in Qpid, correctly handling all socket activity quickly and efficiently. They’re not designed to force failures, and that’s what was needed. It makes no sense to redesign foundational classes in order to make a test harness more elegant.
  • The use is out of the way. It’s a test harness, not the code that has to be maintained and relied on for efficient, correct performance in deployed environments.
  • It’s needs are limited and isolated. SocketProxy handles only two sockets at a time. Performance is not an issue.

Sometimes less is more – it works in abstractions too. The key is to know when it really is best.

Lessons Learned Converting Apache Qpid to Build DLLs on Windows

March 12, 2009

During fall and winter 2008 I worked on the initial port of Apache Qpid to Windows (I blogged about this in January 2009, here). The central areas of Qpid (broker, client) are arranged as a set of shared libraries, qpidclient (the client API), qpidbroker (the guts of the broker), and qpidcommon (supporting code shared by other parts of Qpid). As I mentioned in my previous blog entry, I built the libraries as static libraries instead of shared libraries (“DLLs” in Windows) primarily due to time constraints. Changing the build to produce DLLs for Windows is one of the items I noted that should be done in the future. The primary reasons for building as DLLs are:

  • Probably reduces memory load, and enables easier maintenance of the product. These are standard drivers for using shared libraries in general.
  • Enables expanded use of Qpid in other use cases where dynamic loading is required, such as plug-in environments and dynamically assembled components.
  • Allows the full regression test suite to run on Windows more easily; many of the broker-side unit tests link the qpidbroker library, something I didn’t account for in the initial port.

So, there was clearly a need for switching to DLLs. A couple of other companies saw the value in using DLLs as well: Microsoft, which sponsored my efforts to do the DLL conversion (and some other things coming up), and WSO2, whose Danushka Menikkumbura developed many of the patches for Qpid that went into this project.

Unlike building shared libraries on UNIX/Linux (or most any other OS), it is necessary for the developer to annotate source code to tell the Windows compiler/linker which entrypoints in the library should be “exported”, or made known to external users of the DLL. Conversely, the same entrypoints must be marked “imported” for consumers of the library. The standard way to do this is to define a macro, for example, QPID_CLIENT_EXTERN, that is defined as __declspec(dllexport) when building a DLL and as __declspec(dllimport) when using that DLL. Typically this is accomplished in a header file, such as this example from the Qpid client source:

#if defined(WIN32) && !defined(QPID_DECLARE_STATIC)
#if defined(CLIENT_EXPORT)
#define QPID_CLIENT_EXTERN __declspec(dllexport)
#define QPID_CLIENT_EXTERN __declspec(dllimport)

Then the class or method, etc. that should be exported is annotated with QPID_CLIENT_EXTERN. When developing C++ code, the most direct way to make classes and their methods available is to mark the class as exported:


This works well in many cases (this is how ACE does nearly all of its exporting, for example). However, in Qpid this technique quickly hit some problems. The reason is that if a particular class is exported, all of its ancestors have to be exported from DLLs as well (and their ancestors, and so on) as well as any non-POD types exposed in the method interfaces or used as static members of the class that’s exported. Given Qpid’s layers of inheritance and reuse, only annotating classes that users should be using was not going to fly with a simple export-the-classes scheme. Fortunately Danushka was ahead of the curve here and prepared patches to export only necessary methods. For example:

class MyClass {
    QPID_CLIENT_EXTERN int myMethod();
    MyOtherClass data;

Note that now the class is not exported as a whole, but all the methods needed to use it are. In particular, note that MyOtherClass need not be exported. The obvious trade-off here is that we avoid the issues with inheritance hierarchy needing to be exported, but each user-required method must be annotated explicitly. This is the way we primarily did the Qpid DLL work.

In summary, the lesson learned is that exporting individual class members is more tedious to implement, but provides a measure of independence and flexibility in class design and reuse. I originally looked dubiously at the technique of marking individual members, but Danushka’s ideas here ended up making the end result better and easier for users to apply in new projects. The forthcoming M5 release will contain these new improvements to Apache Qpid.

Things I Like About Working on Apache Qpid

March 4, 2009

Qpid logo

I’ve mentioned before that I’ve been working on the Apache Qpid project, particularly on its port to Windows, first available in the M4 release. I also work on other open source projects related to networked applications programming (primarily, ACE). Since no two customers’ needs are the same, it pays to have multiple tools in order to get the best solution for the situation.

Although Qpid graduated as a Apache top-level project (TLP) some time ago, Apache issued a press release about it this week. As I was reading the release, I was reminded of some of the reasons I really enjoy working with the Apache Qpid team.

  • Meritocracy: the Apache way of working forces contributors to demonstrate their skills and desire to contribute over time before being admitted to the set of people who can commit to the source code repository. Prior to this step, code must be vetted by more senior members who assist with integrating and testing as well as helping newcomers along on the path the committership.
  • Intellectual property rights handling: Lawyers get the sweats when dealing with some open source projects because of the fear of intellectual property rights issues which may arise down the road. Apache has put a very nice system in place for ensuring that contributions to the project have all intellectual property rights properly assigned so there are no issues that users need to worry about.
  • Quality: I’ve been very impressed by the experience, skill, and professionalism of the Apache Qpid development and project team. I’m proud to be a part of this team and they inspire me daily to reach higher and make Qpid the best it can be.

I’m pleased to have customers that see the value in sponsoring work on Qpid because the resulting product provides fantastic enterprise messaging functionality without the exorbitant enterprise messaging price tag. I’m currently working to improve the Qpid user experience for Windows developers as well as reworking the build system to make it easier to keep the Windows and Linux/UNIX builds in synch. Much of the Windows improvements (build the libraries as DLLs, produce an installable prebuilt kit) will be available at the next release (M5) in a month or so. The build improvements will get worked into the development stream after M5.