Posts Tagged ‘networked’

Revised ACE_Dev_Poll_Reactor Fixes Multithread Issues (and more!) on Linux

June 15, 2009

When ACE 5.7 is released this week it will contain an important fix (a number of them, actually) for use cases that rely on multiple threads running the Reactor event loop concurrently on Linux. The major fix areas involved for ACE_Dev_Poll_Reactor in ACE 5.7 are:

  • Dispatching events from multiple threads concurrently
  • Properly handling changes in handle registration during callbacks
  • Change in suspend/resume behavior to be more ACE_TP_Reactor-like

At the base of these fixes was a foundational change in the way ACE_Dev_Poll_Reactor manages events returned from Linux epoll. Prior to this change, ACE would obtain all ready events from epoll and then each event loop-executing thread in turn would pick the next event from that set and dispatch it. This design was, I suppose, more or less borrowed from the ACE_Select_Reactor event demultiplexing strategy. In that case it made sense since select() is relatively expensive and avoiding repeated scans of all the watched handles is a good thing. Also, the ACE_Select_Reactor (and ACE_TP_Reactor, which inherits from it) have a mechanism to note that something in the handle registrations changed, signifying that select() must be called again. This mechanism was lacking in ACE_Dev_Poll_Reactor.

However, unlike with select(), it’s completely unnecessary to try to avoid calls to epoll_wait(). Epoll is much more scalable than is select(), and letting epoll manage the event queue, only passing back one event at a time, is much simpler than the previous design, and also much easier to get correct. So that was the first change: obtain one event per call to epoll_wait(), letting Linux manage the event queue and weed out events for handles that are closed, etc. The second change was to add the EPOLLONESHOT option bit to the event registration for each handle. The effect of this is that once an event for a particular handle is delivered from epoll_wait(), that handle is effectively suspended. No more events for the handle will be delivered until the handle’s event mask is re-enabled via epoll_ctl(). These two changes were used to fix and extend ACE_Dev_Poll_Reactor as follows.

Dispatching Events from Multiple Threads Concurrently

The main defect in the previous scheme was the possibility that events obtained from epoll_wait() could be delivered to an ACE_Event_Handler object that no longer existed. This was the primary driver for fixing ACE_Dev_Poll_Reactor. However, another less likely, but still possible, situation was that callbacks for a handler could be called out of order, triggering time-sensitive ordering problems that are very difficult to track down. Both these situations are resolved by only obtaining one I/O event per ACE_Reactor::handle_events() iteration. A side-effect of this change is that the concurrency behavior of ACE_Dev_Poll_Reactor changes from being similar to ACE_WFMO_Reactor (simultaneous callbacks to the same handler are possible) to being similar to ACE_TP_Reactor (only one I/O callback for a particular handle at a time). Since epoll’s behavior with respect to when a handle’s availability for more events differs from Windows’s WaitForMultipleObjects, the old multiple-concurrent-calls-per-handle couldn’t be done correctly anyway, so the new ACE_Dev_Poll_Reactor behavior leads to easier coding and programs that are much more likely to be correct when changing reactor use between platforms.

Properly handling changes in handle registration during callbacks

A difficult problem to track down sometimes arose in the previous design when a callback handler changed handle registration. In such a case, if the reactor made a subsequent callback to the original handler (for example, if the callback returned -1 and needed to be removed) the callback may be made to the wrong handler – the new registered handler instead of the originally called handler. This problem was fixed by making some changes and additions to the dispatching data structures and code and is no longer an issue.

Change in suspend/resume behavior to be more ACE_TP_Reactor-like

An important aspect of ACE_TP_Reactor’s ability to support complicated use cases arising in systems such as TAO is that a dispatched I/O handler is suspended around the upcall. This prevents multiple events from being dispatched simultaneously. As previously mentioned, the changes to ACE_Dev_Poll_Reactor also effectively suspend a handler around an upcall. However, a feature once only available with the ACE_TP_Reactor is that an application can specify that the application,  not the ACE reactor, will resume the suspended handler. This capability is important to properly supporting the nested upcall capability in TAO, for example. The revised ACE_Dev_Poll_Reactor now also has this capability. Once the epoll changes were made to effectively suspend a handler around an upcall, taking advantage of the existing suspend-resume setting in ACE_Event_Handler was pretty straight-forward.

So, if you’ve been holding off on using ACE_Dev_Poll_Reactor on Linux because it was unstable with multiple threads, or you didn’t like the concurrency behavior and the instability it may bring, I encourage you to re-evaluate this area when ACE 5.7 is released this week. And if you’ve ever wondered what good professional support services are, I did this work for a support customer who is very happy they didn’t have to pay hourly for this. And many more people will be happy that since I wasn’t billing for time I could freely fix tangential issues not in the original report such as the application-resume feature. Everyone wins: the customer’s problem is resolved and ACE’s overall product quality and functionality are improved. Enjoy!

Things I Like About Working on Apache Qpid

March 4, 2009

Qpid logo

I’ve mentioned before that I’ve been working on the Apache Qpid project, particularly on its port to Windows, first available in the M4 release. I also work on other open source projects related to networked applications programming (primarily, ACE). Since no two customers’ needs are the same, it pays to have multiple tools in order to get the best solution for the situation.

Although Qpid graduated as a Apache top-level project (TLP) some time ago, Apache issued a press release about it this week. As I was reading the release, I was reminded of some of the reasons I really enjoy working with the Apache Qpid team.

  • Meritocracy: the Apache way of working forces contributors to demonstrate their skills and desire to contribute over time before being admitted to the set of people who can commit to the source code repository. Prior to this step, code must be vetted by more senior members who assist with integrating and testing as well as helping newcomers along on the path the committership.
  • Intellectual property rights handling: Lawyers get the sweats when dealing with some open source projects because of the fear of intellectual property rights issues which may arise down the road. Apache has put a very nice system in place for ensuring that contributions to the project have all intellectual property rights properly assigned so there are no issues that users need to worry about.
  • Quality: I’ve been very impressed by the experience, skill, and professionalism of the Apache Qpid development and project team. I’m proud to be a part of this team and they inspire me daily to reach higher and make Qpid the best it can be.

I’m pleased to have customers that see the value in sponsoring work on Qpid because the resulting product provides fantastic enterprise messaging functionality without the exorbitant enterprise messaging price tag. I’m currently working to improve the Qpid user experience for Windows developers as well as reworking the build system to make it easier to keep the Windows and Linux/UNIX builds in synch. Much of the Windows improvements (build the libraries as DLLs, produce an installable prebuilt kit) will be available at the next release (M5) in a month or so. The build improvements will get worked into the development stream after M5.

Analysis of ACE_Proactor Shortcomings on UNIX

January 22, 2009

I’ve been looking into two related issues in the ACE development stream:

  1. SSL_Asynch_Stream_Test times out on HP-UX (I recently made a bunch of fixes to the test itself so it runs as well as can be on Linux, but times out on HP-UX)
  2. Proactor_Test shows a stray, intermittent diagnostic on HP-UX: EINVAL returned from aio_suspend()

Although I’ve previously discussed use of ACE_Proactor on Linux (https://stevehuston.wordpress.com/2008/11/25/when-is-it-ok-to-use-ace-proactor-on-linux/) the issues on HP-UX are of a different sort. If the previously discussed Linux aio issues are resolved inside Linux, the same problem I’m seeing on HP-UX may also arise, but it doesn’t get that far. Also, I suspect that the issues arising from these tests’ execution on Solaris are of the same nature, though the symptoms are a bit different.

The symptoms are that the proactor event loop either fails to detect completions, or it gets random errors that smell like the aiocb list is damaged. I believe I’ve got a decent idea of what’s going on, and it’s basically two issues:

  1. If all of the completion dispatch threads are blocked waiting for completions when new I/O is initiated, the new operation(s) are  not taken into account by the threads waiting for completions. This is basically the case in the SSL_Asynch_Stream_Test timeout on HP-UX – all the completion-detecting threads are already running before any I/O is initiated and no completions are ever detected.
  2. The completion and initiation activities modify the aiocb list used to detect completions directly, without interlocks, and without consideration of what affect it may have (or not) on the threads waiting for completions.

The ACE_Reactor framework uses internal notifications to handle the need to unblock waiting demultiplexing threads so they can re-examine the handle set as needed; something similar is needed for the ACE_Proactor to remedy issue #1 above. There is a notification pipe facility in the proactor code, but I need to see if it can be used in this case. I hope so…

The other problem, of concurrent access to the aiocb list by threads both waiting for completions and modifying the list is a much larger problem. That requires more of a fundamental change in the innards of the POSIX Proactor implementation.

Note that there are a number of POSIX Proactor flavors inside ACE (section 8.5 in C++NPv2 describes most of them). The particular shortcomings I’ve noted here only affect the ACE_POSIX_AIOCB_Proactor and ACE_POSIX_SIG_Proactor, which is largely based on the ACE_POSIX_AIOCB_Proactor. The newest one, ACE_POSIX_CB_Proactor, is much less affected, but is not as widely available.

So, the Proactor situation on UNIX platforms is generally not too good for demanding applications. Again, Proactor on Windows is very good, and recommended for high-performance, highly scalable networked applications. On Linux, stick to ACE_Reactor using the ACE_Dev_Poll_Reactor implementation; on other systems, stick with ACE_Reactor and ACE_Select_Reactor or ACE_TP_Reactor depending on your need for multithreaded dispatching.

There’s No Substitute for Experience with TCP/IP Sockets

December 31, 2008

The number of software development tools and aids available to us as we begin 2009 is staggering. IDEs, code generators, component and class libraries, design and modeling tools, high-level protocols, etc. were just speculation and dreams when I began working with TCP/IP in 1985. TCP and IP were not yet even approved MIL-STDs and the company I worked for had to get US Department of Defense permission to connect to the fledgling Internet. The “Web” was still 10 years away. If you wanted to use TCP/IP for much more than FTP, Telnet, or email you had to write the protocol and the code to run it yourself. The Sockets API was the highest level access we had at the time. That is a whole area of difficulty and complexity in and of itself, which C++ Network Programming addresses. But the API is more of a usage and programming efficiency issue – what I’m talking about today is the necessity of experience and understanding what’s going on between the API and the wire when working with TCP/IP, regardless of the toolkit or language or API you use.

A lot of software running on what many people consider “the net” piggy-backs on HTTP in one form or another. There are also many helpful libraries, such as .NET and ACE, to assist in writing networked applications at a number of levels. More specific problem areas also have very useful targeted solutions, such as message queuing systems and Apache Qpid. And, like most programming tasks, when everything’s ideal, it’s not too hard to get some code running pretty well. It’s when things don’t work as well as you planned that the way forward becomes murky. That’s when experience is needed. These are some examples of issues I’ve assisted people with lately:

  1. Streaming data transfer would periodically stop for 200 msec, then resume
  2. Character strings transferred would intermittently be bunched together or split apart
  3. Asynchronous I/O-based code stopped working when ported to Linux

The tendency when a problem such as this comes up is to find out who, or what, is to blame. In my experience, the first attempt at blame is usually laid on the most recent addition to the programming toolset – the piece trusted the least and that’s usually closest to the application being written. For ACE programs, this is usually why I get involved so early.

I’ve spent many years debugging applications and network protocol code. I spent way too much time trying to blame the layer below me, or the OS, or the hardware. The biggest lesson I learned is that when something goes wrong with code I wrote, it’s usually my problem and it’s usually a matter of some concept or facility I don’t understand enough to see the problem clearly or find the way to a solution. That’s why it’s so important to understand the features and functionality you are making use of – there’s no substitute for experience.

Helping my clients solve the three problems I mentioned above involved experience. Knowing where to target further diagnosis and gathering the right information made the difference between solving the problem that day and staring at code for days wondering what’s going on. Curious about what the problems were?

  1. Slow-start peculiarity on the receiver; disable Nagle’s on the receiving side.
  2. That’s the streaming nature of TCP. Need to mark the string boundaries and check for them on receive.
  3. Linux silently converts asynchronous socket I/O operations to synchronous and executes them in order; need to restructure order of operations in a very restricted way, or switch paradigm on Linux.

Although each client initially targeted blame at the same place, the real solution was in a different layer for each case. And none involved the software initially thought to be at fault.

When you are ready to begin developing a new networked application, or you’re spending yet another night staring at code and network traces, remember: there’s a good chance you need a little more clarity on something. Take a step back, assume the tools you’re using are probably correct, and begin to challenge your assumptions about how you think it’s all supposed to work. A little more understanding and experience will make it clear.

What’s “Networked Programming” all about?

November 15, 2008

When I’m asked what type of work I do, I often seem to grab for the just the right terms to describe it. But it’s a blind spot for me, I guess. I have been writing network protocol software and networked applications for over 25 years, am considered a network programming expert, and have co-authored three books on the subject, but am not real big on buzzwords. When I mention I write software to make networks more useful, people assume it’s a web type of thing.

Actually, I do networked applications and systems involving pretty much anything except the web. When I started doing this, I actually used serial lines and modems. DECnet, ring-net, and I helped implement the TCP/IP stack (twice) back when you needed US DoD permission to connect to the Internet. Although TCP/IP (and it’s assorted related protocols) drive the Internet today, TCP/IP is used in many applications that don’t touch “the Net”. Medical devices, automobiles, cell phones, industrial processes… practically anything involving more than one computer that needs to talk is what I put in the category of “networked application.”

Some people think it odd that I can specialize in such an area. After all, once you get some piece of software running in one computer, it’s pretty straight-forward to talk to another right? Aren’t there standards for that sort of thing? Well, yes there are. And the nice thing about them is that there are so many to choose from. And that’s just in the “plumbing” – once you put a network between two pieces of your system, the number of issues to be aware of and be able to work with explodes. Timing, byte orders, rogue data attacks, accidental complexities… the list goes on and on. And that’s where I come in – my job is to keep these issues from derailing projects, their schedules, and the jobs that depend on them. I love this stuff…

So the major purpose of this blog is to discuss issues related to networked programming and how to do it better. I hope you’ll join in and share your experiences too.

And if you are a buzzword-literate person and have a moment, do you have a better term for this than “networked applications”?