When is it ok to use ACE Proactor on Linux?

The ACE Proactor framework (see C++NPv2 chapter 8, APG chapter 8 ) allows multiple I/O operations to be initiated and completed by a single thread. The idea is a good one, allowing a small number of threads to execute more I/O operations than could be done synchronously since the OS handles the actual transfers in the background. Many Windows programmers use this paradigm with overlapped I/O very effectively.

The overlapped I/O facility is used by ACE Proactor on Windows, and when the time comes for many to port their ACE-based application to Linux (or Solaris, or HP-UX, or AIX, or…) they naturally gravitate toward carrying the Proactor model to Linux. Seems safe, since Linux offers the aio facility, so off they go.

And then it happens. I/O locks up and all progress stops. Why? Because the aio facility upon which ACE Proactor builds is very restricted for socket I/O on Linux (at least through the Linuxes I’ve worked on). The issue is that the I/O operations initiated using aio from the application are silently converted to synchronous and executed in order based on the handle used. To see why this is a problem, consider the following common asynch I/O idiom:

1. Open socket
2. Initiate a read (whether expecting data or desiring to sense a broken connection)
3. Initiate a write that should immediately complete

When the aio operations are converted to synchronous, the read is executed first, in a blocking manner. The write (which really has data to send) will not execute until the read completes. Consider the situation if the peer is waiting for an initial protocol exchange before sending any data. The peer is waiting for the local end to send data, but the local end’s data won’t actually send until the peer sends. But this will never happen, and we have deadlock.

The only way to make a Proactor-based application work on Linux is to follow a strict lock-step protocol. A ping-pong model, if you will. Each side may only have one operation outstanding on the socket at a time. This is fairly fragile and doesn’t suit many applications well. But if you do have such an application model, you can safely use ACE Proactor on Linux.

Note that the aio facility and, thus, ACE Proactor, on HP-UX, AIX, etc. does not suffer from this “silently converts ops to synchronous” problem, so these restrictions don’t apply there.

For an example of how to program this lock-step protocol arrangement, see the ACE_wrappers/tests/Proactor_Test.cpp program – it has an option to run half-duplex.

Nice new site for help – stackoverflow.com

I recently ran across a new site where one can ask and answer questions about all things related to software development. It’s stackoverflow.com and it’s pretty nice.

The questions I’ve read so far are wide-ranging and the answers thorough and correct. There’s a ranking scale so the most correct and helpful answers bubble up to the top and are clearly marked. In addition to strictly technical questions there are also questions about how to do one’s job better, conditions at work, etc.

Since the site is wide open to all software development, it’s dominated by Windows-type things and web development. For those of us like me who are usually interested in networked programming, there are tags for networked, tcp/ip, etc. and there is a level of help there.

The site allows one to post and answer questions without paying a fee to join, so is ad-supported. I think this is fine, as the ads are unintrusive and the wealth of advice and experience there is well worth it. I heartily recommend stackoverflow.com.

What’s “Networked Programming” all about?

When I’m asked what type of work I do, I often seem to grab for the just the right terms to describe it. But it’s a blind spot for me, I guess. I have been writing network protocol software and networked applications for over 25 years, am considered a network programming expert, and have co-authored three books on the subject, but am not real big on buzzwords. When I mention I write software to make networks more useful, people assume it’s a web type of thing.

Actually, I do networked applications and systems involving pretty much anything except the web. When I started doing this, I actually used serial lines and modems. DECnet, ring-net, and I helped implement the TCP/IP stack (twice) back when you needed US DoD permission to connect to the Internet. Although TCP/IP (and it’s assorted related protocols) drive the Internet today, TCP/IP is used in many applications that don’t touch “the Net”. Medical devices, automobiles, cell phones, industrial processes… practically anything involving more than one computer that needs to talk is what I put in the category of “networked application.”

Some people think it odd that I can specialize in such an area. After all, once you get some piece of software running in one computer, it’s pretty straight-forward to talk to another right? Aren’t there standards for that sort of thing? Well, yes there are. And the nice thing about them is that there are so many to choose from. And that’s just in the “plumbing” – once you put a network between two pieces of your system, the number of issues to be aware of and be able to work with explodes. Timing, byte orders, rogue data attacks, accidental complexities… the list goes on and on. And that’s where I come in – my job is to keep these issues from derailing projects, their schedules, and the jobs that depend on them. I love this stuff…

So the major purpose of this blog is to discuss issues related to networked programming and how to do it better. I hope you’ll join in and share your experiences too.

And if you are a buzzword-literate person and have a moment, do you have a better term for this than “networked applications”?