The ACE Proactor framework (see C++NPv2 chapter 8, APG chapter 8 ) allows multiple I/O operations to be initiated and completed by a single thread. The idea is a good one, allowing a small number of threads to execute more I/O operations than could be done synchronously since the OS handles the actual transfers in the background. Many Windows programmers use this paradigm with overlapped I/O very effectively.
The overlapped I/O facility is used by ACE Proactor on Windows, and when the time comes for many to port their ACE-based application to Linux (or Solaris, or HP-UX, or AIX, or…) they naturally gravitate toward carrying the Proactor model to Linux. Seems safe, since Linux offers the aio facility, so off they go.
And then it happens. I/O locks up and all progress stops. Why? Because the aio facility upon which ACE Proactor builds is very restricted for socket I/O on Linux (at least through the Linuxes I’ve worked on). The issue is that the I/O operations initiated using aio from the application are silently converted to synchronous and executed in order based on the handle used. To see why this is a problem, consider the following common asynch I/O idiom:
- Open socket
- Initiate a read (whether expecting data or desiring to sense a broken connection)
- Initiate a write that should immediately complete
When the aio operations are converted to synchronous, the read is executed first, in a blocking manner. The write (which really has data to send) will not execute until the read completes. Consider the situation if the peer is waiting for an initial protocol exchange before sending any data. The peer is waiting for the local end to send data, but the local end’s data won’t actually send until the peer sends. But this will never happen, and we have deadlock.
The only way to make a Proactor-based application work on Linux is to follow a strict lock-step protocol. A ping-pong model, if you will. Each side may only have one operation outstanding on the socket at a time. This is fairly fragile and doesn’t suit many applications well. But if you do have such an application model, you can safely use ACE Proactor on Linux.
Note that the aio facility and, thus, ACE Proactor, on HP-UX, AIX, etc. does not suffer from this “silently converts ops to synchronous” problem, so these restrictions don’t apply there.
For an example of how to program this lock-step protocol arrangement, see the ACE_wrappers/tests/Proactor_Test.cpp program – it has an option to run half-duplex.