When is it ok to use ACE Proactor on Linux?

The ACE Proactor framework (see C++NPv2 chapter 8, APG chapter 8 ) allows multiple I/O operations to be initiated and completed by a single thread. The idea is a good one, allowing a small number of threads to execute more I/O operations than could be done synchronously since the OS handles the actual transfers in the background. Many Windows programmers use this paradigm with overlapped I/O very effectively.

The overlapped I/O facility is used by ACE Proactor on Windows, and when the time comes for many to port their ACE-based application to Linux (or Solaris, or HP-UX, or AIX, or…) they naturally gravitate toward carrying the Proactor model to Linux. Seems safe, since Linux offers the aio facility, so off they go.

And then it happens. I/O locks up and all progress stops. Why? Because the aio facility upon which ACE Proactor builds is very restricted for socket I/O on Linux (at least through the Linuxes I’ve worked on). The issue is that the I/O operations initiated using aio from the application are silently converted to synchronous and executed in order based on the handle used. To see why this is a problem, consider the following common asynch I/O idiom:

  1. Open socket
  2. Initiate a read (whether expecting data or desiring to sense a broken connection)
  3. Initiate a write that should immediately complete

When the aio operations are converted to synchronous, the read is executed first, in a blocking manner. The write (which really has data to send) will not execute until the read completes. Consider the situation if the peer is waiting for an initial protocol exchange before sending any data. The peer is waiting for the local end to send data, but the local end’s data won’t actually send until the peer sends. But this will never happen, and we have deadlock.

The only way to make a Proactor-based application work on Linux is to follow a strict lock-step protocol. A ping-pong model, if you will. Each side may only have one operation outstanding on the socket at a time. This is fairly fragile and doesn’t suit many applications well. But if you do have such an application model, you can safely use ACE Proactor on Linux.

Note that the aio facility and, thus, ACE Proactor, on HP-UX, AIX, etc. does not suffer from this “silently converts ops to synchronous” problem, so these restrictions don’t apply there.

For an example of how to program this lock-step protocol arrangement, see the ACE_wrappers/tests/Proactor_Test.cpp program – it has an option to run half-duplex.

Advertisements

Tags:

8 Responses to “When is it ok to use ACE Proactor on Linux?”

  1. Analysis of ACE_Proactor Shortcomings on UNIX « Steve Huston’s Networked Programming Blog Says:

    […] I’ve previously discussed use of ACE_Proactor on Linux (https://stevehuston.wordpress.com/2008/11/25/when-is-it-ok-to-use-ace-proactor-on-linux/) the issues on HP-UX are of a different sort. If the previously discussed Linux aio issues are […]

  2. Gokul Says:

    Is it safe to user ACE Proactor in Solaris? I think there is a separate implementation in ACE to take advantage of the Asynchronous I/O implementation in Solaris.

    • stevehuston Says:

      Yes, Gokul, you are right. Solaris is better for two reasons: it doesn’t suffer from the same “silently convert to synch” problem as Linux does, and there is a separate Proactor implementation on Solaris that takes advantage of Solaris’s features better.

  3. selfredemptiontion Says:

    is linux even now not suitable for proactor? why no bright guys invent a IOCP-like mechanism on linux? while hardware and logic are the same, can epoll work as effectively as iocp ?

    • stevehuston Says:

      I just did a quick check on RHEL 5 and 6 and things look better on first check. I’ll look into it further when I get some free time!

  4. Kedar Says:

    Hello Steve,

    Per your observations, does the Linux use of AIO still require interleaved reads/writes? Is there any way to get around the constraint? For instance, if the reads are handled via reactor and writes via proactor (using AIO)?

    Regards,
    Kedar

    • stevehuston Says:

      Hi Kedar… yes, as far as I’ve seen the asynch operations are still handled the same way. You could split them as you suggest, but once you put the reactor back in, why use the proactor?
      I have been recommending use of ACE_Dev_Poll_Reactor on Linux instead of ACE_Proactor – that usually scales very well and performs well.
      -Steve

  5. When to use ACE Proactor « Touché Says:

    […] From : https://stevehuston.wordpress.com/2008/11/25/when-is-it-ok-to-use-ace-proactor-on-linux/ […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: