VMI 2 Send/Receive Tutorial

This document describes a simple send/receive program for VMI 2. Through this tutorial, a programmer will learn some background about general VMI concepts as well as the user-visible API functions necessary for opening and accepting a connection and for sending and receiving message data.

The following files are needed for this tutorial:

Makefile - Makefile to build both sender and receiver
vmisend.h and vmisend.c - Source code for sender
vmirecv.h and vmirecv.c - Source code for receiver

An archive of the necessary files may be downloaded here.

1. Background

VMI is a middleware layer that transparently aggregates multiple lower-level communication interfaces with the goal of enabling binary portability of applications over any of the lower-level interfaces supported by VMI. An application compiled and linked to the VMI libraries can use different communication interfaces simply by changing a configuration file before the application is launched. During initialization, VMI dynamically loads modules ("devices") that allow VMI to talk to the desired communication interfaces. In addition to devices that talk directly to communication interfaces, other devices can filter communication data. That is, these devices receive communication data, examine and manipulate it in some way, and then pass the communication data on to another device.

VMI devices are linked together in chains. VMI initially defines two chains, a send chain and a receive chain, although additional chains can be created dynamically. When a message is sent, each device on the send chain has an opportunity to modify the message data before passing it to the next device on the chain and finally into the network. Similarly, when a message is received, each device on the receive chain has an opportunity to modify the message data before passing it to the next device on the chain and finally to the application.

Message and control instructions are passed between devices on a chain using I/O Request Blocks (IRBs). Each IRB contains a stack, a status field used for return values, the connection that the IRB is associated with, and other miscellaneous pieces of state. As an IRB travels through the devices on a chain, the IRB's stack encapsulates the state for the IRB on each device. This per-device IRB state includes the message data in a field called a slab, input and output arguments to the device, and a pointer to a completion function to call when the IRB is completed (along with a pointer to an arbitrary context to pass to this function). When an IRB reaches the last device on the send chain, the device may either complete the IRB by processing it immediately or may pend the IRB if there is some long processing delay expected. The device completes pended IRBs asynchronously as quickly as possible. When an IRB completes (either immediately or after being pended), the completion function for the IRB is invoked.

VMI handles the details of passing IRBs through the devices on a chain and presents a simple but thorough API to the programmer. Generally, a programmer using the user-visible VMI API needs to deal with two primary data structures.

Buffers -- Most messaging layers require a contiguous block of "registered" memory that is used for communication operations. This registered memory is usually pinned into physical memory. VMI contains calls to register and deregister buffers, and these calls are forwarded to the appropriate devices as needed.

Generally, a programmer deals with buffers (VMI_BUFFER) or pointers to buffers (PVMI_BUFFER) when sending messages in VMI. The programmer calls VMI_Buffer_Register() to register a buffer for message data and then calls additional API functions to send the contents of the registered buffer through a connection.

Buffer Ops and Slabs -- Because buffer registration is usually an expensive operation, registering every piece of communication data individually is undesirable. VMI solves this problem by defining a structure called a buffer op which represents a contiguous block of memory within a buffer. A slab contains multiple buffer ops and is used to represent data that is non-contiguous. VMI can reduce memory copies and increase performance by employing scatter/gather operations that add and remove buffer ops to and from slabs.

VMI gives the programmer a "streaming interface" abstraction as a way of allowing the programmer to easily write high performance code. A VMI stream can be thought of as a uni-directional point-to-point path through which slabs flow. A programmer can process these slabs as they arrive individually without waiting for an entire message to arrive before delivering it.

VMI provides the ability to "grab" a slab when it is received. If the application is not ready to process the data in the slab, the application can request ownership of the slab from VMI. VMI is then prevented from reclaiming the memory allocated to the slab. Later, when the application has the resources available to process the slab, it receives the data from the slab and then informs VMI that the slab may be released and the memory for the slab deallocated. In this way, an application does not need to spend time buffering data from the slab if resources are not immediately available to handle the data.

Generally, a programmer deals with slabs and buffer ops when receiving messages in VMI. The programmer registers a message handler that VMI invokes asynchronously whenever a slab is received from the network. The slab contains one or more buffer ops that the programmer can read out of the slab and handle immediately. Alternatively, the programmer may grab the slab and read the buffer ops at a later time.

2. Source Code

This section of the tutorial examines the source code for a simple application that uses VMI to send a message between a sender process and a receiver process. The concepts described in the Background section above are reflected in the source code below.

2.1. Sender

The sender operates by opening a connection to the receiver, creating a message to send to the receiver, and finally sending the message to the receiver. The discussion below illustrates the main points in the sender source code.

The sender begins by initializing VMI as shown in the code fragement below. All processes that use VMI must initialize it with a call to VMI_Init(). This call, like most calls in the VMI API, returns a VMI_STATUS return code. The macro VMI_SUCCESS() can be used to ensure that a success status was returned, and the program displays and error message and exits if not.

  /* Initialize VMI. */
  status = VMI_Init (argc, argv);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Init()", status);
  }

After VMI is initialized, the sender calls openConnection() to open a connection to the receiver. (The details of openConnection() are described later.) The important point to understand is that when a connection is opened in VMI, the connection process is first initiated and then completes asynchronously at a later time. The code in the fragment below loops until the connection open completes. Inside the loop, calls are made to VMI_Poll(). This API call allows VMI to make progress in sending and receiving data and in opening and completing connection requests.

  /* Connect to our peer. */
  openConnection (remoteKey, remoteHost);
  while (!connected) {
    status = VMI_Poll();
    if (!VMI_SUCCESS (status)) {
      VMI_perror("VMI_Poll()", status);
    }
  }

A fragment of code from openConnection() is shown below. The critical points are the call to VMI_Connection_Create() which creates the initial connection object, the calls to VMI_Connection_Allocate_IPV4_Address() and VMI_Connection_Bind() to bind the connection to the receiver's network address, and the call to VMI_Connection_Issue() to initiate the connection process. As described previously, the connection operation will complete asynchronously at a later time.

An optional feature of the connection process is the "connect data". The feature allows the programmer to send a buffer of data to the receiver as part of the connection process. The receiver can use this connect data for several purposes such as deciding whether to accept or reject the connection.

  /* Get a connection handle. */
  status = VMI_Connection_Create (&connection);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Connection_Create()", status);
  }

  /* Get our username. */
  if (!getpwuid (getuid())) {
    fprintf (stderr, "Unable to get username.\n");
    exit (1);
  }
  userName = getpwuid(getuid())->pw_name;

  /* Allocate a remote IPv4 NETADDRESS. */
  status = VMI_Connection_Allocate_IPV4_Address (remoteHost,
                                                 0,
                                                 userName,
                                                 remoteKey,
                                                 &remoteAddress);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Connection_Allocate_IPV4_Address()", status);
  }

  /* Now bind the local and remote addresses. */
  status = VMI_Connection_Bind (*localAddress, *remoteAddress, connection);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Connection_Bind()", status);
  }

#ifdef CONNECT_DATA
  /* Allocate and initialize some connection data. */
  status = VMI_Buffer_Allocate (sizeof (CONNECT_MESSAGE), &conn_data_buffer);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Buffer_Allocate()", status);
  }
  conn_data = (CONNECT_MESSAGE *) VMI_BUFFER_ADDRESS (conn_data_buffer);
  conn_data->cookie = htonl (42);

  /* Establish a connection. */
  status = VMI_Connection_Issue (connection,
				 conn_data_buffer,
                                 processConnectResponse,
				 (PVOID) conn_data_buffer);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Connection_Issue()", status);
  }
#else
  /* Establish a connection. */
  status = VMI_Connection_Issue (connection, NULL,
                                 processConnectResponse, NULL);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Connection_Issue()", status);
  }
#endif

In the call to VMI_Connection_Issue() above, a reference is made to a function processConnectResponse(). Connection requests are completed asynchronously, and when the connection completes VMI invokes the completion handler specified in the issue request. The code fragment below shows the processing that takes place in processConnectResponse(). One of the parameters passed to the function by VMI is a status variable that can be examined to determine whether the connection was accepted by the receiver, rejected by the receiver, or whether some kind of error condition was discovered. If the connection was accepted, we set the global variable "connected" to allow the busy-loop in main() to continue.

  switch (status)
  {
    case VMI_CONNECT_RESPONSE_ACCEPT:
      DEBUG_PRINT ("Our peer accepted our connect request.\n");
      connected = TRUE;
      break;

    case VMI_CONNECT_RESPONSE_REJECT:
      fprintf (stderr, "Our peer rejected our connect request.\n");
      exit (1);
      break;

    default:
      fprintf (stderr, "Error occurred while establishing connection.\n");
      exit (1);
      break;
  }

  /* Free our resources. */
  if (context) {
    (void) VMI_Buffer_Deallocate ((PVMI_BUFFER) context);
  }

The following code fragment is from the function createMessage(). It creates a message of a defined size that can be sent to the receiver. The key thing to see in this function is the call to VMI_Buffer_Register() which takes the pointer to the memory holding the message and returns a PVMI_BUFFER pointing to a registered buffer that can be used to send the message.

  *msg = malloc (msgSize);
  if (*msg == NULL) {
    fprintf (stderr, "malloc() failed\n");
    exit (1);
  }

  msgPtr = (PUCHAR) *msg;
  bytesRemaining = msgSize;
  iter = 0;
  while (bytesRemaining > 0) {
    bytesWritten = snprintf (msgPtr, bytesRemaining,
        "All work and no play makes Jack a dull boy #%lu.  ", ++iter);
    if (bytesWritten <= 0) {
      break;
    }
    msgPtr += bytesWritten;
    bytesRemaining -= bytesWritten;
  }
  ((PUCHAR)msg)[msgSize-1] = '\0';

  status = VMI_Buffer_Register (*msg, msgSize, msgBuffer);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Buffer_Register()", status);
  }

Finally, the message is sent in function sendMessage(). As shown in the code fragment below, this involves creating a new stream to the receiver by calling VMI_Stream_Begin(). At a minimum, a call to VMI_Stream_End() is necessary to send at least one message fragment on the stream. Additional fragments may be sent with one or more calls to VMI_Stream_Send_Fragment().

  /* Send the first fragment of a message. */
  status = VMI_Stream_Begin (connection,
                             NULL,
                             NULL,
                             0,
                             NULL,
                             NULL,
                             FALSE,
                             &stream);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Stream_Begin()", status);
  }

  /* Send the message repeatedly as fragments. */
  if (FRAGS_PER_MSG > 1) {
    for (i = 0; i < FRAGS_PER_MSG-1; i++) {          /* -1 for end */
      status = VMI_Stream_Send_Fragment (stream,
                                         msgBuffer,
                                         msg,
                                         msgSize,
                                         FALSE);
      if (!VMI_SUCCESS (status)) {
        VMI_perror ("VMI_Stream_Send_Fragment()", status);
      }
    }
  }

  /* Send the last fragment of the message. */
  status = VMI_Stream_End (stream,
                           msgBuffer,
                           msg,
                           msgSize,
                           FALSE);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Stream_End()", status);
  }

  return status;

2.2. Receiver

The receiver operates by setting a receive handler function in the VMI library, waiting for a connection from the sender, and then waiting for the sender to send the message. The discussion below illustrates the main points in the receiver source code.

Like the sender, the receiver begins by initializing VMI with a call to VMI_Init(). After VMI initializes successfully, the receiver sets a receive handler by calling the macro VMI_STREAM_SET_RECV_FUNCTION(). When VMI receives message data on a stream, it asynchronously invokes the function specified in this macro.

Next, as shown in the following code fragment, the receiver calls VMI_Connection_Accept_Fn() to set a connection handler function within the VMI library. When a new incoming connection is received, VMI asynchronously invokes the connection handler which makes a decision about whether to accept or reject the connection. (Details of the connectAccept() function are given later.) The code loops, calling VMI_Poll() to allow VMI to make progress until the global variable "connected" is set, which happens in the connection handler.

  /* Wait for a connection from our peer. */
  status = VMI_Connection_Accept_Fn (connectAccept);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Connection_Accept_Fn()", status);
  }
  while (!connected) {
    sleep (0);
    status = VMI_Poll();
    if (!VMI_SUCCESS (status)) {
      VMI_perror ("VMI_Poll()", status);
    }
  }

After the connection is open, the receiver simply loops calling VMI_Poll() until the global variable "finished" is set to TRUE, as shown in the code fragment below.

  /* Poll VMI to receive the message. */
  while (!finished) {
    status = VMI_Poll();
    if (!VMI_SUCCESS (status)) {
      VMI_perror ("VMI_Poll()", status);
    }
  }

The connection handler function connectAccept() is shown in the code fragment below. The critical portions of the code are when the newConnection argument is saved in the global variable "connection" for use in other portions of the receiver program, and when the global variable "connected" is set to signal the main program that the connection is open. Also, the function uses VMI_CONNECT_RESPONSE_ACCEPT as the return code to tell VMI that it wishes to accept the connection. In this simple example, the sender's connection is always accepted, but we could instead make some sort of decision about whether to accept the connection or not and return VMI_CONNECT_RESPONSE_REJECT instead to reject the connection.

As in the sender, several lines of code are used to process "connect data" if desired.

#ifdef CONNECT_DATA
  /* Read the connection data. */
  if (size != bytesRemaining) {
    return VMI_CONNECT_RESPONSE_ERROR;
  }
  status = VMI_Slab_Save_State (slab, &slabState);
  if (!VMI_SUCCESS (status)) {
    DEBUG_PRINT ("VMI_Slab_Save_State() returned error code 0x%08X.\n",
                 status);
    return VMI_CONNECT_RESPONSE_ERROR;
  }
  status = VMI_Slab_Read (slab,
                          &bytesRemaining,
                          (PVOID) &conn_data);
  if (!VMI_SUCCESS (status)) {
    DEBUG_PRINT ("VMI_Slab_Read() returned error code 0x%08X.\n", status);
    return VMI_CONNECT_RESPONSE_ERROR;
  }
  status = VMI_Slab_Restore_State (slab, slabState);
  if (!VMI_SUCCESS (status)) {
    return VMI_CONNECT_RESPONSE_ERROR;
  }
  if (bytesRemaining) {
    DEBUG_PRINT ("%lu leftover bytes were found.\n", bytesRemaining);
    return VMI_CONNECT_RESPONSE_ERROR;
  }

  /* At this point, we could read conn_data->cookie if we cared to. */
#endif

  connection = newConnection;
  connected = TRUE;

  DEBUG_PRINT ("We have accepted our peer's connection.\n");

  return VMI_CONNECT_RESPONSE_ACCEPT;

Finally, the following code fragment from receiveMessage() is where the receiver actually reads the message data. The number of bytes of message data in the slab is obtained from the macro VMI_SLAB_BYTES_REMAINING(). Then the appropriate number of bytes is read from the slab with a call to VMI_Slab_Read() and printed on the display. After the message data is received, the global variable "finished" is set so the main program knows to exit cleanly. Finally, the function calls VMI_SLAB_DONE to signal VMI that the slab may be deallocated. If the application had wanted to retain ownership of the slab, it could have returned VMI_SLAB_GRAB instead to grab the slab. It would then be the applications responsibility to retain a pointer to the slab and call VMI_Release_Slab() when finished with it.

Before the programmer can read from the slab, a call must be made to VMI_Slab_Save_State() to save the contents of the slab stack.

  sz = VMI_SLAB_BYTES_REMAINING (slab);

  /* Save the slab state. */
  status = VMI_Slab_Save_State (slab, &slabState);
  if (!VMI_SUCCESS(status)) {
    VMI_perror ("VMI_Slab_Save_State()", status);
  }

  status = VMI_Slab_Read (slab, &sz, &buffer);
  if (!VMI_SUCCESS (status)) {
    VMI_perror ("VMI_Slab_Save_State()", status);
  }

  printf("%s\n", (PUCHAR) buffer);

  if (command != VMI_STREAM_END) {
    return (VMI_SLAB_DONE);
  }

  DEBUG_PRINT ("Completed receiving message.\n");

  finished = TRUE;

  return VMI_SLAB_DONE;

3. Compiling and Running

To compile the sender and receiver simply issue a "make". Note that you may have to edit the Makefile to update the path for INCLUDE_DIRS and LIB_DIRS at the top of the file to point to the correct locations.

Next, you must configure your environment to include an environment variable VMI_SPECFILE that points to an XML input file describing the VMI devices you wish to use. For example:

  $ export VMI_SPECFILE=/opt/vmi-2.0a-1-gcc/specfiles/gm.xml

Also, unless the VMI shared object libraries are in a well-defined location on your system, you will need to point your LD_LIBRARY_PATH to the proper location where they are installed. For example:

  $ export LD_LIBRARY_PATH=/opt/vmi-2.0a-1-gcc/lib

Once your environment is configured, you first execute the receiver program:

  $ ./vmirecv recvkey

and execute the sender program on a different host:

  $ ./vmisend sendkey recvkey recvhost

If everything works as it should, the receiver will display the message data that it receives from the sender.