The CompletionPort parameter represents the completion port to wait on. The
lpNumberOfBytesTransferred parameter receives the number of bytes transferred after a completed I/O operation, such as WSASend or WSARecv. The lpCompletionKey parameter returns per-handle data for the socket that was originally passed into the CreateIoCompletionPort function. As we already mentioned, we recommend saving the socket handle in this key. The lpOverlapped parameter receives the WSAOVERLAPPED structure of the completed I/O operation. This is actually an important
parameter because it can be used to retrieve per I/O–operation data, which we will describe shortly.
The final parameter, dwMilliseconds, specifies the number of milliseconds that the caller is willing to wait for a completion packet to appear on the completion port. If you specify INFINITE, the call waits forever.
Per-handle Data and Per-I/O Operation Data
When a worker thread receives I/O completion notification from the GetQueuedCompletionStatus API call, the lpCompletionKey and lpOverlapped parameters contain socket information that can be used to continue processing I/O on a socket through the completion port. Two types of important socket data are available through these parameters: per-handle data and per-I/O operation data.
The lpCompletionKey parameter contains what we call per-handle data because the data is related to a socket handle when a socket is first associated with the completion port. This is the data that is passed as the CompletionKey parameter of the CreateIoCompletionPort API call. As we noted earlier, your application can pass any type of socket information through this parameter. Typically, applications will store the socket handle related to the I/O request here.
The lpOverlapped parameter contains an OVERLAPPED structure followed by what we call per-I/O operation data, which is anything that your worker thread will need to know when processing a
completion packet (echo the data back, accept the connection, post another read, and so on). Per-I/O operation data is any number of bytes contained in a structure also containing an OVERLAPPED structure that you pass into a function that expects an OVERLAPPED structure. A simple way to make this work is to define a structure and place an OVERLAPPED structure as a field of the new structure.
For example, we declare the following data structure to manage per-I/O operation data:
typedef struct {
OVERLAPPED Overlapped;
char Buffer[DATA_BUFSIZE];
int BufferLen;
int OperationType;
} PER_IO_DATA;
This structure demonstrates some important data elements you might want to relate to an I/O operation, such as the type of I/O operation (a send or receive request) that just completed. In this structure, we consider the data buffer for the completed I/O operation to be useful. To call a Winsock API function that expects an OVERLAPPED structure, you dereference the OVERLAPPED element of your structure. For example,
PER_IO_OPERATION_DATA PerIoData;
WSABUF wbuf;
DWORD Bytes, Flags;
// Initialize wbuf ...
WSARecv(socket, &wbuf, 1, &Bytes, &Flags, &(PerIoData.Overlapped), NULL);
Later in the worker thread, GetQueuedCompletionStatus returns with an overlapped structure and completion key. To retrieve the per-I/O data the macro CONTAINING_RECORD should be used. For example,
This macro should be used; otherwise, the OVERLAPPED member of the PER_IO_DATA structure would always have to appear first, which can be a dangerous assumption to make (especially with multiple developers working on the same code).
You can determine which operation was posted on this handle by using a field of the per-I/O structure to indicate the type of operation posted. In our example, the OperationType member would be set to indicate a read, write, etc., operation. One of the biggest benefits of per-I/O operation data is that it allows you to manage multiple I/O operations (such as read/write, multiple reads, and multiple writes) on the same handle. You might ask why you would want to post more than one I/O operation at a time on a socket. The answer is scalability. For example, if you have a multiple-processor machine with a worker thread using each processor, you could potentially have several processors sending and receiving data on a socket at the same time.
Before continuing, there is one other important aspect about Windows completion ports that needs to be stressed. All overlapped operations are guaranteed to be executed in the order that the application issued them. However, the completion notifications returned from a completion port are not
guaranteed to be in that same order. That is, if an application posts two overlapped WSARecv operations, one with a 10 KB buffer and the next with a 12 KB buffer, the 10 KB buffer is filled first, followed by the 12 KB buffer. The application's worker thread may receive notification from
GetQueuedCompletionStatus for the 12 KB WSARecv before the completion event for the 10 KB operation. Of course, this is only an issue when multiple operations are posted on a socket.
To complete this simple echo server sample, we need to supply a ServerWorkerThread function. The following code outlines how to develop a worker thread routine that uses per-handle data and per-I/O operation data to service I/O requests.
DWORD WINAPI ServerWorkerThread(
LPVOID CompletionPortID) {
HANDLE CompletionPort = (HANDLE) CompletionPortID;
DWORD BytesTransferred;
LPOVERLAPPED Overlapped;
LPPER_HANDLE_DATA PerHandleData;
LPPER_IO_DATA PerIoData;
DWORD SendBytes, RecvBytes;
DWORD Flags;
ret = GetQueuedCompletionStatus(CompletionPort, &BytesTransferred,(LPDWORD)&PerHandleData, (LPOVERLAPPED *) &PerIoData, INFINITE);
closesocket(PerHandleData->Socket);
GlobalFree(PerHandleData);
GlobalFree(PerIoData);
continue;
}
// Service the completed I/O request. You can // determine which I/O request has just // completed by looking at the OperationType // field contained in the per-I/O operation data.
if (PerIoData->OperationType == RECV_POSTED) {
// Do something with the received data // in PerIoData->Buffer
}
// Post another WSASend or WSARecv operation.
// As an example, we will post another WSARecv() // I/O operation.
Flags = 0;
// Set up the per-I/O operation data for the next // overlapped call
ZeroMemory(&(PerIoData->Overlapped), sizeof(OVERLAPPED));
PerIoData->DataBuf.len = DATA_BUFSIZE;
PerIoData->DataBuf.buf = PerIoData->Buffer;
PerIoData->OperationType = RECV_POSTED;
WSARecv(PerHandleData->Socket, &(PerIoData->DataBuf), 1, &RecvBytes, &Flags, &(PerIoData->Overlapped), NULL);
} }
If an error has occurred for a given overlapped operation, GetQueuedCompletionStatus will return FALSE. Because completion ports are a Windows I/O construct, if you call GetLastError or
WSAGetLastError, the error code is likely to be a Windows error code and not a Winsock error code.
To retrieve the equivalent Winsock error code, WSAGetOverlappedResult can be called specifying the socket handle and WSAOVERLAPPED structure for the completed operation, after which
WSAGetLastError will return the translated Winsock error code.
One final detail not outlined in the last two examples we have presented is how to properly close an I/O completion port—especially if you have one or more threads in progress performing I/O on several
sockets. The main thing to avoid is freeing an OVERLAPPED structure when an overlapped I/O operation is in progress. The best way to prevent this is to call closesocket on every socket
handle—any overlapped I/O operations pending will complete. Once all socket handles are closed, you need to terminate all worker threads on the completion port. This can be accomplished by sending a special completion packet to each worker thread using the PostQueuedCompletionStatus function, which informs each thread to exit immediately. PostQueuedCompletionStatus is defined as
BOOL PostQueuedCompletionStatus(
HANDLE CompletionPort,
DWORD dwNumberOfBytesTransferred, ULONG_PTR dwCompletionKey, LPOVERLAPPED lpOverlapped );
The CompletionPort parameter represents the completion port object to which you want to send a completion packet. The dwNumberOfBytesTransferred, dwCompletionKey, and lpOverlapped parameters each will allow you to specify a value that will be sent directly to the corresponding parameter of the GetQueuedCompletionStatus function. Thus, when a worker thread receives the three passed parameters of GetQueuedCompletionStatus, it can determine when it should exit based on a special value set in one of the three parameters. For example, you could pass the value 0 in the dwCompletionKey parameter, which a worker thread could interpret as an instruction to terminate.
Once all the worker threads are closed, you can close the completion port using the CloseHandle function and finally exit your program safely.
The completion port I/O model is by far the best in terms of performance and scalability. There are no limitations to the number of sockets that may be associated with a completion port and only a small number of threads are required to service the completed I/O. For more information on using completion ports to develop scalable, high-performance servers, see Chapter 6.