Programmer-Visible Performance Tradeoffs

The UDM model and the direct VNI architecture give the programmer a lot of freedom. The UDM model gives each application the illusion of a private network, virtualized resources and control over every logical message passed through the network. In practice, there are two subtle performance issues that the programmer faces when using the direct VNI.

The combination of one-way, asynchronous messages and unlimited buffering means that the problem of flow control has largely been pushed up to the level of the UDM user. This design decision is a benefit – the application does not incur the cost of unnecessary flow control protocol overhead – but also a burden. An application must either limit the flow of messages by construction or make use of a user-level library that includes a flow control protocol.

The message system includes a coarse form of flow control (“overflow control”, described in Chapter 5) but only as a last resort defense against temporarily or permanently runaway applications.

The buffering system breaks deadlock scenarios, but use of buffering incurs performance costs.

For instance, an application is free to attempt to send a message from within an atomic section, but if a deadlock occurs it is detected by a timeout only after some amount of delay. Further, an application may choose to depend on the existence of a fixed amount of buffering, but buffering has a real cost. An astute UDM programmer will depend on the buffering system only in the “optimistic” sense in which is was intended, i.e., only when the chance of an actual deadlock is known to be low.

Effectively, correctness is separated from performance. It is possible to write an application with little attention to the effects of the message system and then concentrate effort on the performance of sections that matter. There are two performance effects to keep in mind. First, the buffered delivery path incurs extra cost. Second, the virtualization of buffering can introduce even higher costs when

Buffering w/paging Buffering

in free frames

(out of swap space) Fast

case

Buffered case

Direct access

Figure 3-5. Direct virtual network interface operating modes as a state diagram. In the fast, common case, at left, message reception is handled entirely in software. Buffering has additional costs which increase as demands for buffer storage space increase.

physical memory resources are low. These effects are described below and quantified in Chapters 4 and 7.

Figure 3-5 summarizes the operating modes of theFUGUsystem as a state diagram model. In the ordinary case, on the left, UDM messages are extracted directly out of the hardware NI at full speed using polling or upcall-based interrupts. Next, buffering, when invoked, adds some cost but operates similarly to a memory-based system as long as buffer space is available in physical memory. Third, virtual buffering system can continue to buffer messages for an application even when physical buffering space is exhausted by paging and buffering simultaneously. This case is quite slow but breaks potential deadlocks arising from dynamic allocation of buffer space. Finally, at the right, it is possible to exhaust the application’s virtual buffer, but only by exhausting the application’s swap space.

The messages to the programmer are two. First, some applications will give better performance if a little protocol overhead is added for the sake of avoiding buffering. We have no direct examples of this effect; all our sample applications (in Section 7.4) avoid buffering naturally. Second, it is possible to write programs that consume even “effectively unlimited” buffers. Like any system that supports unacknowledged messages, theFUGUprogrammer (or library) must either keep the message generation rate below the message consumption rate or provide synchronization to do so in order to avoid filling up the buffer.⁶

3.4 Discussion

This chapter has described the programmability, protection and performance aspects of the direct virtual network interface architecture. The model and protection features are accepted as design choices while the focus is on providing performance in the face of those choices. The two-case delivery and virtual buffering techniques described above will be revisited in detail in the next chapter. At the level of the architecture, however, several alternate design decisions or extensions are possible.

6Further, given the in-order delivery used in theFUGU implementation of the direct VNI, any flow control scheme must make sure to clear the buffer before reenabling messages. Flow control schemes based on, for instance, fixed-sized windows may never clear the buffer.

Bulk Transfer. The direct VNI as described here is oriented to support for small messages where the primary consideration is low latency from processor to processor. Large messages for bulk transfer have the different goal of high bandwidth from memory to memory. The direct VNI is compatible with efficient solutions to bulk transfer (see [49]). Appendix A describes limited extensions for bulk transfer actually implemented and used inFUGU.

Source Buffering. The direct VNI applies buffering only at the receiver and largely for reasons of protection. It is possible to invoke buffering for performance reasons as well. Further, buffering for performance might use buffering at the sender as well as buffering at the receiver. Transparent buffering at the sender could be implemented with very similar techniques to the ones described here used at the receiver.

User Pinning. Table 3-2 lists user pinning as a means of managing physical resources. User pinning could be extended to be as flexible as virtual buffering. The key trick is that the application (or library) has to be able to renegotiate for buffer space at the moment a message arrives to avoid dropping any messages.⁷ Alternatively, virtual buffering could (and should) be extended to take advantage of advice from sophisticated applications that know the amount of buffering needed. A dynamic default policy plus a means for using advice is probably the right way to approach any resource management problem.

Bulk transfer, source buffering and user pinning are possible extensions to the direct VNI but do not change its basic architecture based on two-case delivery and virtual buffering. The next two chapters describe the two-case delivery and virtual buffering architectural techniques in detail, respectively.

7FUGU’s Exokernel operating system does in fact use user-level virtual memory in exactly this way.

Chapter 4