Resource Management

Virtual buffering gives the operating system the opportunity to manage physical resources used for buffering. Given that freedom, the operating system then needs a policy for allocating and deallocating physical page frames used for virtual buffering. This policy needs to coexist with other consumers of page frames in the operating system, i.e., the virtual memory manager and the file cache manager.

We have not proposed or evaluated any such policies, other than the resource management implied by overflow control. One interesting question is what information is needed to best optimize buffer usage. For instance, message rates could be measured as well as simple buffer sizes. A second interesting question is how virtual buffer management should interact other demands for physical memory, i.e., virtual memory and file caching. The page frame cache could be partitioned or unified.

A third interesting question is what kind of API one might export to take advantage of application

“advice”.

5.4 Discussion

The virtual buffering technique raises several research questions. Beyond basic feasibility, the major question is whether, having given the user the offer of “unlimited” buffering, the demand for real buffering is in fact overwhelming. The second experiment in Chapter 7 (Section 7.4) will show that demand for buffering remains low in our sample applications, even when buffering is artificially induced. The sample applications provide adequate flow control implicitly due to their intrinsic characteristics. While flow control can always be added in a user-level library, we will show that the conditions for avoiding excessive buffering in the direct VNI are quite mild. Further, the overflow control mechanism described above in Section 5.2 is evaluated in Section 7.5, where we show that it can effectively throttle runaway applications, to their benefit and to the benefit of the rest of the system.

In addition to the major question, three research questions were raised above in this chapter and only partly addressed. First, the second network, used for deadlock avoidance and as part of overflow control, is intended to be used rarely and thus as a component has only a second-order impact on performance. However, the exact performance requirement of the second network in a real system is unknown. The bandwidth and latency issues described in Sections 5.1 and 5.2 are described in additional detail by Lee [42]. Second, the domain of overflow control mechanisms and policies is broad; the scheme described in Section 5.2 is only one simple possibility. Third, as mentioned in Section 5.3, we have not addressed the resource management questions in virtual buffering beyond the use of our overflow control policy.

Cross-Domain Messages. Another issue that is only minimally addressed in the direct VNI is the question of cross-domain messages. Cross-domain messages are incompatible with the idea of unlimited buffering. The problem is that one application can flood another application’s swap space with messages, which implies that the applications must trust one another. There are three possible approaches to reconciling virtual buffering with cross-domain communication between mutually distrustful applications:

The first approach is to allow cross-domain communication with unlimited buffering but to store the buffered messages in swap space charged to the sending application.

The second approach is to abandon unlimited buffering for cross-domain messages and instead provide some other guarantee, such as a fixed-size buffer negotiated at the time the cross-domain connection is set up. Protocols built atop cross-cross-domain messages would the have to explicitly take the buffer size limit into account.

The third approach, taken in the direct VNI architecture, is just to outlaw cross-domain messages. Either the server or the client must be distributed (even if only as a “stub”) so that inter-process communication is always local to a processor. This organization is actually quite natural for a scalable workstation in which all the kernels are trusted. For instance, a distributed server such as the web server used for the experiments in Chapter 7, can optimize its internal communication and perform caching transparently. The server interface abstraction is decoupled from the server’s communication requirements, which is a useful property.

Applicability. Since the mechanisms of virtual buffering are non-trivial, it is convenient to be able to implement the mechanisms in software. Using virtual memory is particularly natural when the

processor initiates all the buffering because existing support for virtual memory (e.g., the processor’s TLB) is reused. It requires a relatively complex DMA engine or coprocessor to manipulate virtual memory independently [25, 56, 65, 79].

However, virtual buffering is usable in any system that employs buffering. For instance, a system that performs limited buffering in hardware could implement virtual buffering by using interrupts to dynamically expand the buffers [55, 56, 79]. One simple way to perform limited buffering would be to give the hardware a small table of active page frames indexed by the GID. The hardware would perform buffer insertion into these pages automatically until the page frame was full and then cause an interrupt. The hardware methods have all the same software complexity issues as our all-software method, albeit with a less demanding time constraint since traps occur less often.

The interrupt handler that responds to a hardware buffer overflow must still be able to allocate a page frame in an unusual situation and the system must still address deadlock, overflow control and resource management.

The previous chapter and this chapter have described the architectural techniques of two-case delivery and virtual buffering used to support programmability, protection and performance in the direct VNI. These chapters have been oriented to the architecture, although we have covered many of the implementation details as concrete examples. The next chapter describes the experimental system in which the direct VNI was implemented and evaluated.

Chapter 6