We present an example upgrade to demonstrate the features of our model. The example system is a distributed file system, like SFS [76] or AFS [61]. The system has two classes of nodes: clients and servers. Servers store and control access to files. Clients use files by interacting with servers.
In version 1 of our system, servers control access to files using Unix-style permissions. That is, a server keeps a set of nine bits,Or, Ow, Ox, Gr, Gw, Gx, Wr, Ww, Wx, for each file that indicates whether the file owner (O), file group (G), and the rest of the world (W) can read (r), write (w), or execute (x) the file. Permissions are part of a server’s persistent state; they are stored in metadata blocks in the file system.
Unix-style permissions are adequate for small collaborative settings, but they are not expressive enough for larger systems. For example, permissions cannot designate a file as writable for one group of users, read-only for another group, and inaccessible to the rest.
To provide greater control over permissions, file systems like AFS [61] and SFSACL [64] keep an access control list for each file. That is, each file has a mapping from user or group names to a set of permissions.
In version 2 of our system, servers keep an access control list for each file. Access control lists are part of a server’s persistent state; each ACL is stored in the first 512 bytes of the file it protects.
This change does not affect how clients use files, because the server hides ACLs from clients, i.e., when a client reads the first 512 bytes of a file, it sees the file’s data, not its ACL. However, this change does affect how clients manage file permissions, so the protocol between clients and servers must change.
Thus, the upgrade from version 1 to version 2 consists of two class upgrades: the first changes the file server implementation fromPermServer(a server that uses permissions) to AclServer(a server that uses access control lists), and the second changes the client implementation to support management of access control lists.
2.5.1 Model
We do not model the clients, because they have no methods.
PermServerhas the following methods (the client’s identity is an implicit parameter to all server methods):
getPerms(f) Returns the owner, group, and permission bits for filef.
setPerms(f, owner, group, bits) Sets the owner, group, and permission bits for file f, or throws NoAccessif the client is notf’s (current) owner.
canAccess(f, mode) Returns true iff the client can access file f in mode mode according to f’s permissions.
We omit the methods for creating, reading, writing, and removing files from this model because they are unaffected by our example upgrade (these methods usecanAccessto check whether the client is allowed to do the requested operation).
AclServerhas the following methods:
getACL(f) Returns the access control list for filef.
setACL(f, acl) Sets the access control list for filef, or throwsNoAccessiff’s ACL does not grant the client permission to modify the ACL.
canAccess(f, mode) Returns true iff the client can access file f in mode mode according to f’s access control list.
The rest of the server’s methods (those for manipulating files) are unchanged.
Now we consider the components of the two class upgrades.
2.5.2 Scheduling Functions
We have to define scheduling functions for both the client and server class upgrades.
We want to avoid disrupting service, so the client scheduling function should attempt to avoid signaling the upgrade layer while the user is active. The SF can do this by signaling late at night or while the client has had little activity for a long time. Alternately, the SF could periodically display a dialog box to the user requesting that the client be allowed to upgrade.
The server scheduling function also needs to avoid disrupting service, but if there is just one server, the best the SF can do is schedule the server upgrade when there is low client activity.
However, if servers are replicated, we can use a rolling upgrade [16, 33, 102] that upgrades just one replica at a time. In this case, the SF signals the upgrade layer when all other replicas with lower IP addresses have upgraded. Since IP addresses define a total order on servers, this SF implements a rolling upgrade.
2.5.3 Simulation Objects
The server upgrade changes the server’s type. Our scheduling functions allow clients to upgrade before servers or vice versa, so we must use simulation objects to enable upgraded clients to use non-upgraded servers and non-non-upgraded clients to use non-upgraded servers. This means we need two SOs: a future SO that implementsAclServerwhile the node is running thePermServerimplementation and a past SO that implementsPermServerwhile the node is running theAclServerimplementation.
The challenge in defining these SOs is that some ACLs cannot be expressed as permissions. We explain how to reason about this challenge and define the SOs in Section 3.5.
2.5.4 Transform Functions
The client class upgrade does not change the client’s persistent state, so it requires no transform function. However, the server class upgrade requires a TF to convert the permissions for each file to access control lists. Permissions are stored in metadata blocks in the file system, whereas ACLs are stored in the first 512 bytes of the files themselves [64].
Therefore, the TF needs to change the representation of every file on the server. For each file in the file system, the TF copies the file’s data to scratch space, reads the file’s permissions from the metadata blocks, converts these to an ACL (as described in Section 3.5), writes the ACL to the beginning of the file, then writes the file’s data after the ACL. We have implemented this TF, as described in Section 8.2.2.
This TF is slow, because it needs to read and write each file’s data twice (to the scratch space and back to the file). This takes a long time for large file systems, and the server is unavailable while the TF is running. One way to reduce this downtime is to allow the server to recover immediately and run the TF incrementally, e.g., transform each file only as it is accessed. Supporting incremental transforms is future work; we discuss how to support them in Section 10.3.1.
Chapter 3
Specifying Upgrades
Our approach allows nodes to upgrade asynchronously and yet communicate, even though upgrades may make incompatible changes. We accomplish this by augmenting nodes to support multiple types simultaneously. This enables a node’s non-upgraded clients to use the node’s pre-upgrade type and enables upgraded clients to use the node’s post-upgrade type.
This chapter explains how to specify the relationship between a node’s pre-upgrade and post-upgrade types as part of defining an post-upgrade. This specification guides the design of the simulation objects and transform function for an upgrade. The specification also tells clients what changes to expect when they upgrade from one version to the next and switch from using the pre-upgrade to the post-upgrade type.
The definitions in this chapter are informal, but we aim for them to be precise enough to enable developers to reason about programs and upgrades. Formalizing our model is future work.
3.1 Specifications
There are various ways of defining specifications; in this thesis, we use informal specifications in the requires/effects style [72] to define the preconditions and postconditions of a method.
• The requires clause defines constraints on the arguments to a method and the state of the object before the method runs.
• The effects clause describes the behavior of the method for all inputs not ruled out by the requires clause: the outputs it produces, the exceptions it throws, and the modifications it makes to its inputs and to the state of the object.
class IntSet AnIntSetis a mutable, unbounded set of integers IntSet() effects: this = {}
void insert(x) effects: thispost = thispre∪ {x}
void delete(x) effects: x∈thispre ⇒ thispost = thispre− {x}, else throws NoSuchElementException boolean contains(x) effects: returns x∈this
Figure 3-1:Specification forIntSet
Figure 3-1 gives a specification for type IntSet(a set of integers) in this style. We use thispre to denote the pre state of the object and thispost to denote the post state. All examples of method specifications in this thesis are total, so none has a requires clause.
3.1.1 Failures
Each method call terminates either normally or by throwing an exception. Furthermore, we assume that any call can terminate by throwing a special exception that indicates failure. Such termination happens, e.g., if a remote method call fails because the target node is unreachable. Failures are unpredictable can happen at any time. This model is common in distributed object systems: Java RMI [79] allows any remote method invocation to fail by throwing RemoteException, and Sun RPC [99] allows any remote procedure call to fail with error codes indicating the unavailability of a server or individual procedures.
We extend the notion of failure to include cases when a node is unable to handle a call correctly because the version of the caller is different from that of the receiver. These failures appear to the caller as node failures, so the caller may try an alternate node or a workaround, or it may wait and try the call again later. Of course, such failures and delays disrupt service, so we want to avoid causing calls to fail when possible.
3.1.2 Subtypes
We review behavioral subtyping [73] briefly here.
One type is a subtype of another if objects of the subtype can be substituted for objects of the supertype without affecting the behavior of callers that expect the supertype. Subtypes that satisfy this substitution principle support three properties (as stated in [72]):
Signature Rule The subtype objects must have all the methods of the supertype, and the signatures of the subtype methods must be compatible (contravariant argument types, covariant return and exception types) with the signatures of the corresponding supertype methods.
Methods Rule Calls of subtype methods must “behave like” calls to the corresponding supertype methods, i.e., subtype methods may weaken the precondition and strengthen the postcondition of the corresponding supertype methods.
Properties Rule The subtype must preserve invariants and history properties that can be proved about supertype objects.
Invariants are “properties true of all states” of an object, and history properties are “properties true of all sequences of states” of an object [73]. History properties define how an object evolves over time, and they are derived from the specification of the object’s type. For example, if we remove thedeletemethod from IntSet, then we can derive the property that later sets are always supersets of earlier ones (regardless of the intervening method calls). Clients can rely on this property when reasoning about programs that use this type, e.g., they know that once an element is in a set, it will always be in the set.