System model and definitions

4.2.1 System model

The system consists of a collection of n processes,p₁,p₂, ,p_n, that are connected by channels. There is no globally shared memory and processes communicate solely by passing messages. There is no physical global clock in the system. Message send and receive is asynchronous. Messages are delivered reliably with finite but arbitrary time delay. The system can be described as a directed graph in which vertices represent the processes and edges represent unidirectional communication channels. Let C_ij denote the channel from processp_ito processp_j.

Processes and channels have states associated with them. The state of a process at any time is defined by the contents of processor registers, stacks, local memory, etc., and may be highly dependent on the local context of the distributed application. The state of channelC_ij, denoted bySC_ij, is given by the set of messages in transit in the channel.

The actions performed by a process are modeled as three types of events, namely, internal events, message send events, and message receive events.

For a message m_ij that is sent by process p_i to process p_j, let sendm_ij andrecm_ijdenote its send and receive events, respectively. Occurrence of events changes the states of respective processes and channels, thus causing transitions in the global system state. For example, an internal event changes the state of the process at which it occurs. A send event (or a receive event) changes the state of the process that sends (or receives) the message and the state of the channel on which the message is sent (or received). The events at a process are linearly ordered by their order of occurrence.

At any instant, the state of process p_i, denoted by LS_i, is a result of the sequence of all the events executed by p_i up to that instant. For an event e and a process state LS_i, e∈LS_i iffe belongs to the sequence of events that have taken process p_i to stateLS_i. For an event e and a process state LS_i, e∈LS_iiffedoes not belong to the sequence of events that have taken process p_ito stateLS_i.

A channel is a distributed entity and its state depends on the local states of the processes on which it is incident. For a channelC_ij, the following set of messages can be defined based on the local states of the processes p_iandp_j [12]:

Transit transitLS_i LS_j= m_ij sendm_ij ∈LS_i

recm_ij ∈LS_j

Thus, if a snapshot recording algorithm records the state of processesp_iand p_j as LS_i andLS_j, respectively, then it must record the state of channelC_ij astransitLS_i LS_j.

There are several models of communication among processes and different snapshot algorithms have assumed different models of communication. In

91 4.2 System model and definitions

the FIFO model, each channel acts as a first-in first-out message queue and, thus, message ordering is preserved by a channel. In the non-FIFO model, a channel acts like a set in which the sender process adds messages and the receiver process removes messages from it in a random order. A system that supports causal delivery of messages satisfies the following property: “for any two messagesm_ij andm_kj, if sendm_ij−→sendm_kj, then recm_ij

−→recm_kj.”

Causally ordered delivery of messages implies FIFO message delivery. The causal ordering model is useful in developing distributed algorithms and may simplify the design of algorithms.

4.2.2 A consistent global state

The global state of a distributed system is a collection of the local states of the processes and the channels. Notationally, global stateGSis defined as

GS={

iLS_i,

ijSC_ij}.

A global stateGSis aconsistent global stateiff it satisfies the following two conditions [16]:

C1: send(m_ij)∈LS_i⇒m_ij∈SC_ij⊕rec(m_ij)∈LS_j(⊕is the Ex-OR operator).

C2: send(m_ij)∈LS_i⇒m_ij∈SC_ij ∧rec(m_ij)∈LS_j.

ConditionC1states the law of conservation of messages. Every messagem_ij that is recorded as sent in the local state of a process p_i must be captured in the state of the channelC_ij or in the collected local state of the receiver process p_j. Condition C2 states that in the collected global state, for every effect, its cause must be present. If a messagem_ij is not recorded as sent in the local state of processp_i, then it must neither be present in the state of the channelC_ij nor in the collected local state of the receiver processp_j.

In a consistent global state, every message that is recorded as received is also recorded as sent. Such a global state captures the notion of causality that a message cannot be received if it was not sent. Consistent global states are meaningful global states and inconsistent global states are not meaning-ful in the sense that a distributed system can never be in an inconsistent state.

4.2.3 Interpretation in terms of cuts

Cuts in a space–time diagram provide a powerful graphical aid in representing and reasoning about the global states of a computation. A cut is a line joining an arbitrary point on each process line that slices the space–time diagram into a PAST and a FUTURE. Recall that every cut corresponds to a global

Figure 4.2 An interpretation in terms of a cut.

m₃

m₄ m₅

m₁

m₂

Time e₁¹ e₁²

e₃¹ e²₃

e₄¹ e₄²

e₃³ e₃⁴ e₃⁵

e₁³ e₁⁴ e₂¹ e₂² e₂³ e₂⁴

C₁ C₂

p₁

p₂

p₃

p₄

state and every global state can be graphically represented by a cut in the computation’s space–time diagram [3].

A consistent global state corresponds to a cut in which every message received in the PAST of the cut has been sent in the PAST of that cut. Such a cut is known as a consistent cut. All the messages that cross the cut from the PAST to the FUTURE are captured in the corresponding channel state.

For example, consider the space–time diagram for the computation illustrated in Figure 4.2. Cut C₁ is inconsistent because messagem₁ is flowing from the FUTURE to the PAST. Cut C₂ is consistent and message m₄ must be captured in the state of channelC₂₁.

Note that in a consistent snapshot, all the recorded local states of pro-cesses are concurrent; that is, the recorded local state of no process casually affects the recorded local state of any other process. (Note that the notion of causality can be extended from the set of events to the set of recorded local states.)

4.2.4 Issues in recording a global state

If a global physical clock were available, the following simple procedure could be used to record a consistent global snapshot of a distributed system.

In this, the initiator of the snapshot collection decides a future time at which the snapshot is to be taken and broadcasts this time to every process. All processes take their local snapshots at that instant in the global time. The snapshot of channel C_ij includes all the messages that process p_j receives after taking the snapshot and whose timestamp is smaller than the time of the snapshot. (All messages are timestamped with the sender’s clock.) Clearly, if channels are not FIFO, a termination detection scheme will be needed to determine when to stop waiting for messages on channels.

However, a global physical clock is not available in a distributed system and the following two issues need to be addressed in recording of a consistent global snapshot of a distributed system [16]:

93 4.3 Snapshot algorithms for FIFO channels

I1: How to distinguish between the messages to be recorded in the snapshot (either in a channel state or a process state) from those not to be recorded.

The answer to this comes from conditionsC1andC2as follows:

Any message that is sent by a process before recording its snapshot, must be recorded in the global snapshot (fromC1).

Any message that is sent by a process after recording its snapshot, must not be recorded in the global snapshot (fromC2).

I2: How to determine the instant when a process takes its snapshot. The answer to this comes from conditionC2as follows:

A processp_j must record its snapshot before processing a messagem_ij that was sent by processp_iafter recording its snapshot.

We next discuss a set of representative snapshot algorithms for distributed systems. These algorithms assume different interprocess communication capa-bilities about the underlying system and illustrate how interprocess commu-nication affects the design complexity of these algorithms. There are two types of messages: computation messages and control messages. The former are exchanged by the underlying application and the latter are exchanged by the snapshot algorithm. Execution of a snapshot algorithm is transparent to the underlying application, except for occasional delaying some of the actions of the application.