Fault-Tolerance: Checkpointing in Distributed Asynchronous Systems

(1)

Master Recherche en Informatique - Novembre 2010

Fault-Tolerance: Checkpointing in Distributed Asynchronous Systems

Achour Most´efaoui

Irisa/Ifsic, Universit´e de Rennes [email protected]

http://www.irisa.fr/asap/

Checkpointing Distributed Computations 1

Some Failure Types

• Software errors

• Process failures

⋆ Crash failure

⋆ Send/Receive omission

⋆ Arbitrary (Byzantine)

• Link failures

⋆ Omission/duplication failure

• Clock/Performance failures

How to Tolerate Failures?

• Debbuging/Validation.

• Duplication of processors and memories.

• Faul-tolerant software.

⋆ Data replication.

⋆ Fault-tolerant services (consensus, NBAC, etc.)

⋆ Checkpointing/Rollback-Recovery.

Computation Model

Asynchronous Distributed System

• Set of processes: P₁, . . . , P_n.

• No shared memory.

• No global clock.

• Fail-stop processors (crash failures).

(2)

What Is Rollback?

P

σ4

P

σ4

P

e1 e2 e3

σ4

e5

e4

• Failure occurrence: restart at a safe state.

• Necessity to save safe states.

Local States and Local Checkpoints

• Local History of P_i: e_i,1 e_i,2 · · · e_i,s · · ·

• Local State:

⋆ initial state of P_i: σ_i,0

⋆ σ_i,s is obtained by applying the event e_i,s to the local state σ_i,s−1

• Local Checkpoint:

A local checkpoint C is a recorded state (snapshot) of a process.

A local state is not necessarily recorded as a local checkpoint, so the set of local checkpoints is only a subset of the set of local states.

Distributed Computation: Example

Pi

Pj

Pk

Ik,1 Ik,2 Ik,3

Ij,1

m1 m2

m3 m4

m5

m6 m7 Ci,3

Ci,2

Ci,1

Ci,0

Cj,3

Cj,2

Cj,1

Cj,0

Ck,3

Ck,2

Ck,1

Ck,0

A State of a Distributed Computation?

• Remind: There isno global memory andno global clock

• Impossibility to compute instantaneously a global snapshot

• Illustration of Chandy-Lamport (1985)

• A global checkpoint is a set of local checkpoints, one from each process.

(3)

A pair of Mutually Consistent Checkpoints

Pj

Pi

A Missing Message

Pj

Pi

?

• The message is missing ⇒ recording.

An Orphan Message

Pj

Pi

?

• The two local checkpoints are definitelyinconsistent.

Consistency Rules of a Global Checkpoint (CL 1985)

Let C_1,x₁, C_2,x₂, . . . , Cn,xn be a global checkpoint. Each pair (C_i,x_i, C_j,x_j) of local checkpoints respects:

R1 ∀m, send(m) ∈C_i,x_i ⇒delivery(m) ∈C_j,x_j or M is recorded (no missing).

R2 ∀m, send(m) 6∈C_i,x_i ⇒delivery(m) 6∈C_j,x_j (no orphan).

(4)

Consistent Global Checkpoints

• Consistent Global Checkpoint

A global checkpoint is consistent if it has no message delivered and not sent (no orphan message).

• Message recording can easily overcome the problem of missing messages.

Meaning of a Global Checkpoint

Pi

Pj

Pk

Ik,1 Ik,2

Ci,0 Ci,1 Ci,2

Cj,0 Cj,1 Cj,2 Cj,3

Ci,3

Ck,3

Ck,1

Ck,0

m3 m4 m6

m₂ m1

Ik,3

m₅

Ck,2

m7 Ij,1

• Does a consistent global checkpoint represent a real state of the computation?

Why to Compute Global Checkpoints?

• Rollback recovery.

• Stable/Unstable properties detection

• Monitoring

• etc.

Caracteristics of a Global Checkpoint

• To be as close as possible to a real state (monitoring)

• To be the most recent possible (rollback)

• To have as few forced checkpoints as possible (properties detection)

• etc.

(5)

Example: Rollback Recovery

Pi

Pj

Pk

Ik,1 Ik,2

Ci,0 Ci,1 Ci,2

Cj,0 Cj,1 Cj,2 Cj,3

Ci,3

Ck,1

Ck,0

m3 m4 m6

m2

m1

Ck,2

m7 Ij,1

Limits of Chandy-Lamport’s Consistency Rules

• Chandy-Lamport result applies to global checkpoints:

⋆ Considering a global checkpoint, one can say whether it is consistent or not.

• Question:

⋆ Considering a single local checkpoint taken by a process, How to know whether there exists a consistent global checkpoint to which it belongs?

Checkpointing protocols must ensure that each local checkpoint belongs to at least one global checkpoint with:

1 No orphan messages.

2 No missing messages.

Limits of Chandy-Lamport’s Consistency Rules: Example

Pi

Pj

Pk

Ik,1 Ik,2

Ci,0 Ci,1 Ci,2

Cj,0 Cj,1 Cj,2 Cj,3

Ci,3

Ck,3

Ck,1

Ck,0

m₃ m₄ m₆ m₂

m1

Ik,3

m₅

Ck,2

m₇

Ij,1

• Hidden dependencies.

(6)

Theorem of Netzer and Xu (1995)

• Considering a subset of local checkpoints (possibly one local checkpoint), one can say whether this subset can be extended to form a consistent global checkpoint.

Z-Paths

A relation exists from local checkpoint A to local checkpoint B if there exist a Z-path from A to B.

B

A

Z-Paths

A Z-path exists from local checkpoint A to local checkpoint B if and only if:

• A precedes B within the same process, or

• a sequence of messages [m₁, m₂, . . . , m_q] (q ≥ 1) exists such that:

1. A precedes send(m₁) in the same process, and

2. for each m_i, i < q, delivery(m_i) is in the same or earlier interval as send(m_i+1), and

3. delivery(mq) precedes B in the same process.

Causal Z-Paths, Z-Patterns and Z-Cycles

• A Z-Path is Causal iff for each m_i, i < q, we have delivery(m_i) →^hb send(m_i+1).

• a Z-Path has a Z-Pattern iff ∃i such that:

send(m_i+1) →^hb delivery(m_i).

• a Z-Cycle is a Z-Path going from a local checkpoint C to the same local checkpoint C.

(7)

Causal Z-Path: Example

Pi

Pj

Pk

Ik,1 Ik,2

Ci,0 Ci,1 Ci,2

Cj,0 Cj,1 Cj,2 Cj,3

Ci,3

Ck,3

Ck,1

Ck,0

m3 m4 m6

m2

m1

Ik,3

m5

Ck,2

m7 Ij,1

Z-Cycle: Example

Pi

Pj

Pk

Ik,1 Ik,2

Ci,0 Ci,1 Ci,2

Cj,0 Cj,1 Cj,2 Cj,3

Ci,3

Ck,3

Ck,1

Ck,0

m3 m4 m6

m2

m1

Ik,3

m5

Ck,2

m7 Ij,1

Basic Theorem

• A local checkpoint C is Useless if it cannot belong to any consistent global checkpoint.

• Netzer-Xu Theorem (1995): A local checkpoint C is useless iff it is involved in a Z-cycle.

Basic Checkpoints

• Some local states of each process called local checkpoints are saved on stable storage.

⋆ periodically,

⋆ upon the reception a signal,

⋆ according to the value of a predicate

⋆ according to the OS convenience (light-load, etc.)

⋆ etc.

(8)

Uncoordinated Checkpointing: Example 1

Each process has its own checkpointing policy.

Pj

Pi

Risk : domino effect.

Forced Checkpoints

Pj

Pi

Pk

• In order each local checkpoint belongs to at least one consistent global checkpoint, some processes may have to take additional checkpoints (forced checkpoints).

Checkpointing Protocols

• Coordinated Checkpointing

⋆ only Forced checkpoints, no Domino effect.

• Uncoordinated Checkpointing

⋆ only Basic checkpoints, possibly: Domino effect.

• Communication Induced Checkpointing

⋆ Forced + Basic checkpoints, no Domino effect.

Chandy-Lamport’s Coordinated Protocol (1985)

This protocol is based on the use of control messages:

markers.

Pj

Pi

marker

• A marker is a message that can neither overtake nor be overtaken by any other message sent on the same unidirectionnal channel.

(9)

Communication-Induced Checkpointing

• No communication ⇒ no Z-cycles.

• Avoid Z-cycles formed by application messages.

⋆ Detect Z-cycles: control information carried by messages

⋆ Break Z-cycles: forced checkpoints

Breaking a Z-Cycle

Pj

Pi

Pj

Main Idea of the Protocol

• Idea: Asssociate a Lamport Timestamp with each local checkpoint.

• Theorem: If for any pair of checkpoints C_j,y and C_k,z: Z-path from C_j,y to C_k,z ⇒ C_j,y.t < C_k,z.t,

then no checkpoint can be involved in a Z-cycle.

A Protocol: Second Step

Each message carries the value of its sender’s clock at sending time.

• Init: cl_i := 0

• Upon the definition of a local checkpoint cl_i :=cl_i+ 1

• When P_i sends a message m:

m.cl :=cl_i; send(m, m.cl)

• Upon the reception of a message (m, m.cl) cl_i :=max(cl_i, m.cl)

(10)

A Protocol: First Step

Use a lamport clock to timestamp checkpoints.

Pi

Pj

Pk

m3 m4

m2

m1 m5

m7

3

4 4 3

2 1

1

2

1 2 3

2 4 5

• Does not take into account hidden dependencies (non- causal Z-paths).

Hidden Dependencies

• Timestamps of messages increase along causal Z-paths.

• Timestamps of messages should increase along all Z-paths.

To Checkpoint or Not to Checkpoint

Pk

Pi

Pj

m1

m2 C_k,z Cj,y

Pk

Pi

Pj Cj,y

m1 C_k,z m2

a. b.

C_i,x

m₁.t ≤ m₂.t m₁.t > m₂.t

General Structure of the Protocol

• Init:

cl_i := 0;. . .

• When P_i takes a local checkpoint

cl_i :=cl_i+ 1; < resetting of data structures >

< Take a local checkpoint timestamped with cl_i >

• when P_i sends a message m:

m.cl :=cl_i; send(m, m.cl, . . .)

• Upon the reception of a message (m, m.cl, . . .) if < condition > then < take a ckpt >; (*forced*)

(11)

A First Condition to Checkpoint

• sent to_i[k] has the value true iff P_i has sent a message to P_k since its last checkpoint.

• min to_i[k] keeps the timestamp of the first message P_i sent to P_k since P_i’s last checkpoint.

C ≡(∃k : sent to_i[k]∧m₁.t > min to_i[k])

Refining the Condition (1)

Pi Cj,y

Ck,z

Pk

Pj

m2 µ2

m1

Pi Cj,y

Ck,z

Pk

Pj

m2

m1

µ1

a. b.

Refining the Condition (2)

• cl_i(k)= value of P_k’s local clock as perceived by P_i (P_i can obtain this knowledge with a classical piggybacking technique).

(m₁.t≤m₂.t)∨ P, where P ≡ (C_i,y.t≤m₁.t≤cl_i(k)< C_k,z.t).

Does cl_i(k) Refers to a Correct Value?

a. b.

Pi Cj,y

Pk

Pj

m1

Pi Cj,y

Pk

Pj

m1

µ1 µ2

C_i,x C_i,x

m2 Ck,z m2 Ck,z

Pi

P_k

m2 µ

C_i,x

C_k,z

causal Z-cycle

(12)

Final Condition

C^′ ≡

(∃k : sent to_i[k]∧(m₁.t > min to_i[k])∧(m₁.t > cl_i(k)∨ C₁))

C₁ being the condition that detects causal Z-cycles

Particular Cases

• C^′′≡ ∃k : (sent to_i[k]∧(m.lc > min to_i[k]))

• C^′′′ ≡(m.lc > min_i)

Checkpointing protocols must ensure that each local checkpoint belongs to at least one global checkpoint with:

1 No orphan messages.

2 No missing messages.

Question: How to ensure “no missing” in a not too much costly way?

Chandy-Lamport’s Coordinated Protocol: Example

Pj

Pi

σj

σi

σj

σi

m1 m2 m1 m2

• Which messages are in-transit (must be recorded)?

(13)

Chandy-Lamport’s Recording Rule

Upon the reception P_i of a marker sent by P_j

• If P_i has not yet taken a local checkpoint:

⋆ no message is in transit wrt this pair of local checkpoints

• If P_i has already taken a local checkpoint:

⋆ all the messages received after σ_i and before the reception of the marker.

Timestamps in a Checkpoint Interval (1)

P_k Pi

Pj

m1

m2 Ck,z Cj,y

P_k Pi

Pj Cj,y

m1 Ck,z

m2

a. b.

Ci,x

m₁.t ≤ m₂.t m₁.t > m₂.t

Timestamps in a Checkpoint Interval (2)

Pi

M ax Rec_i ≤cl_i ≤M in Sent_i

(initially, M ax Rec_i =−∞ and M in Sent_i= +∞)

In-Transit vs. Orphan (1)

Pi

Pj

in-transit orphan

Reversed computation Computation

(14)

Remark: In-Transit vs. Orphan (2)

• Let us consider the computation where all messages are reversed.

• Ensuring each local checkpoint belongs to a least one orphan-free global checkpoint of the reversed computation is equivalent to ensure each local checkpoint belongs to a least one missing-free global checkpoint of the original computation.

M ax Sent_i ≤cl_i≤ M in Rec_i

Recording Messages vs. Checkpointing

• Question: Can a recorded message be missing wrt a global checkpoint?

M ax Sent N R_i≤cl_i≤M in Rec N R_i

No Orphan and No Missing Messages (R1 and R2)

Within each interval the following invariant must be pre- served:

max(M ax Rec_i, M ax Sent N R_i)≤cl_i ≤ min(M in Sent_i, M in Rec N R_i)

Sketch of a Protocol (1)

M in Rec N Li

M in Senti

cli

M ax Reci

M ax Sent N Li

(15)

Sketch of a Protocol (2)

What to do (upon a send or a receive operation) in order to maintain the invariant?

• Increase the value of cl_i.

• Record some sent or received messages.

• Take a forced checkpoint.

Sketch of a Protocol: Example 1

m1

Pi

m2

8 5

2

Variable Before m₁ Before m₂ After m₂

M ax Rec_i −∞ 5 8

M ax Sent N R_i −∞ −∞ −∞

cl_i 2 5 ?

M in Sent_i +∞ +∞ +∞

M in Rec N R_i +∞ 5 5

Variable Before m₂ Record m₁? Ckpt before m₂?

M ax Rec_i 5 8 −∞

M ax Sent N R_i −∞ −∞ −∞

cl_i 5 8 >5

M in Sent_i +∞ +∞ +∞

M in Rec N R_i 5 8 +∞

There is a choice: checkpointing or recording m₁

m1

Pi

m2

8 2

2

Variable Before m₁ Before m₂ After m₂

M ax Rec_i −∞ −∞ 8

M ax Sent N R_i −∞ 2 2

cl_i 2 2 ?

M in Sent_i +∞ 2 2

M in Rec N R_i +∞ +∞ 8

(16)

Variable Before m₂ Record m₁? Ckpt before m₂?

M ax Rec_i −∞ 8 −∞

M ax Sent N R_i 2 −∞ −∞

cl_i 2 ? >2

M in Sent_i 2 2 +∞

M in Rec N R_i +∞ 8 +∞

There is no choice: checkpointing