HAL Id: inria-00070713
https://hal.inria.fr/inria-00070713
Submitted on 19 May 2006
HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
A Performance Evaluation of a Quorum-Based State-Machine Replication Algorithm for Computing
Grids
Jean-Michel Busca, Marin Bertier, Fatima Belkouch, Pierre Sens, Luciana Arantes
To cite this version:
Jean-Michel Busca, Marin Bertier, Fatima Belkouch, Pierre Sens, Luciana Arantes. A Performance
Evaluation of a Quorum-Based State-Machine Replication Algorithm for Computing Grids. [Research
Report] RR-5287, INRIA. 2004, pp.17. �inria-00070713�
ISRN INRIA/RR--5287--FR+ENG
a p p o r t
d e r e c h e r c h e
Thème COM
INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET EN AUTOMATIQUE
A Performance Evaluation of a Quorum-Based State-Machine Replication Algorithm
for Computing Grids
Jean-Michel Busca — Marin Bertier — Fatima Belkouch — Pierre Sens — Luciana Arantes
N° 5287
Août 2004
0 (21354&678'9:!;< =4
?>, @8'A&<$B;:.&-
CEDGF+HI:JLKNM O5DQPSRUTVWMGF
∗
X JYF+ZWK[HRUD\Z^]WK[DQZ∗
X`_ F2]aK[bcFdRUD\P[e+fgT5M O†
X5h KNDQZ^ZWDjiDQHV∗
Xk/T5MQK[FEHFmlZWFEHn]aD\V
∗
oprq^sUtvu/wxzy|{G}~q^sUtW~/\sUsv rr:\~
`t:/:\
r+\tt^prt^p2tE5Qy¡ \¢S£\£¤ y¦¥ +tW~
§©¨ª^«:¬ r®«¯±° r rs²~}G~t^s³~´tµ9t:¶¸·[¹Gr µ/'Q\¶~p2¶sºr »tprt Et:¼½s³2:t 2prt
a» ¸¾r¶¶¿}U¼À~¸¾r rt^©~}~t:s³~^Á5Âópr~9t^+\µt´tÄr¶t/prt^¸9 2~t´\~ sUtW2~`Upr¶t:»\t
¶ µmt^~+\2~t`¶sUt¼½rt:µ\¹~t^»GtW~np+ t`t^r¶¶^ tW+ ^tW~~t^ »\t::sºr ¸2Sr~:Á
o'p+ vt^2nÅ5µ9t³r\+Q~t³¾EpÆÇQ r rsÈ+~ 2¶m+mÇQ r rsU·[¾+~t^É~ t:·[s³\pr¸2t
t:2¸:¶c¸\¿prsp+ ³\¸t^tW~U:\~pc¼Ê¶¶ rtW~U¸É2¶¶}Ë~}G2p2\r 2~vsºt:[ÁÌjt
~pr µÍpr 2pÉprt&t:» ¶ 2¶d¼Ît^`¸sºr¶t:sºt^\ ¸\dp2v¸pr\ rpm~¸sºr¶tÅpr¶~UÇQ r rs
:2~ 2¶Æ+Æt:r¶¶^ ¸\ƶ¸prsÈtÄpr¶¾r¸ºÏtW~E2~t³¸sºtΣQÐѶ µt^p2mp2 º¼S
t:\ r/¶»tt^r¸:¶'¶\¸prsÒ¶Îr2\r t:2r¿¸\2~:Á
ÓÎÔQÕgÖ× Ø ¬ Ùª¯ t^¼½\s³2tW~:Å ° r\ 2s?{G}~t:s³~^Å/t^r¸:¶ÚÅ5{QtAx\pr¶rtÅusºr ¸r
Û r~:Á
∗
ÜÀÝÞßà†
áâaãä ÝåÊæãçNèêéQãàëàëÝì8'9 .¦54&:&9 '-ÀÈ.º'-
./-n|<5:/- .È99
ª ¯ tW~S~}~q:sºt^~´tvÇQ r\ rs³~´~\\t^~´+~ 2¶2~S¾r¶t:Ï:rG rtW~SÇQ rÀEt:sºtt:Q
sU^¸¶t:¶t^~Et:¼½s³2:t^~t:¶Î~+\r¸¾2¸¶¿©tW~~}~q:sºtW~~¶¾r rW~:Á+~t³rE^Å
2 2~t:Är¸\\2~¶t: rv ¶¸~ ¶m:sºsºtAsº }t:msº^¸¶t:¶t³t^sº2~ t©^+\2~t³t&~t^»Gt^~
r:r¶ }W~~ rtW~¸¶¶t^~tA^ r[Á´2~t³¾r r^ÅÀ2 2~2\+Q~\2~'&¼½~ rrtA:2~ 2=¸\
rtÇ\ 2 rs³~/t r¸\¿prsºtt^r¶¶^ ¶Îts³\pr¸2tºU¾2~tt ÇQ r\ 2s ÇQ rÚ¶q:t
rt^~`+rrtW~¼½2p2t^~ê22~ê r³sºGrq:¶tS2¸t^¸¶t:sºt:Q~}G2prrt\Á S 2~sºQ2~^Å ³a»t^~t
!:» ¶ 2 ¸\& rrt¶sºr¶:sºt:Q¶^t:¶¸t\ÅÇ\ 2ttt\2~ 2=¸\©t:tS¶\¸p2sUt\ÅG¾r¸t^ÇQ rt
~¶sU2¸tW~:År:2 2¶~t:QSr2~/t^~S+¿¸\2~/2r\r¸^t^~"U rt:sº2~St^+\2~t£QÐ ¶¼½^¶t: r#
:t:¶ rÚ r¶\¸p2sUtt:r¶¶^ ¸\\=¶»t~2rnÁ
$,Ø «Wª Ö ®&% ªU¯ t^¼½\s³2tW~:Å\{}G~q^sºt^~êrt´ÇQ r rs³~^Å\/^r¸:¶ÚÅQxp2¸rt ^Å Û ¸¶¸tW~êt
^ r[Á
A B
vÀ5³.&ê
t:r¶¶^ ¸\Ï~´&¹t^}rr\\p'¸sºr »t rtµ9¹Î~t^»Gt^~a» ¶¶¾r¶¸¸}2Î¼Ê r¸·[¶t:+tÁ
C2¶¸ µ/¶rp2¶~rr\\pÚÅsº¶~~¸\· :¸:r~t^»\t:s³a}¾Et¶2~Q t^U¶A~t^»t: +--,-DE1<F8'
+ :Q\r¸2tW¾G}~\sºt9t:2¸:¶ 2\p2 `rt^~t:»tW~nprt:¶s 2r:2~~t:2:}Á{t:»t:
t:2¸:\~t´pr\~tW³v~t¼À2Grt^~µ/pr¶pt´~t:¶t^t^AUtÄpr¶¾r¸¶2t^+t^2t^\9¼Ê¶¶ rtr¾·
¾r¶¸¸}År~Up2/prt »\t:¶Ea» ¶¾r¶¸¸}³¼p2t~t^»Gt¶~\= 2¶¸}©¸sºr »tWnÁ
GS µt^»t:WÅpr~ºrr\\pd¶~t^~µ9϶sº+\Qv:2:t:2~t:Q¸2Î+t^¼½\s³+t^~^Á4C¶~^Å
t:2¸:¶¶2 2:t^~~¶rIH+^Q:\~© »t^prtWr~A¸t:s³~©¼rG:t^~~¶rmE µt^&2rt:µ\¹
¾+2µ/GpÆ+~ rsº¸\KJ¼½\¶2~2:tÅ5¶»t³t:r¶¶^ ¸\jr~p2a»\tA¶ê~t^»\t:t:r¶:~
2tW~~t:»\t:}~t^»Gtêt^ÇQ rtW~ML¾Et^~¶tW~:Å sºQ~t:r¶¶^ ¸\r:~Àt^ÇQ r¶t9rr¿¸\2Gsºt^~~tW~
\μ5rr¶¶^ ¸\·[¶t:»\t:nt^ÇQ rtW~~2t:r¶¶t^~³rtW~t^»\t~t^»\t:/t:r¶¶^~/:2~¶~t^2}\Á9{GtW=·
\2nÅat:r¶¶^ ¸\:~¶rIH+:Q¶}´¶2t^\~t 2Ert9tWÇQ rt^~~nt^~E2~t¶sºtÅ~¸+tprt2tW~~¶r
¼~ 2p©t^ÇQ rtW~~9t:2t:¶¸}³t^ÇQ r¶tW~`³Q=/~t:»t^NÅG¸¼r¶Ú~t^»\t:9t^r¸:\~:Áopr~9¶~~ rt¶~
¸prtºsU\tU¶sUEQ\~¸Ér=¶:tÅnt^ +¶r©prtºr¾2¾r¸¶¸}μ9:t^¶tW'¼Ê¶¸ 2tW~´¼t^
t^ÇQ r¶t^~êp2/~t:»t^t:2¸:\~¾Et¸^ tW©¶t:\rp2¶^¶¸}º¶»t:~t´¶: ¸\2~^Á ´~:N@O ~pr µ9t^©¿
¼½s t^r^~t~ 2}\Å\s³p2¸rtW~`~prtW~tt: tW µ/¸p2¸A\¸»\t:ºÂÃQt^rt:ê~t^sºt:QWÅ\+
2tµ9¹+¸¶2~tUrtÅgs³¹G¶r'¿rtWt^~~}&Î~rt^\Ï~t:»t^t:2¸:\~´ »\t:~t^»t^
ÂÃQt:rt~t:\sºt:Qêp2 t/µt^¸g2ê \= +¶¸}º¶2t^\~t/ »\t:¶2a» ¶¶¾r¸¶¸}v+³rt:»t^Q`prt
~t:»t^ê¼½s7¾+t^¸2v¶¸}©r¶~\rrt^tW©¼½s ¿~¶¸t^Q~^Á
° r\ 2s%~}~t:s³~9t´µ9t:¶¸·[¹Gr µ/³G~`p2 2tW~~êp2t*H+~9~~ rt\Á5Âü½s³¶¸}\ÅGUÇQ r rs
~}~t:s7¶~º:¶¸tW=¶'¼5~ r¾2~t~¼~t^»\t:/t:r¶¶^~^Åt:»\t:}A2¶/¼5µ/prp'¸Qt^~t^^Áêop+r¹~9
prtU¸Qt^~t^¶ÎrEt:}ÅEt^\pÏ~ r¾2~tS·/2sºt:¶}'&ÇQ r rs ·^Ï\=¾Et:p2¿¼¼`prt µ/pr\¸t
t:2¸:\~ \\ rÚŵ/prpct^r 2tW~v~t:»t^vt:r¶:~v¶\c2ct^:tW~t^~ prtG rs ¾Et:º¼´sºt^~~tW~
2t:t^rt^nÁc{¸sº¶¶¶}Å` »\t:¶a» ¶¾r¶¸¸}d¶~Ut^rp2+t^ÚÅ~¸2:tÉ~¸r\¸tÇQ r rs¦~U~ &PA¶t:QU¼½
prtº~t^»\t:/©Et: tÁoprt^~tv\» QtW~/µt^tvt^¶}&t^:\rIQ^t^Î2'¼½s³¶IQ^t^'¶Q'Ç\ +¶¿}
sºt:¶^~:Å/µ/prptÎ 2~t^Ë:sº2t» ¶ 2~AÇQ r\ 2s@:2~ 2=¸\2~Aµ/¸prtÏrprt:WÅ~
rt^~¸¾Et^&¸ N¶¥:¤ OÁ
Û ¶»t^³prtW~tG³rEt:¶t^~^ŵ9tSs³a}ºr µ \~¹Uµ/prtprt:/ÇQ r\ rs ~}G~t^s³~^©¶~ 2tW~~
\ rÀ~t^:2\2t^¾E rÚtW~E2~t5¶sºtÁxÎttWt^\Úµ9¹p2\~n¾Et: 2t:Är¸\¶rpr¶~À¶~~ rtRJSCr
NT O H2~¼½\s³¶UQ:t^prtr¾2¸t^s ¼2sº¶r¸sºUQ:¶rÇQ r rs^tW~~À¶sºt9¾G}¶Q 2¶r´prtr¶2~
¼ V0'!.R-IW 2 V-X5'!.R-IW ¼½Ç\ 2 rsÍ~}~t:s³~:ÁYCr ³2\+Q~tWvE¶}Grsº¸·N¸sºt/¶\¸p2sº~
VH22sºÄG·t^¶a}&¸s³+'sUtW·Ãt^¶a}&¸s³5ÇQ r rs +~ 2¶2~¼½~}~t:s³~/µ/¸p
~Et^IH+E¶\¸tW~:ÅE~ +pÏ~t:t^~´2¶r\~^Á t^^Ågo~ 2p2¸}Q³t[Á"N¶¥[Z O tÄGt^2tW\Cr ~´µ\¹
¾G}Æ2\+Q~¶rd¶¸prs³~ ¼½H22r¸rds³ ÄG·Ãt:a}mr¶sºÇQ r rs):2~ 2=¸\2~vp2Arr¶}
~}~t:s³~´µ/¿pj¾r¸}©E¶\¸tW~:Á:G µ9t:»\t:WÅ2prt^~tvµ9¹~/¼½ +~´\jÇQ r\ rs²:2~ 2=¸\
µ/p2¸¶t'¸c¼Êº »\t:¶t^~E2~tA¸sºtt^+t^2r~ºcp2t'¸\¿prsp+ ³¶~º +~tWË~ºsv 2pc\~º¿
rt:Et:2r~ºprt´ÇQ r\ 2sÍ~}G~t^s ¿~t^¿¼Á ~2Å\\ rê¹Gr µ/¶t^\tÅ\r ~ 2}Up2\~ê~¼Ê9sº2tW
prt³t^~+\2~tº¸sºtA¼Î~t:·[s³p2¸rtºt:r¶: ¸\ɸ\¿prszµ/¸pdp2 ¼9¸~vÇQ r rsU·[¾+~t^
: rQt:2^Á
ÂÃjpr~2+t^^Ånµ9tºtÄr¶tprtº¶~~ 2tU¼¶sºr »G¶r©t^~E2~t ¸sºtº¼9t^r¶¶^ tWj~t:»G¶:t ¾G}
+~¶rÇQ r rsÈ~}~t:s³~^Å2ɵtA2\+Q~tº¾EpÆÎÇQ r\ 2s?\2~ 2=¸\É2mÎÇQ r rsU·[¾+~t^
~t:·[s³p2¸rtÎt:2¸:¶¶\¸prs mp2 ©t^2nÁ oprtÏÇQ r\ 2s@:2~ 2=¸\"¶~©t^~+tW¶¸}
~ r¸tWA¼½\~t:»t^~t:r¶¶^ t^A »\t:\sU2 ¶rº¶2~:Å~9¸¹\t^~r»aQ\tS¼Àp2t+ 2+µ9·
¶t:»\t:\pr¸t^pG}´¼p2t r+t:¸}G¶rrt:µ\¹gÁÀop2têt:r¶¶^ ¸\r\\t^¶»tW~n¼½sYp2t~+r
ÇQ r\ 2sv· ¾2\~tWmr t:r¶¶^ ¸\˶\¸prs'Å`2mEt: t^~v¸cr¶}ɵjrp2\~tW~ ¶Æ¼Ê¸¶ rt·[¼½t^t
^~t^~Ïp2¸t^»t³¶ µÍt^~+\2~tº¶sUt\Á ¸pr\ rpm»\t:}j~¸sºr¶tÅpr¶~vÇQ r rs+~ 2¶m+
t:2¸:¶v¸\¿prs,:s ¾2¸rtW\tp2t:r »tr »G¶t9¶ µt^t^~E2~t9¶sUtp2³t: 2¶
\=¸»\tt:2¸:¶&¸\¿prs'Á
oprt/t:s³¸2rt:5¼+pr~2+t^~5\rUQ:t^v~¼½\¸¶ µ~:Á5{Gt^¶³´r »Gt^~¾2\¹Q\\ r2 ¸¼½\·
s³¶&2At:Ä+Q~tW~`prtsUt^¶~9µ9t´ +~t 2r¸:¶ÚÅQrt:µ\¹³+©~}G~t^sÒ¸t^»t^¶~^Á`{Gt^¶ @
2tW~t^\~prtÇQ r rs%:2~ 2=¸\Ap2 µ9trt^~¸\rt^©³p2¸t^»tS¶ µ t^~+\2~tS¸sºtÁê{t^=¸\&¤
rt^~¸¾Et^~êprtt:r¶¶^ ¸\A¶¸prs µt¶sU2¸t^sUt^Qt^AUpr¶t:»\tSÇQ r rsU·[¾+~t^³~ tSs³p2¸rt
t:2¸:¶ÚÁË{GtW=¸\ ÏrtW~t^Q~v+t^¼½\s³+t©sºtW~ rt^sºt:Q~v¸c» ¸\ 2~t^~A H2 r ¸\2~^Å
:sº2tW³ºpr\~t¼5U~2r¶»tt^r¶¶^ ¶&r:[Á{t^=¸\$TU:2:¸ +t^~^Á
*, ³<5&.
"!$#%&(')
ÌÉtº+~t:©\t:rt^:¸¶t:Q·Ã~t:»t:SsUt^NÁC2´prtº~¹tv¼êa» ¶¶¾r¶¸¸}Î2'¼Ê r¸·[¶t:+tÅ
~t:»t^~gtt:r¶: t^~t:»\t: rtW~:Åa~2tW¸\t:\rpr:¸¶}/¶»t:~t¸:¶2~^ÁÚÌÉtê~~ rsºt
p2 :¸¶t:Q~\::t^~~¶r'©\¸»\t:d~t^»\t:tv r2¿¼½\sº¶}Ï~¶¾r t^j¸m~2:tU2jr¸t^¶Ïµ/¸p
t^~Et^=A~t^»\t:9t^r¸:\~:Á
ÌÉtsUt^Úv~t:»t^\~9Ut:t:sº¸2¶~´~ts³pr¶rtÁ`{Gt^»\t:9µ\¹G¶\\º¶~/^ t^\UQ:tWº¸Q
*
81;E,+++ ŵ/prQ~trtW~~¸2p2rt9prt~t^»\t:~ tÅ2 ER."E,+++ ŵ/p2\~t92tW~~¶r
¶t^a»\t^~Úprt~t^»\t:~ t r2p2rtWnÁÀÌÉt~~ rsºtêp2prtµ9¹G¸Q~ r¾rsº¿tW~t:»t^~ns³¸r¶}
:2~~~¼µ/¸ttWÇQ rt^~~^Á
- ./'0213546879&('
u¶¶t:Q~´2Î~t:»t^/t^r¸:\~t~rtWγ\nÅ+µ/pr¶pε9t»G¸t^µ ~ºµ9· ¶t:»t^npr¶t:pQ}\Å2~
~pr µ/d¶ C¶2Ám¥RJ prt©¶ µt^¸t^»t^ê~¼½sºt^ɾG}j2Grt^~\\ rEt^ɸQ S~[Lprt© rrEt: ¶t:»\t:
:2~~~¼ ´~/¸Qt:2rt^t^©p2\ rp&¸\rºp2 rÀrtµ9¹~:Á
opr~ºpr¶t:pr:9»G¸t^µ%sºt:~UµÉrEt:¶t^~Ut:\r¸rÉ\sUsv rr:¶crt:a}G~^Á C¶~^Å
:sºs 2r¶^ ¸\t^¶a}~¾Etµ9t:t^'µºrtW~/r¶}&rt:Et:2'\&prt S~/p2prt2Grt^~/¾Et:¶r
2Á`{GtW+nÅ\¶:E:sºs r2¶^ ¶&t:a}~;:½µ/¸p2¸' <t»\t:}U~pr92³ prt:\2~Q^Å
µ/p2t:t^~~Q\sºs rr:¶'t:a}~=:½¾Etµ9t:t^©µ9 ´~><t¶r³2&~ r¾tW=/´¸t:WÁ
-? @BA(CDE')7 7F&;')
G H%À¬ Ô Ø Ù Ô % I ÌÉtU\~~ rsºtvp2~t:»t^St:r¶¶^~´2j¶¸t^\~´^jtÄ+t^¶t:+tv:\~pr·N¼Ê¸¶ rt^~^Å
p2 ¶~^Åprt:}Us³a}vt-H2r¸¶»t:¶}º~ºEt: ¸rê~\sºt/2sÍ+\¸Q궺¶sºtÁ5ÌÉtS~~ 2sUtp2
prtrt:µ\¹©sºa}t:a}©\¶\~tsUtW~~\t^~9p2 ´:¸¶t:Q~S+~t:»t^~t:Ärp2r\tÅr¾2 *CÂCwt:
JLK JNM
Lower Level Upper Level
Local Area Network Local Area Network Local Area Network
Wide Area Network
C¸\ rtº¥5J5oµ9·¸t^»t:grt:µ\¹³p2¸t^pG}
~rt^~t:»tWA¾Etµ9t:t:ÎQ}³+¶/¼5+t^t:~:Á
Õ ®¬ ØgÕ Ø Ù Ô % I ÌÉt:2~t:prtsºt:n¼À2¶g~}G2prG}Ur\+Q~tWA¶ Në
O
ŵ/prp
~¶r r t^~vp+ ^Å`¼½ºt:»\t:}Ét:Ät^ r¶ÚÅprt:t't©¾E r22~v\ËrtW~~U~Et:tWr~º2ËÆsºtW~~\t
2~sU~~¶É¶sºt^~^Å`¸pr\ rpdprt^~tA¾E r+r~vt©r ¹Gr µ/Æ2dprt:}d2¸}Éprm ¼t:º~\sºt
2r¹Gr µ/U¶sUt\Á`opr~`sºt:E¶¸ µ~ +~ ¸sºr¶t:sºt:Q -,-8W'U<-7- ¼Ê¸¶ rtStt^;N¤ OÅ
µ/p2¶p¸ rÎ^¾+t 2~t^©A~¶»t´p2t+~t^2~ +~r\¾r¶t:s'Á
#%&5' >,-n`
Û ¶»t^ÎA~t^»\t:2prtv~t:S¼¸~t:2¸:\~:År\ r\\À¶~©tW~¶Î³ÇQ r\ rs;\2~ 2=¸\&p2
sº¶r¶sUUQ:tW~´prtU~t^»\t:St^~E2~t¶sºt ¼½\´t^»t^}ζ¶t:Q^Ågµ/prt^t^»t:´¿~S¶: t^nÁÂÃQ r¸¶»t^¸}\Ågprt
:2~ 2¶Us 2~`¾+t~ 2pUp+ ¼½êt:»t^}:¸¶t:QWÅp2t:t/tĶ~~êÇQ r\ 2sp+ ê\Q¶2~5\r¸} prt
~t:»t^t^r¸:\~p+ St¶\~t^~/ºprt :¸¶t:Q^Á9ÌÉt:H2~St:»G¶t:µt:Ķ~¶rAÇQ r rs +~ 2¶2~
2©prt^¸t:¶t:» 2t´³\ r/r¾r¶t:s'År+Aprt:'µ9trt^~¸¾Et r~¶ ¶'+&¸~ ¸\2¶tÁ
? "7 7F ! @B 4; (4'C
° 2 rsÈ:2~ 2¶2~v:˾+t'¶»QtWƶ\µ9j^ t^\¶t^~^ÁÏx¿}:ʶ~2sUtWÆ»\¶r <
ÇQ r\ 2s ~}~t:s³~:Ågµ/prt:prt^~¶sU2¸tU~¶ N¶¥^
O
\Sµ9t:¶pQtW~S¸ Në
O
Ågµt^t ¸Q 2:t^ H2~+
s³¹tU rÏprt(H2~:t:\}ÁÂÃjprtW~tW~~}~t^sº~^Ånt^\pj~t:»t^´t:r¶:A~~~¸\rt^G rsv¾+t^¼
»\t^~^ÅÀ2ÉG}Ï~t¼9t:2¸:\~\ prt:¶rs³ \¸}¼êp2tA~ rs?¼»t^~~¾G}j:2~ 2=¸\
ÇQ r rs'Áº ¸pr\ rpd~¶sºr¶tÅÀ»\¸r'ÇQ r rsz~}~t:s³~tU\=¶»tU¾Et^: 2~t prt:}jp2a»tvprt
¾Et^~a» ¶¶¾r¸¶¸}ÁAw´Ïprt³prt:p22nÅnprt:}j~'tÄpr¶¾r¸&tºÇQ r rsÑ~UQ:tºµ/prpj¶sº2¶~
Et:¼½s³2:t´Q¶2~¶&t:s³~¼¶\\&2sºt^~~tsºr¶tĸ}Á
oprt^ÚÅ&N¶¥ @,O ~pr µtWp22¶QÇQ r\ rs ~}~t:s³~^¾Et9tW 2tW´»¶r´ÇQ r\ rs ~}G~t^s³~:Å
µ/p2¶p'¶t^Aprt rt^~¸\¼rt:µ ÇQ r rs +~ 2¶2~^Á S¸¼p2t^~t \2~ +=¶+~¼Ê¶À¶'prt
~t^:2m^ t:\}ϸmp2 prt^}Ét:¶}ÉÆÎ»G¶ 2t:\sºt:rt^sUt^Q¼S~t^»\t:t:r¶:~
º¾r r¶¶¶\t:~tW=¸r³~t~:Á`opG 2~^ÅrÇQ r rs ~}~t:s³~/:&¾Et¾r r¶¿´ 2~¶r(H2r¸t2t^=¸»\t´2¶rt^~
N¶¥^£ OÅGµ9·Ã¶sUt^2~¸\A¶2~*N¶¥ O \t^t^~ Në OÅG³¸t ¼½t^µ ¼prt^s'Á`oprt^~tÇQ r\ 2s%~}~t:s³~9p2a»\t
&~s³¸¶t:Ç\ 2 rs ~UQ:tp2j»¶r©ÇQ r rs;~}~t^sº~^Ån2Ï~\sUtv~\rpr~¶^ tW\2~ 2=¸\2~:Å
~ 2pº~: rsv¾r¶¸rµ¶¶~%N¶¥[T OÅp+a»t¸s³¸QU2v¸s³2a» ¶¶¾r¸¶¸}µ/¿pUt^~Et^=prt^¸
ÇQ r\ 2s ~UQ:t\Á
ÂÃm2¿¸\ÚÅÇQ r\ 2s?\2~ 2=¸\2~¼¾+pm:t:\¸tW~:d¾EtA\s ¾r¶rt^Ï'¼½\s?pr¶t:·
p2¶^SÇQ r\ rs@~}~t^sº~^Å/~Art^~¸¾Et^"¶ N OÁ oprt\\/¼pr¶t:pr¶^:2~ 2=¸\2~º~³
t^r 2tÇ\ 2 rs¦~UQ:tµ/pr¶¸ts³¶\¶r¶r¼Ê¸³a» ¶¶¾r¶¸¸}Å\U:G»t^~t:¶}Å5¶2t^\~t©a» ¸¾r¶¶¿}
µ/p2¸¶tsº¸Q¶r¸2v¼Ê¸ÇQ r\ 2s ~UQ:t\Á
ÌÉt'¾2~t:»t³p2 ³\2~ 2=¸rÉÇQ r rs³~ ¼p2t H+~U^ t^\}Ér¶}mt^ÇQ r¶tW~Ï\prt^U
\¸»\t:G rsv¾+t^&¼ ~t^»\t:©t:2¸:\~ :½\ƶ»t:Q 2s ¾Et:¼»tW~A¶ p2tjsº\~&\t:rt^´^~t <=Å
µ/p+ t^»t^Ap2t^~tjt:r¶:~tÁÒopr~s³¹tW~©¸'t^~} 2~tj~sºtÉ\r¸¶2:¸t^¶ÀÅS~ +p
\~v:sºs 2r¶^ ¸\Ët:a}Åêµ/prt: ~t^¸tW=¶rÏprtÎ~t:»t:vt^r¶¶^~ p2Us³¹\t© r jÇQ r\ 2sÁ9}
:Q~^ÅÇQ r\ 2sº~5¼+prtS~t^:2U:t:\}t/+~ 2tW ¾G} +~¶rµt^¸¸·ÃtH22t^v r¶tÅ\µ/prp
2t^»t:Q/~t^»\t:t:r¶¶^~9¼½s%¾Et:¶rU¼½t:t:¶}&p2\~t:&\::¸2 º~ 2pÎ2¿¸\2Ú:¸t^¶ÀÁ
?- @ " ( "!
op2tv~¶ ¶µ9tº\´~´Aµ·[¶t:»\t:sº¿}'ÇQ r rs²~}G~t^s;sºrEt^\prtvp2¸t^pG}©¼
prtA r+t:¸}G¶rÎrt:µ\¹gÁ Û ¶»t^dÎ~t:»t^t^r¶¶^ tWÉm~t^»t^ ´~ 2Ém~t^»t^rt^~
¶Ét^p ÅÚ'ÇQ r rs²~´¾2 r¸¸¾G}\2~t^¶r' :½¶t³¾rtW¹G¶r <sº¿}μ`p2t ´~prt
~t:»t^¶~St^r¸:tW©\ÚÅ+2\ prt:¶r³ :¶t¾rt^¹G¸r <sº¿}©¼`~t^»\t:t:r¶¶^~/¶'t^\p¼
prt^~t S~^Áopr~ê\2~ 2=¸\Ut^2~ rtW~p2 G} µ9ÇQ r rs³~¶\t:~tW= ¸tW~êrtSrtÅ
µ/p2¸¶t¹t:t^r¶rUprt~IQ^t¼prt¶Qt:~t^=¸\\~~s³¶À\~+Q~~¸¾2¸t\Á
C¶2ÁÉÏ~p2 µ~ mt:Ärsºr¶t©¼S~ 2pË:2~ 2¶d¼½UÏ~t:»t^t^r¶¶^ tWdmprt:t
´~:Á
{t:»t:9t:r¶¶^~9tS¶: t^©©2Grt^~¥¥2Ï¥ @ ¼ ¥ÅG&rt^~r¥Åv2 @ ¼ Å
2Î\rt @5@ ¼ @ Áoprt"H+ rt t^r¶~´µ9©ÇQ r rs³~-J9prt"H+~ÇQ r\ rs;~22~ S~
¥³2mÅ2d\\¶2~p2t©~t^»\t:t:r¶¶^~prQ~t^j¾G}ÏrtW~³¥¥\Åê¥ @ Å¥³2mÁ&oprtA~tW\2
ÇQ r\ 2s;~22~ ´~©2 @ Ån+Ï\Q¶2~prt³~t:»t:St^r¸:\~Sp2\~t^ϾG}2Grt^~rÅg @ +
@5@ Áoprtµ9ºÇQ r\ 2sº~¶Qt^~t^//rt rÁ
oprt9t^~¼½pr~\2~ +=¶ ¶~µ9¼½nÁ C¶~WÅprG\~¸rSprt9s³ \¸} r¶tprt rrEt:
¶t:»\t:`¼prtApr¶t:pG}¶¸ µ~~t^¸tW=prt S~p2vt³¶\~t^~G}j¶»t:d¶¶t:Q µ/prt^
:2~ 2¶rÇQ r rs'ÅpG 2~t^2~ r¶r\¶s³9::t^~~¶sºtA »t^ ¶ÇQ r\ 2s?\2~ 2=¸\2~:Á
{t^\2nÅQ 2~¸rU ÇQ r\ rsÒ:2~ 2=¸\³ ¼½ rprt^9~r¶¿/~t^»\t:`t:2¸:\~êp2t ¶t:»\t:g¶¸ µ~
v¾2 r¸Ç\ 2 rs³~êµ/¸p'v~s³¶¸t^/~UQ:t\ÁÌÉtQ¶©prG\~t´vs³ ¸}º:2~ 2¶©9pr~¶t:»\t:
¾Et^^ 2~t¸ r=¶:t©prt'G rs ¾Et:º¼~t^»\t:Ut^r¶¶^~v+t^ ;GtW~v2³tÄrt^t^cϼ½t^µ r2¿~:Å
s³¹Q¶r¸ r2rt^:t^~~} t^~ :sºr¶tijÇQ r rsÍ:2~ 2¶2~µ/¸p&~s³¸EÇ\ 2 rsÍ~IQ^tÁ
678'9:!;< =4
op2tU~t·s³pr¶rtvt^r¸:¶Ï¸\¿prs;µtUrE\~tv:s ¾2¸rtW~prt³~22ÇQ r rsU·[¾+~t^
2 Ut:r¶: ¸\&¸\¿prs N¸¥\¥
O
2St^~t^t^r¹ ~ÇQ r rsU·¾2~t^³sv +Út:Ä:¸ +~¶¶¸prs
N¶¥W
O
ÁÉÌjt»G¸t^µ prt'~t:»t:U~ t&~U2¸tWt©¼Sr Îp2 ³¶¸t^Q~v rE2 tµ/prt^Ë~ 2¾rsº¿¶rÉ
JLK JNM
N33
N11
N21 N22 N23
N13 LAN 1
LAN 2 LAN 3
C¶ rt&JoÀµ9·¶t:»t^¶~9s³ \¸}AÇQ r rs ~}~t:s
t^ÇQ rtW~WÅ92cµtÎ 2~tprtjÇ\ 2 rsU·[¾2\~tWËrÉt:r¶¶^ ¸\ ¸\¿prs¦d¹\t:t: \¹m¼~ t
2ErtW~´\:\~~/t^ÇQ rtW~~:Á S~¶¸t^\tWÇQ rt^~~Ssºa}'¾Et ¶Qt:¶t^a»tWnÅ2µ9t +~t prtºÇQ r rsU·[¾+~t^
sv +êt:Är¶ 2~¶m¸\¿prszÏ sº¶^¶¸}jtWm2dµ/¸t³¾+¹jprt©~t:»t^ 2t:Q ~ tA¶
~sºtÇQ r rs 2©pG 2~/t^2~ 2t
'! <W+-M-88W Á
«^¬ Ô %HN®« HØ % Ø ¬ HÊ« I pA2 t^r¸:»a¸ 2t¶~`tWºµ/¿pA»t^~¶ºG rs ¾Et:
rt:r¶r©¸~´¼½t^~prrt^~~^ÁSÂÃÏ\rt:&t^\Îr rÅn©:¸¶t:QÇQ rt:¸tW~S~sºtvÇQ r\ 2s7©\t~t´¼
» ¸ rta»\t:~¶ÎG rsv¾+t^+¶~^ÅÚ2Ït:¸2~\~Sprtº rt^Q» ¶ rtU¼prt³r©prtUrtºµ/¸pjprt
p2¸\prt^~`»t^~¸\vG rsv¾+t^^ÁÂót^5µ/¿tr rÅQ¶¸t^Q`+t^¼½\s³~5µ9~t:2~[J H2~^Å\¿tWvprt
~t¼/»a¸ 2ta»\t:~¶jG rs ¾Et:+¶~¼½sÈ~\sºtAÇ\ 2 rs'Å+mprG\~t^~'rt^µÍ»\t:~¶ÉG rs ¾Et:
\tW t:`p2G}º¼ÚprttW³»\t:~¶³G rsv¾+t^~[LQprt^&¸ rE2 tW~/¶+¼Úp2t2 t^r¶¶^~¸&prt
ÇQ r\ 2sÒUp2trt:µY» ¶ rt2©prt2t:µY»t:~¸\©G rs ¾Et:WÁ
$ « % Ô ® %ª HØ % Ø ¬ HÊ« I tW~t:t:r¹\ ~¸\¿prs ¶~5´+t^sº~~¶·¾2~t^ ¶¸prs
p2 /t^¶»tW~ê¼½s%xÏt:¹ aµ ~DN¶¥^£ OÁ5ÂÃ&\rt:U¾¶Ap2t¶¹gÅGU¶¶t:Q~t:+r~ I M, sºtW~~\t
v~sºtSÇQ r\ rs'ÅG2³prt^Aµ9¿~¼½\9¸g~t^»\t:`t:2¸:\~`¶³prtÇQ r rsÍv~t:+³¾2¹º
=RE17.
sºtW~~\tÁÂÃ&\t^êºt^¸tW~tp2t¶¹gÅ\prt¶¸t^Q/~¶sU2¸}©~t:+r~
E-I,+-
sUtW~~\tvprt~sºt
ÇQ r\ 2sz¿\¾¸2t^jprt³¶¹¼½s'ÁAoÎrt^»t^\tWG· ¶G¹'¼½s?: 2¸2&µ/prt^m+ rt^Q
¶¹tWÇ\ 2t^~~Àt9~ 2¾rsº¿t^ÚÅt^\ptWÇQ rt^~~Ú¸sºtW~sº+tWnÅ2prt9¸\¿prsYt:2~ rt^~Àpr r\p
prt 2~t¼ 8 +-&8E
2
WI.
sºt^~~tW~Úp2 `:2: rt:Q¸¹t^ÇQ rt^~~5tQtW¶ ¸sºt^~sº
\rt:WÁ
§%
Ø ¬ H½« ˪v®
Ø Ø
ª H½« HØ I
Ìjt\¾2~t:»t´p2S¾+pζ\¸p2sº~\+t^t¶'µºrp2\~tW~/¶
prtt^rt:Ú^~tÅ22©p2prtW~t2p2~t^~p2a»\tsºpr¶rº¸sº¶r2ÁÌjt^prt^t:¼½tsUt^\tprt
µ9v¸\¿prs³~`¾G}º »t^¶\\¸2prtsºt^2¸rv¼Úsºt^~~tW~:ÅQ¶2~t^\³¼:sv¾r¸2¸rprt^s ~tWÇQ rt:·
¶¸¶}Á`op2¶~t^~ r¸~¶'º:2:¶~tµ·[rp+~t~t·s³pr¶rtt^r¶¶^ ¶'¶\¸p2sÁ
ÂÃÏprtºt:s³¶r¶rA¼êpr~~tW=¶ÀÅnµt"H2~rtW~t^Q´pr µÇQ r\ rs³~tv~t:¶t^t^Ͼ+t:¼½tv~ r¾·
sº¸¸rÉÎt^ÇQ rt^~^Åprt:Ƶ9t©t:¸9µ/¸t&2dt^\dtWÇ\ 2t^~~2tW~~¶rθƼʶ¸ rt·[¼½t:tA:\~tW~:Å
2 H22¶¶}Aµtt:Ärsº¸2t´prtp22¶¸rº¼À¼Ê¶¶ rtW~:Á
; A!; 79 (54;7 @ ') ' E"!
° 2 rs¦~t^¸tW=¶c¶~UEt:¼½sºt^Æ\Æprt:¸¶t:QA~¶t'+Æ:: r~vt:»\t:}ɶsUtÎɶ¸t^Qº~t:22~U
t^ÇQ rtW~/ºprtv~t^»\t:WÁ`oprt ~t:¶t^=¸\'r:t^~~¶~/¾+~t^'¶G^À2 Up2 Sprt¼Ê¸¶ rt tt^
¶2~2t³ rrr¶rÎ\mt^pm:¸¶t:Qrt©r}Q+sº¶^¶¸}É\sU2 tW~2d¹\t:t^2~ r·[·2 t\Á&oprtW~t
2 +~~`¶Art:µ\¹Ut:a}~¾Etµ9t:t:ºprt´:¸¶t:Q2ºt^\p pr\~¶r ~t:»t^t^r¶¶^~^Å\+
prtt^~¶s³ tWv~t=:M< \ .5
* <5¼Et^\pv~t:»t:t:r¶¶^~^Á´~¶rpr~5¶¼½s³ ¶ÀÅÇQ r\ 2s ¶~
:Q 2t^A¾Q}A~t:¶t^¶r ´~p2t¶\~t^~ prt:¸¶t:Q/2©¾G}A+~t:¶r \r¸}Upr\~t´¼
prt~t^»\t:t:r¶¶^~êp2St´t:EtW³U¾Et M< Á̸p2¸'t^\p Å+~t^»\t:9t^r¸:\~tprQ~t^
2\sÒ³t^2~ 2t¶\¾2¶2¶r+Á`/t¼Á*NZ O ~p2 µ~9p2/pr~2rQp¶~/prtsº\~tPA:¸t^\
¶t:s³~S¼êtW~E+~t ¶sUtº+r:a» ¶¶¾r¸¶¸}µ/p2t:Ï +~¶r&©t^r¶¶^ ¶Ïr\Àp2
¶¹~92 rr¶¹G~t:2¸:\~:Á
;- 4 E'' ;')C 4 'C C !
À« %HÔ I C¶2Á @ \ ¶¸2t^~³prtÏ2tW~~¶rm¼mµ/¿tt^ÇQ rt^~^ŵ/prpt^ÇQ r¶tW~ºµËrp2~t^~
2ɼ½ 2sºt^~~t^~[JÅÅ !!"$#Å$%&"'UÁ Cr t^pdsºt^~~t\ÅÚprt H2\ rt
¶2:tW~ ¶Æ¾r¹\t~ ¿~ sºtWr¶rÏt:Q¶r'prt'rt:2¸:¶mrG:92dprts 2
t:Är¶ 2~¶rG:[ÅGtW~Et^¶»t:¶}Á
oprtA:¸¶t:QH2~ ~t^2r~(,sºtW~~\tvÎt^\pÉ~t:»t^t:2¸:¶dprtAÇQ r\ rszp2 ¿
p+~ê~t:¶t^tWnÁoprt)sºt^~~tS:Q¸+~p2t´~t:»t:·¸t^»t:t^ÇQ rtW~2tW~~^ÅG~`µ9t:¶g~ê
¶¹ t^ÇQ rtW~WÅ\+Uprt/¸sºtW~sººp2 r2¶ÇQ rt^¸}U¶rt:QIH2t^~ê¿WÁ5Ìprt:A~t:»t^t^r¸:\~t^t^¸»\tprt
"sºt^~~t\Å prt^}v2tW~~prt´t:s ¾Et^rt^º¶G¹Ut^ÇQ rt^~\:\r¸rprtSs 2EtÄr¶ 2~¸\
¸\¿prs'ÁÆÌprt^"É~t:»t:Ut:r¶¶^Ït^»t^\ 2¶¶}mQ~U^t^~~vÉp2t':¸¶t:QWÅ긩~t^2r~U¾2¹Æ
Lsºt^~~t\Å2+:2~t:~¿~t^¿¼¶¹t^&¼½\pr~S:¸¶t:Q^Áoprt*Lsºt^~~t:Q¸2~
prtA 2t:Q ~ t³¼9prt©t^r¸:2ÅÀ¶2µ/¸pm¸~ »t^~¸\ÏG rsv¾+t^^Å5~µ9t:¶ê~prtA\Qt:r¶}
¾Et^¶rÎprt©¶¹jt^ÇQ rtW~ ¶sUtW~sºÚÁjÌprt:mprt&:¸¶t:QUp2~v¶¶t^=t^Ƹ!$+ sºt^~~tW~
¼½s7p2tvÇQ r rs'Å2¸´t¶2~\~/p2tv 2t:QS~t:»t:´~ tprtvrt¾Et^¶r³prtpr¶p2t^~´»t^~¶
G rsv¾+t^^Á
oprt:¸¶t:Qprt^~t:+r~9¾2¹A, !)"$# sºtW~~\tUprtÇ\ 2 rs'ÅQµ/p2¶p'\\¶2~prt: r·
t:Qê~t:»t^`~ t2º¸~`»\t:~¶UG rs ¾Et:WÅ\¸\rµ/¿pAt:¶t^~ttWÇQ rt^~^ÁÌprt:A~t:»t^t^r¶¶^
t^:t:¶»t³- !)"$#zsºt^~~tÅ¿¶2~¸~p2t©~Et^:FH+t^m~ tA~prt© 2t:Q ~t:»t:~tA+
2tW~~t^~prtAt^ÇQ rtW~ \¶2~p2v~ tÁ'oprt:dprt©~t:»t:t:2¸:' r2¸¹~¸~t:¸¼/2d~t^2r~
¾+¹d./%&"'7sºt^~~t\ŵ/prpÆ\r¸}m:Q¸2~prt'~t^»\t:·[¶t:»\t:t:r¶}Á C¶2¸¶}Å5prt&~t:»t^
t:2¸:vrG:t:tWr~9ºprtrtÄGEt:2¶r0YtWÇQ rt^~^ÅG¸¼G}Á
JLK JNM
Phase 1
Phase 2
Server Client
RESULT [reply, −]
Time the server is locked Time to acquire lock
Time to process request WRITE [request, lock+timestamp]
STATE [state+version, granted+timestamp]
APPLY [state+version, release]
C¶ rt
@
J/t^r¸:¶'rG:g »t^»G¶t:µ"·êÌÉ¿tt^ÇQ rtW~
Ø ® À¬a¬ Ô « × ¬ H½« Ô ª I C¶2Á¤&t:¶~prt rEt: ¸\θÉ:~t¼ê+ rt^QSµ/¸t
t^ÇQ rtW~~À~t:QÀ¾G}´µS:¸¶t:Q~:Åu´¥2 uÁu´¥`¶\t:~Úµ/¸pprt9Ç\ 2 rsL¼prt:t~t^»\t:Út:r¶:~
{n¥Å2{rU+{ @ Å2+'uv¶\t:~9µ/¿pÎU~t^:2ÇQ r rs'ÅrSt:r=t^nŵ/p2¶p'¶\t:~tW=~µ/¸p
prt H2~ÇQ r\ 2s \t:r¶¶^&{rrÁ´Ìp2¸¶tº{2³~´t^r\tW'¸mu´¥&~Sµ/¿tÅg¸t^:t:¶»t^~´ ´¶sUt
t 1
sºt^~~t¼½szuÅEµ/prpεt \~~ rsºt¾Et^~A¸ µ9t:/¸sºtW~sºÎp+Éu´¥ ~/µ/¸tÁSÂÃ
\rt:Îa»\¶j+t:Q¶`t^\G·[¶¹gÅ{r©prt^t:¼½t ¶t^~'tW:¶5p2t³+t^sº~~¸\ϸQtW
'u´¥¾G}Î~t:2¶rA¸.&% sºt^~~t\Á/Ìprt:mu´¥t^t^¸»\t^~prt &% YsUtW~~\tÅE¿
~~¸¶5¸jp2t"H2~rp+~tv¼ê¿~µ/¸t\Ågp2~¿p2\~r\¸¶t^t^j¶+$! sºt^~~tW~}t:^Á
op2t:t¼½tÅu´¥prt^U~t:+r~5¾2¹#* )"Lsºt^~~tê{rÅa¼½\\t~¾+\ prt $+msºtW~~\t
{2Up2~5 +~´~t:QWÅ++'µ9¿~¼½´{2vA~t:2'¸´Q¶ÚÁêÌp2t:juº\sºr¸t:tW~/¿~µ/¿tÅg{rº~t^2r~
©u´¥º2t:µ $!sºtW~~\tÅGt 2t^¶r©u ~µ/¸t\År2u´¥rt:tWr~~r\s³NÁ
;-? ' & ' ('CD 4')C C !!
9}mt-H2r¸¶ÚÅprtr:t^~~¶r¼´t^\mtWÇQ rt^~UGtW~ rºp+r\t³prt'~t:»t:U~ tÁËu2~t·
ÇQ rt^Q¶}Å9¿A~³rArt^:t^~~}ÉÆ\sU:¸¶}Æt^\Ë+cµ/¸tξ2\¹ÆdsUIH2tWc»\t:~¶c¼prt
~tU¸É~\sºtUÇQ r\ rs'Ånµ/prpÏt^¸¶sº¸+ tW~´prtºrt^t^¼`p2tU¶¹G¸r&sºt^p2r¶~s² 2~t^¼½µ/¸t
t^ÇQ rtW~~:Áoprt/rt^~~¸r¼Et^\ t^ÇQ rtW~pQ +~r¶}¹\t^~52t/rp2~t/2º~t:»\t:t^vt^ÇQ rt^~~
^tÄt^: t:2: rt^\¸}³µ/¸pUµ/¿ttWÇQ rt^~^Ŷsºr »G¶rº¾+p't^&t^ÇQ rt^~~t^~E2~tS¸sºt
2& »t:¶gtWÇQ rt^~pr r\prr WÁ
Quorum of server replicas
S2 S3
C2
INQUIRE
YIELD S1
STATE
WRITE
STATE
RESULT APPLY C1
C1−S normal messages C2−S messages and resulting messages on C1
C¸\ rt¤0J`u¶¶t:Q¸Qt:=¸2Uµ/¸pκ~t:»t:/ÇQ r\ rs%·9u2: rt:Qµ/¸tW~
; ( ;4')C/ !;&;!!
op2tv¶¸»\t:rtW~~2Ï~ ¼½t:}μêprtºt^r¶¶^ ¶j¶¸prs²¶prtºrtW~t^2tU¼¼Ê¶¸ 2tW~St:¶}Îjprt
t^»t^\ 2¶¶}\· Et:¼½t^=¼Ê¸¶ rtSt:t^\`p+ r2~êAt^»t^}vrtÁ G µ9t:»\t:Wų¶sº+\QêE¶Qê
:2~t:¶~p+ 9prt´¼Ê¶¸ 2ttt^9s³a}ºsº¹t´sU~¹\t^~92©t:\rt:\ 2~¸}vt^+\v:¸¶t:Q\9
~t:»t^t:2¸:º~p+a»Q¶rU¼Ê¶¶t^nÁ
Ô
¬
Ô
¬Ï¬
Ô
%-HÊ®2ªH%¬
Ô I
Ìp2t:d&~t:»t^´t:r¶:³¼Ê¸~ r¶r©prtºrG:t^~~¶r&¼&µ/¸t
t^ÇQ rtW~WÅÚprtA¶¸»\t:rtW~~¼prt©¸\¿prsz~t:+~ rt^ɾG}prt³¼Ê¸¶ rtAtt^=Wŵ/p2¶pdt:»\t:Q +¶¸}
t:E~Àp2tê¼Ê¸¶ rtêSprt/¶¸t^\WÁoprt/:¸¶t:Qprt^v:2t^Qprt rt:Qµ/¸t9¾Q}~t:2r¸r~Et^:FH+
!"ËsUtW~~\tS³t:»\t:}A~t^»\t:t^r¶¶^~9¸prtÇQ r rs'År2&t:Et^ t^¶}At¶t^~9prt~sºt
µ/¸t\+t^¶º³~ 2::t^~~¶»tÇQ r\ rs³~ 2\¸+p2tEt: ¸\º:sºr¶tt^~ê~ 2^t^~~¼½ r¸¶}Á5 S~~ rsº¸r
prt:tSt:Ķ~~ ¶t^\~rt´¸¶»tÇ\ 2 rs'Å\~ 2tW~~¼½ rgsºr¶t¸\A~ê\ 2Qt^t^vvt:»\t:Q 2¸¶}UG^ r
¾Et^^ 2~tp2tG rs ¾Et:`¼n+Q~~¸¾2¸tÇQ r\ rs³~~YH22¿tÅG2nÅ\¾G}Ut-H2r¸¶ÚÅ\p2t/¼Ê¶¸ 2tt:tW=\`µ/¶¸
t^»t^\ 2¶¶}A~'t^+\¸2 ¼Ê¶~t¼Ê¶¸ rt^~^Á
JLK JNM
request
Client Server replicas
Leader Backup 1 Backup 2
reply
C¸\ rt J` /=¸»\tt:2¸:¶r\
%HÔ « [ H%¬
Ô I
Ìprt^A¶¸t^Q¼Ê¶¶~´ r¸2A³µ/¸t\+t^¶ÚÅE~\sºt~t^»\t:/t:r¶¶^~/s³a}
t:s³¸m¶¹t^dµ¸¶rμ½\ prt !)"$#?sºtW~~\tºp2v¶2: t^~vµ/prpË~ tAÏ 2~t©¼½vprt
µ/¸tÁdoprt~¼½t}d¼/prt¸\¿prsÈtWÇ\ 2¸t^~p2vprt^~t©t:r¶:~vÏ2º rr¶G¹Éprt:s³~t:¶»t^~
2\¸Eprt:}A\:ÇQ r¶tSprtsº\~ r·[· 2 t~t^»\t:9~ t\ÁopQ +~:ÅGµ/prt^Ap2tS¼Ê¸¶ rtt:tW=\t:E~
϶¸t^\º~ p2a»G¸r'¼Ê¸¶t^ÚÅ`~t^»\t:t:r¶¶^~v~vÎt^ »\t:}rt^r rt³¶Æ\t^Étt:sº¸rt
prtrt:µ9t^~U~t&jrr¶}Ámopr~Ur:t^ rt©:2~¶~~v¶ËtÄtW ¸rjp2t:2~t:2~ 2~v¶¸prs
rt^~¸¾Et^¶ Në O sºr³¸À~t:»t^t^r¸:\~:ÅtWp'~t^»\t:/t:r¶:UrE\~¸rº¸~: rt^\~tÁ¼
prt³t^ »\t:}&2tW rtºµ9\~´ 22prt^j¾Et^: 2~tU¼&¼Ê¶~t³tt^=¸\j¼prtA¶¶t:Q¼Ê¶¸ rtÅÀprt
~t:»t^Àt:r¶:t:2¸¶t^~µ/¸pv t^t´prt &)!"$# sºt^~~têp2prt:}t^»t:Q 2¶¸}t^:t:¶»tÅ
2©prt:¸¶t:Qt^~~prt rt:Q9µ/¸tEt: ¸\ÚÁ
³8j5:/vÆ6 /-À-
op2tUÇQ r\ rs²:2~ 2=¸\Ï+Îp2tUt:2¸:¶Ï¸\¿prs²t^~¸¾Et^¸jpr~2Et:p2a»\t ¾Et:t^
¶sºr¶t:sºt:QtWj\~2¼êprtS}G2sUº \t:Qt:r¶¶^ ¸\jtt^2~¸\ :S S <r¶¼½\s N¶¥W OÁ
S SÒ~A¼½sºt^µ\¹¼½rt^~¸\r¸2©¼Ê r¿·N¶t:Qt^Q· ¾2~t^rr¶¶^ ¸\2~:Ånµ/p2¶pj\t~
s³\~~¸»\t©\t:Q³~}~t:s³~Ut:2¸ }\t^m »\t: ¶2Ïp2 rrtµ9¹~:ÁdºtW¶¸}dr »G¶rt^~Urr¶: ¸\
rt:»t^¸\rEt:~µ/¸pcÏt^ rU¶»t©t:2¸:¶Ær[Å5µ/prpƵt 2~t&\~vÏt:¼½t:t:2:t©E¶\
¶Î 2tÄEt:¸sºt:Q~:Áopr~rG:Ú¶sºr¶t:sºt:Q~´³¶t^rt: a¾2\¹Q 2&~prt:sºt¶Îµ/prp'prt¶t^\t:
t:2¸:º~t:¶¸UQ:tW~¶2:sº¸2³tWÇ\ 2t^~~¾Et¼½t¼½µ¸2p2t:s7³¾2\¹Q 2&t:r¶:~^År~/~pr µ/'¶
C¶2Á`Á
Virtual node 12
Virtual node 1m Virtual node 11
256 Mb RAM P IV 1.4 GHz
256 Mb RAM P IV 1.4 GHz
256 Mb RAM P IV 1.4 GHz
DummyNet
Router
Linux PC 2 Linux PC n FreeBSD PC
Virtual node 21
Virtual node 2m
Virtual node n2 Virtual node 22
Virtual node nm
Virtual LAN 2 Vitual LAN n
Virtual LAN 1
Virtual node n1
Linux PC 1
128 Mb RAM P III 600 MHz
100 Mbps switched Ethernet
C¶ rtT J`ot^~ê¶¼½s
=' 4 79')! EC @ '0 !
§ %HN®« HØ I Cr\³p2tÏr rE\~tμp2t+t^¼½\s³2t't^~~^ŵtÎ +~tÏË rsºs }2r¸:¶ÚÅ
µ/p2¶p\2~¶~~³¼m~¶r¶tÎt^r¶¶^ tW"~t^»\t:WÅ^tW~~t^c¾G}c~t:»t^¶¸t^Q~^ÁLoprt~¼µt'¶~
¶sºr¶t:sºt:QtWɸQa»a+jt:Ät^ rt^É 2~¶r{G r~Qa»a'{ ÅÀ»\t:~¶c¥Á¤2Á¶¥ÅÀ2Éxζ~ +~tW
¼½\\sUsv rr:¶'~ r2+\WÁ
oprt´~t^»\t:5µ\¹G¶\\2¸}U+~~~5¼nµ/¿t/tWÇ\ 2t^~~^ÁÀÂót^5t^rt: tprt^a»G}¶\\U:·
r¿¸\2~:ÅEt^\p¶¸t^QSt^+tW t^¶}Î~t^2r~SAtWÇ\ 2t^~©prtU~t^»\t:/µ/¸pr\ Q}¶\t:»t^r¸2©t:a}Á
op2tv~t:»t:´t^r¶¸tW~&t^ÇQ rtW~~Sµ/¸pr\ t:a}Åg~©p2´prtUsUtW~ rt^Ît:E2~t¸sºtU¶~t:Ä\=¸}
prt¶sºt¸+ 2tW¾Q}Aprtt:r¶: ¸\&2\g2&rtµ9¹A\sUsv rr:¶'t:a}~:Á
ÎÔ « × Ø ¬ Lª H %Ê« HØ I oA¾+tº¾2¸t ©Et:¼½s tÄ+t^¶sºt:Q´\ϳ\nÅEµtv 2~tU³rt:µ\¹
~¶s r ¸\t:G»G¸2sUt^Q´ rrr¶r©\prtvp+µ9t r¶¼½\sz~pr µ/¶ C¶2Á TÁÂÃÏpr~pr¿·
t^ rtÅntWpÏ:sºr t:~~
¼êprtºnÅgµ/prt:t^\~StWp¼êp2tQa» ´xÏ~´ rr2¸r
\É':sºr t:¶sU2¸t^sUt^Q~rtº¼9p2tUt:¶t:» \
Áopr¶~¶~\pr¸t^»tW¾G}sºt^2~¼9prt
ÂÃ22 +´ rsºs } St N¶¥a O ~¼µt¸2~¸¶t^&&p2t\ t^^ŵ/prp't +~tWA³2¸¶&prt
¶:nrtµ9¹A2©A\r&sUtW~~\tS2~sU~~¶t:a}~:Á
ÌÉtêt^s r t%H+¸¶ 2~\&H2\ r¶¼&H+»t S~^Å µ/prpt`2¸·[µ/~t`2rt^t^pr r\p
¥W££sº~·Ãt:a}&:sºs r2¶^ ¶j¸¶r¹~^ÅntÄrt^ ´~v¥º+jÅnµ/p2¶pjs 2~ +~tUrtU¼êprtºprt:
´~U\~v˶Qt:sºt^ t©pr\ÚÅ`t^~ r¸¶r϶cÉ££s³~v:sºs r2¶^ ¶ct:a}d¾+t:µt^t:mprt:s'Á
p Y¼½tW rtW~~¸Ä©rtW~:ÅG rsv¾+t^tW
Å+µ/prt:t
¶~prtG rs ¾Et:/¼Àp2tt^¸t^»aQ +
~vprt'G rs ¾Et:º¼prt'rtµ/¸pr¶Ëprt Á tW~vÄrp2\ rpËÄ&TÏpr\~A~t^»\t:vt^r¸:\~:Å
µ/p2t:t^~rtW~Än¥Up2\~:¸¶t:Q~-Lµ/prt:d 2~¸2&prt³¶»tºr:[ÅgprtA~t:»t^´¶t^\t:t:r¶:&¶~
¶:t^&rt³¥aÁ
JLK JNM
êÔ ¬ H Ô «^ª ¬ Ø « Ø ® Ø % I oÀÎ: »t:µ/¶tUr\tv¼~¸ + ¶+~:ÅÚµ9tUt:» ¶ 2 t prtº+t^¼½\·
s³2t³¼9prt³µ9't^r¸:¶drG:~ r2t^ ¶¸\pQ 2dprtWa»G}¸QnŸm» ¶ 2~ ¶¸t^\~+
~t:»t^9:&H+ r ¸\2~:Å¿pr r\p¸µa}~9¸&¼Ê¶¸ 2t:·N¼½t:t:\~tW~:Á
t^~S r\2~~~¶Ît:»t^}&¶¸t^\~t^2¶rº¼½\sÑ¥W££U @ ££Uµ/¿ttWÇQ rt^~~9³prtv~t^»\t:WÅ
rt:Et:2¶rÎ\jprt³G rs ¾Et: ¼/¶¸t^\~:Á&/t:EtWjt^~E2~tU¶sUtW~tºsºt^~ rt^jjprt&¶¸t^\
~tÅE /p2t r2¸:¶'¸t^»t^NÅE2'a»t^tWA »t^¶À¶¶t:Q~S¼`º¶»t^ rÚÅE2&prt:Î »\t:
¶t^\~prt^t r+~¼prt~sºt}G+t\Á
- ')C C3 !;& C ;C C "!
H g« %Ø rÙ I Âà prt H2~º+º¼´prt'tÄEt:¸sºt:Q~:Å9µt'sºt^\~ 2tt^+\2~t©¸sºtÎ r2rt:³¶¸\p\
¶\\¾G}Îp2a»G¶r'~¸2¶tU:¸¶t:QÇQ rt:}'prt³~t:»t^^Á op2tU:¸¶t:Q¶~¶:t^jjrt'¥¥\ÅÚ2Ïprt
~t:»t^º¶~A~ 2^t^~~¸»\t:¶}mt^r¸:tWc »\t:(H2»\t ²:&H+ r ¸\2~:Å2sºtW u´¥pr r\p uÁ
u\&H2\ r¶ u´¥:sºr¶~t^~v2¸} ¥Å ujsºr~tW~ È¥'2"Å u @
:sºr¶~t^~
;¥\Å2
@
Å2d~&¼½\pÚÁVCr\t^pɼprtW~t Ò:&H2\ r ¶ÀÅnµtºsºt^\~ rtºt^+\2~t
¸sºtºµ/¸p"¥Å
@
+É&~t^»\t:t:r¶¶^~´+t^
ÅnprtUG rsv¾+t^¼9~t:»t^´t:2¸:\~´2¶r
¼½sÑ¥´©rÁ
C¶2Á dr¶~&prtda»t^tÎt^~+\2~t¶sUtÉ r2rt:'¸¶pQ¸Q"¼½\©prtÉ\=¸»\tj+ÇQ r rs
2\¶~^ż½Ut^p˼prt~t:»t^ \&H2\ r¶2~^Áj ´~UtÄ+tW=t^nÅ`t^~E2~tA¸sºt¶~U¸»\t:ƾG}
2tµ9¹t:a}~5µ/prt:º¶\t:· ´~`\sUsv rr:¶ºt9¶G»\¸»\t^nÅ2v¿ê~\r¸}v~¶¸\pQ¶}vgt^tW
¾G}ÎprtAG rsv¾+t^¼/~t^»\t:t^r¶¶^~Et: Á C¶2Á&¶~Î~p2 µ~Sp2 ¼½\ \&H2\ r¶2~ u
pr r\p u @ Åaprtt^~+\2~tê¸sºt¼+prt/ÇQ r rs rQ~raĸs³ t:¶}Sµ/t/\~p2¸\pU\~Àp2
¼`p2tU\=¶»tvr:[Áopr~¸¶¶ 2~tW~´Ï¶sUEQgt:t:2:t ¾Etµ9t:t:Ïprtvµ9©rG:~-J
\v\rtp2+nÅprt¶»t9r\¸G»\¶»t^~¶¼Ep2t~t^»\t:t^r¸:\~5µ/prt:Urt^~~¸rt^ÇQ rt^~^Å
¾2 p2¶~2tW~~¶r'r¶}Ït^ÇQ r¶tW~rt³t^ÇQ rtW~·[t:2¸}Ïrt:a}Ldprt³prt: p22ÚÅÀprt©ÇQ r rs
2\+\r¸}³¶G»\¸»\t^~êUÇQ r rs ¼~t^»\t:9t^r¸:\~:ÅQ¾r prt¸Qt^\=¸\AtWÇQ r¸t^~êµ9 t^ÇQ rt^~·
t:2¸}Ét:a}~:ÁA¼/µ9tA:¸
.V1-
¼Î~tv¼/~t^»\t:t^r¶¶^~prtAs³ Ķs rsÈsºsv rr: ¸\
rt:a}´¾Etµ9t:t^vG}´µ´t:¶t:sºt:Q~À¼rprt9~t:^Åapr¶~sºt^2~Àp2prtsºt:t:¼2prt9Ç\ 2 rs³~Àsv 2~
¾Et ¸tW~/µ/¶:t ~~s³¶À~/prt r¶sUt:t^¼prt t^r¸:¶' 2©¼½\/prtvÇ\ 2 rs r
©t:ÄGp2¸¾r¸¸ µ9t:\´tWÇQ 2tW~E+~t¸sºtvp2Îp+ ¼p2tv¶»tvr\NÁopr~´:2¸¶Ï¶~
sºt:9¶&:&H2\ r ¶
u·5p2tsºt:t:9¼Àprtt^r¶¶^ ¶³\\ rA~/££s³~ê+ºprtsºtt:
¼Et^»t^}UÇ\ 2 rs~¥^£\£s³~·2ºµt^³~t:tp+ `prtt^~+\2~t9¸sºt^~ê¼Eprtµr~5p2a»\t
~¶sU¶/» ¶ rt^~^Á
GS µt^»t:9µ9t^'~t^tp22tµ9¹At:a}~:Ÿp2 r\p©rt^\sº¸2Q^Åtrp2t\r¶}³¼Ê
p2 ¶ 2 rt^2ttW~E+~t/¸sºt\År~¶2tµ9tSrtp2 9p2tt^~E2~t/¶sºt´¼Úp2tÇQ r\ rs r
~9= 2¶¶}A~¶¸\pQ¶}U¶ µ9t:`p2©p29¼Úp2t\=¶»t´r:2¶ \&H2\ r¶ uÅ\¼½9prt
~sºtG rsv¾+t^¼`~t:»t:/t:2¸:ºEt: Á+oprt tW~&¼½/pr¶~´Et^t^2t\Å2µ/prpζ~St:»\t:'sºt
2tW¾r¶t r2t^/prt^a»G}A¶\\©:2¸¶+~:År~¶»t^&¾Et:¶ µÁ
ÎÔ Õ %Ø rÙ"I ÂÃdprtA~tW\2É2¼prt³t:Ä+t^¶sºt:Q^ŵtºsºt^\~ 2tºt^~+\2~tU¸sºtA r+t:
p2t^a»G}¸QnÅ µ/p2t:vprt~t:»t^~5t^r¶¶^ tW »\t: :&H+ r ¸\ urÁ5u¶¸t^Q~t9 rr¸¼½sU¶}
r¶~¸¾2 tW »t^Úrt^~¥¥`pr r\pv¥\ÅW2µ9t:tW tê¸2:tW~¸2/¸Q¾G}¸2:tW~¸2/prtG rs ¾Et:
¼:¸¶t:Q~+t^ Åp2tnG rsv¾+t^/¼¶¸t^\~r¶rv¼½\s7 ©\ÁÌjtt:Eµ³~t~¼