Pacemaker: Fighting Selfishness in Availability-Aware Large-Scale Networks

HAL Id: inria-00305620

https://hal.inria.fr/inria-00305620v2

Submitted on 8 Jan 2009

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

Large-Scale Networks

Fabrice Le Fessant, Cigdem Sengul, Anne-Marie Kermarrec

To cite this version:

Fabrice Le Fessant, Cigdem Sengul, Anne-Marie Kermarrec. Pacemaker: Fighting Selfishness in

Availability-Aware Large-Scale Networks. [Research Report] RR-6594, INRIA. 2008, pp.33.

�inria-00305620v2�

a p p o r t

d e r e c h e r c h e

N

0

2

4

9

-6

3

9

9

IS

R

N

IN

R

IA

/R

R

--6

5

9

4

--F

R

+

E

N

G

Thème COM

Pacemaker: Fighting Selfishness in

Availability-Aware Large-Scale Networks

Fabrice Le Fessant — Cigdem Sengul — Anne-Marie Kermarrec

N° 6594 — version 2

Centre de recherche INRIA Saclay – Île-de-France

Parc Orsay Université

Fabri e Le Fessant

∗

, CigdemSengul

∗

, Anne-MarieKermarre

†

ThèmeCOMSystèmes ommuni ants Équipes-ProjetsAsap

Rapportdere her he n°6594version2initialversionJuly2008 revisedversionJanuary200934pages

Abstra t: Inthispaper,weintrodu ePa emaker,as alableandlightweight proto oltomeasurereliablytheavailabilityofpeers. Tothebestofour knowl-edge,Pa emakeristheonlyproto olresilienttothepresen eofselshpeers,i.e. peerslyingabouttheiravailabilityandminimizingtheir ontributiontothe sys-tem. Pa emakerreliesonanovelpulse-basedar hite ture,where asmallset of trustedpeersregularlyoodthenetworkwith pulses ontaining ryptographi values. Colle ting these pulses enablespeers to later prove their presen e in thesystematanytime,using ryptographi signatures. Thisnewar hite ture over omesmany limitationsofping-basedsystems,and anbeeasily deployed on ad-ho networks and so ial-based topologies. Simulationresults showthat ourproto olprovidesa urateavailabilitymeasurementseveninthepresen eof selshpeers. Furthermore,ourresultsareveriedbyexperimentsinPlanetlab, whi halsoillustratethedeployabilityofPa emakerinrealnetworks.

Key-words: peer-to-peer, ryptography,availability,monitoring

∗

INRIASa layIledeFran e

†

Résumé: Lamesurededisponibilitédansunréseaupair-à-pairpeutrevêtir unetrèsgrandeimportan epourbeau oupd'appli ations ollaboratives.Ainsi, ette information est inestimable pour identier les pairs les plus stables, ou les groupes de pairs similaires par leur disponibilité. Cependant, omme de nombreusesappli ationsveulentré ompenserlespairslesplusstables,ilexiste unein itation laire pour lespairsàmentir surleur disponibilité réelle. Dans e papier,nousprésentonsunproto ol légeret s alabequi permet auxn÷uds demesurerladisponibilité d'unpairenprésen edepairségoïstes. Dansnotre proto ole, haque pair est hargé de maintenirsa propredisponibilité en ol-le tant despulsations disséminées parune entité de onan een utilisantdes signatures ryptographiques.Celles- ipermettentàtoutpairdevérierpardes hallengeslesinformationsdedisponibilitétransmiseparunpair.

1 Introdu tion

A peer-to-peer network is omposed of thousands of independent omputers, whi haggregatetheirresour esovertheInternettorun ollaborativedistributed appli ations. Su h networks are subje t to high dynami s: omputers(peers) mayjoinandleavearbitrarilyorbesubje ttofrequentdis onne tions. However, it has been observedthat peers with high availability in the systemare more likelyto remainin thesystemfor alongertime [4,16,23℄. Asa onsequen e, manypeer-to-peernetworksrelyonpeeravailabilitytomeasurethestabilityof peers,andusethisparametertosele tpeersforspe i purposes. Forexample, themoststablepeers anbeele tedassuper-peers[9,24℄,orasprivilegedpeers tostorerepli as[5,2, 7,10℄.

Yet, to the best of our knowledge, urrent resear h does not address how availability anbemeasuredse urelyande iently. Aswedis ussinSe tion9, urrentsystemseither useexpensiveand in ompletemeasurement te hniques, orrelyonpeerstogiveanhonestestimationoftheiravailabilityinthenetwork. Thefa tthatstablepeersareusuallyrewardedinapeer-to-peernetwork reates a lear in entive to appear very stable, for instan e by lying about the real availability,inordertobegrantedabetterstatusinthenetwork. Subsequently, su h selsh nodes may get a ess to moreresour esthan they should beable toa ess(i.e.,freeriders). Forinstan e,inapeer-to-peerba kupsystem,peers thatlieabouttheirstabilitymightgetundueandundeserveda esstostorage onthemoststable peersin thesystem.

Motivatedbytheseobservations,wepresentasimpleandlightweight proto- ol, alled Pa emaker,to measureavailabilityin atrustedwayin peer-to-peer networks. ThemainideaofPa emakeristodisseminatepulsesbytrustedpeers in the network. These pulses are then used by peers asproofs of presen e in thesystemat agiventime. Essentially,throughthis simples heme, peers are able to verify the a ura yof theavailability laimsby randomly hallenging ea hother. Sin e hallengedpeersareexpe tedtousethe orre tpulseforthe queriedperiod, Pa emakerisableto dete tselsh peerstryingtoappearmore available in thesystem. Anoverviewoftheproto olisgiveninSe tion 2, and the ompletespe i ationin Se tion4.

Inadditionto providinghigh a ura y in availability measurements, Pa e-makerisalsohighlys alable. Itonlyrequiresthatpeersare onne tedtoenough neighborstoformaredundantmeshtopropagatethepulses. Thisrequirement makesPa emakersuitableforbothstru tured (DHTs)and unstru tured (gos-sip[24℄)peer-to-peernetworks,butalsofortopologieswithlimited ommuni a-tions,su hasad-ho networks andso ial-basedtopologies[20℄. Here,wefo us onasimplemesh network,to illustratethat Pa emakerinheritsthes alability hara teristi s of the underlying network. We des ribe our system model in furtherdetailinSe tion 3.

Themain ontribution of ourresear his providing ea hpeer ase ure way tonotifyitsavailabilitytootherpeersinthesystemina ompletelydistributed manner andthrough lo al ommuni ations(i.e., ommuni ationisonly ne es-sarywith neighbors)usingstandard ryptographi me hanisms. Inthis paper, weonly onsider the ase of selsh nodes, whi h are trying to gaina ess to moreresour esthantheyareallowedtobyappearingmoreavailablethanthey really are,for instan e,by lying. Fornow,wedidnot onsiderthe asewhere

nodesmay olludetoimprovetheiravailability,andwedis ussthisde isionin Se tion9.3.

WeevaluatedPa emakerboththroughanalysis,simulationsandPlanet-Lab deployment. Oursimulationresults,whi harepresentedin Se tion6, onrm thatPa emakerprovideshighlya urateavailabilityinformationwithverylow ostfor both real and syntheti workloads. Furthermore, Pa emaker remains highly s alable due to its light load. Se tion 7 presents the performan e of Pa emakerunderdierentkindsofselshbehaviorsandshowsthatPa emaker is still able to provide high a ura y. For instan e, when 5000 peers out of 100,000lie abouttheiravailability, Pa emakerisableto dete t thesenodes in lessthan 5daysby sending hallenges onlyon e aday and drive theerror in availability measurements ba k to negligible. Similarly, Pa emaker is able to toleratewelltheee t of30%selshpeers, whi h stopdisseminatingpulses in the hopes of improving their availability by redu ing the availability of their neighbors. Finally, wedeployedPa emakerona170-nodePlanet-Labtestbed, whi h again onrmedaverygood mat h betweenthe measuredand thereal availability(seeSe tion8). Basedonoursimulationresultsandourexperien e with Planet-Lab deployment, we on ludethat Pa emaker provides a simple, low- ost,s alableanda uratewaytomeasurepeeravailabilityinthepresen e ofselshpeers.

2 Pa emaker in a nutshell

Pa emakeris asimple and lightweightproto ol to tra kpeer availabilityin a large-s ale system. In Pa emaker, ea h peer is in harge of maintaining its own availability measure and providing it to other peers. Yet this de lared availability anbearbitrarily he kedinapeer-to-peerfashioninordertodete t selshpeers.

Inanutshell,Pa emakerworksasfollows: aserverisin hargeofperiodi ally disseminating pulses in the system, say onepulse per hour. Su h pulses are propagatedinthesystemon ebyallthepeersinthenetworktotheirneighbors. Ea hpeermaintainsalistofthepulsesithasheardofandusesthislisttoprove its availability in the system. Using this simple s heme, Pa emakerprovides de entralizedveri ationofpeeravailabilityin thepresen eofselshpeers.

Selsh behaviors onsidered in this paper in lude, for instan e, trying to obstru tpulse dissemination or laiming,untruthfully, being onne ted tothe systemwhennot. Totoleratesu h selshbehaviors,pulses aregenerated and signedbyatrustedentity. The signature ertiesthe asso iationbetweenthe pulseanditsdiusion time. Hen e,whenpeerssendtheiravailabilitytoother peers, they might re eive a hallenge in return. More spe i ally, a peer

A

may ask a peer

B

to provide a proof for a subset of the time periods that peer

A

laims it wasavailable. A liaris dete ted easily sin epeers should be ableto omputesu h aproofusingthepulses orrespondingto the hallenged periods. Notethat wedonot onsiderthe asewhereapeerpropagatespulses indenitelyto provideother peers with pulses generated when they were not online. Su hmali iousbehaviorsarepartoftheproblemof olludingpeers,and justdis ussedin Se tion9.3.

3 Model 3.1 Denitions

The goalof Pa emakeristo se ure themeasure of peer availability in a peer-to-peer network. Peer availability anbe dened bytwometri s a ordingto the ontext:

ˆ Theratioof timethatthepeerspent onne tedto thenetwork,whi h is avalueintheinterval

[0, 1]

. Thismetri anbeuseddire tly toestimate thestabilityofthepeeroritslife expe tan y.

ˆ Theintervalsoftime whenthepeerwas onne tedto thenetwork. This metri anbeusedtodete tregularpatternsinpeerbehaviors,predi t fu-ture onne tionsanddis onne tions,ordierentiatebetweenatemporary dis onne tionfromadenitivedeparture.

Pa emakerprovidesanapproximationofthese ondmetri ,fromwhi htherst one anbederived. Essentially,wedenethesystemavailability astheaverage availability over all peers in the network. Finally, a group of peers an be atta hedaspe i lassofavailability dependingontheasso iatedappli ation. Forinstan e, peerswith availabilitygreaterthan95% anbe onsidered tobe in thesuper-peer lass.

In a system that favors highly available peers, peers may exhibit various selshandmali iousbehaviors:

ˆ Opportunisti peersonlyfulllthestepsoftheproto olrequiredtoget agood status,butwithoutimpa tingthestatus ofotherpeers.

ˆ Lazy peersonly fulll the stepsof theproto ol requiredto get agood status, and do so even when su h a behavior an impa t the status of otherpeers.

ˆ Lyingpeerstrytoimprovetheirownstatusbylying. Theydon'timpa t dire tly the status of others, but they might get undeserved a ess to resour es.

ˆ Colludingpeers ollaboratewithea hothereithertoimprovetheirown statusortodisruptthesystem.

Pa emaker se ures the measure of availabilityagainst the rst three types of behavior, whi h are all selsh. Dealing with olluding peers is dis ussed in Se tion 9.3.

3.2 Network Model

We onsider alarge-s alenetwork(morethantensofthousandsof omputers) omposedofnodes(orpeers), onne tedbya ommuni ationmedium,typi ally IP. Weassumethereexists alogi aloverlaynetwork whereea h nodeisaware ofasmallportionofthenetwork,i.e. itknowstheIPaddressesofasetof

D

max

neighbors. Thisistypi allythe asein bothstru turedandunstru tured peer-to-peernetworks. Inthenetwork,peers ommuni atebysendingasyn hronous messages. Althoughthereisnoboundon ommuni ationdelays,mostmessages

areassumed to bere eivedafter ashort delayand assumed to be lost after a longerdelay. Althoughnotarequirement,weexpe tnodesto onne tthrough FIFO hannels, whi h enfor es sequentiality of messages. Additionally, there alsoexistsaglobal lo k,withwhi h omputer lo ksareloosely oupled. This is ne essary for a peer to know at whi h periods (the periods are onsidered system-wise)itwas onne tedtothesystem. Hen e,peershaveanapproximate agreementontime.

Pa emaker relies on the existen e of a trusted entity. In this paper, we assumethere exists parti ular peers, whi h are alled theservers. Pa emaker doesnotrequirethenon-serverpeerstoknowtheidentityoftheserversorany otherpeer. However,itisassumedthattheoverlaynetworkis onne tedenough to ensurethat everypeer in the network is rea hablefrom at least oneofthe servers.Sin eservershaveaspe i roleintheproto ol,theymayhaveahigher degreethan D

max

. This helps, forinstan e, to prevent Sybilatta ksthat try to ir leserversto disruptthediusionofpulses. Forthesakeofsimpli ity,in therestofthepaper,we onsiderasingle-serversystem. Thisassumptiondoes notae t ourresults,sin eno ommuni ationisrequiredbetweentheservers. 3.3 Cryptographi Model

Weassumethatpeershavea esstostrong ryptographi primitives,spe i ally forpubli -privatekeyoperations,whi harethefollowing:

ˆ generate_pair(): Generates a new pair of publi -private keys. The ommon usageis that the private key, K

priv

, is kept se retby thepeer, whilethepubli key,K

pub

, isknowntootherpeersinthesystem. ˆ sign(data, K

priv

): Returns asignature for datausing the private key

K

priv

.

ˆ verify(S, data, K

pub

): Veriesthat S is asignature fordatathat was reatedusing theprivatekeyK

priv

asso iatedwithK

pub

.

ˆ hash(data): Returnsthehashofdata.

We assume that there is a spe ial pair of keys, one publi ( alled KS

pub

) known by all peers in the system, one se ret ( alled KS

priv

) known only by theserver. Ea h peer

p

in the system also owns a pairof keys, noted K

p,pub

and K

p,priv

, to sign data. We also dene H

p

= hash(K

p,pub

), and use it as unique identier for

p

in the network. These keys should also be used by a peer-to-peer appli ation runningon topof Pa emakerto prevent selsh peers from easily hangingtheir identity when they are dete ted. Furthermore, we assumethereexistsawayforpeerstoex hangetheirpubli keysbyeitherusing dedi atedmessagesordueto ryptographi ommuni ationproto olsalreadyin pla e(su hasTLS[6℄). Finally,weassumetheseoperationsprovideahighlevel ofse urity(i.e.,itisalmostimpossibletobreakthe ryptographi propertiesof these fun tionsby su h as havinga ollisionin the hashfun tion) in thetime limitsneededfortheappli ation[15℄.

Server

011

111

011

011

111

111

111

100

011

010

000

000

010

010

100

000

100

110

100

100

Figure1: Allpeersare onne tedthrougharedundantmeshtotheserver. Here,the numberineverynoderepresentsa3-bitavailabilityhistoryas(3rd,2nd,1st)rounds. Thearrows depi tthe pulsepropagation. Everyhour,anew pulseispropagatedin themeshbytheserver.

4 Pa emaker in Detail

Pa emakeris omposedofthreesub-proto ols: (1)thepulsedissemination pro-to ol; (2) the availability inquiry proto ol and (3) the availability veri ation proto ol, whi h are presented in theirrespe tive se tions in the remainder of thisse tion.

4.1 Pulse dissemination proto ol

TheserverinPa emakerisin hargeofgeneratingonepulseoveragivenperiod of time

P

. The dissemination of a pulse onsists of ea h peer forwarding it on e to all its neighbors. An example of the pulse dissemination is depi ted in Figure 1. The gureshowsa redundant mesh network, where there exists multiple pathsbetweenea hpeerin thenetwork. Thisredundan yisessential tode reasetheimpa tofpeersthatdonotfollowtheproto ol(i.e.,theimpa t ofnodesthat donotforwardthepulse totheirneighbors).

Apulse T

i

,whi his generatedbytheserverfortime

i

, isatuple(

i

, K

i

pub

, K

i

priv

, S

i

), where K

i

pub

and K

i

priv

is anew fresh publi -private keypair. The publi -private key pairs are generated by the server on-demand. The pulse alsoin ludes S

i

,whi hisasignatureof(

i

,K

i

pub

)usingtheservers'privatekey KS

priv

. Theserverdiuses the pulse to itsneighborset (NS) at time

i

( ode Fig.2).

Everypeerkeepsanhistoryofthesepulses, representingitspresen einthe network during awindow of time N

t

× P

. On re eipt of a new pulse, a peer rst he ks if it has already re eived the same pulse and if not, veries the

Serverattime

i

: let ( K

i

pub

, K

i

priv

) = generate_pair(); let S

i

= sign( <i, K

i

pub

>, KS

priv

); let T

i

= Pulse(

i

, K

i

pub

, K

i

priv

, S

i

);

∀q ∈

NS

server

, send(

q

, T

i

);

Figure2: Pulsegenerationat theserver.

Node

p

re eivingT

i

=Pulse(

i

,K

i

pub

,K

i

priv

,S

i

): if T

i

_∈

_/

History

p

and verify(S

i

, <i, K

i

pub

>, KS

pub

) then add(History

p

, T

i

);

∀q ∈

NS

p

, send(

q

, T

i

); end if

Figure3: Pulsediusion byapeer.

Node

p

sendingto

q

itsavailabilityattime

i

: let bits = n e w bitfield[

N

t

℄; for

x

in [1..

N

t

℄ if

∃

T

j

_∈

History

p

| j ∈ [i − xP, i − xP + P [

then bits[

x

℄ := 0 else bits[

x

℄ := 1 end if end for let S = sign( <

i

, bits >, K

p,priv

) send(

q

, Availability(

i

,bits ,S) );

Figure4: Advertisementofavailabilitybyapeer.

authenti ity of the pulse. If the pulse is indeed generated by the server, it updatesitshistoryandforwardsthepulse toitsneighbors( odeFig.3). 4.2 Inquiry Proto ol

Depending on the appli ation, peers need to be able to he k the availabil-ity of other peers. This might be done either regularly orjust the rst time they onne t to ea h other. The veri ation of availability requires know-ing the pulse history of peers. For this purpose, ea h peer sends a message Availability(

i

,biteld,S),where

i

isthe urrenttime, andbiteldis anarray ofbitsofsize

N

t

, ontaining,forea hperiod,1ifithasthepulse,and0 other-wise( odeFig.4). Themessagealso ontainsasignatureofthebit eldusing the peer private key, K

p,priv

. This signature an be used to prove later that themessagewassentbythepeer,in parti ular,ifthepeer doesnotreplyto a hallenge.

Node

p

re eiving Challenge(

i

,non e) fromnode

q

: if Pulse(

i

, K

i

pub

, K

i

priv

, S

i

)

∈

History

p

then let reply = sign( < non e, H

p

, H

q

>, K

i

priv

) send(

q

, Proof(

i

, non e, K

i

pub

, S

i

, reply) ); end if

Figure5: Replytoa hallengebyapeer (onebittosimplify)

Node

q

re eivingProof(

i

,non e,K

i

pub

,S

i

,reply)fromnode

p

: if (

i

, non e,

p

)

∈

hallenges

q

and

verify(S

i

, <i, K

i

pub

>, KS

pub

) and verify(reply, < non e, H

p

, H

q

>, K

i

pub

) then good_reply(

p

) else bad_reply(

p

) end if

Figure6: Veri ationofaproofbyapeer (onebittosimplify).

Usingthebiteldofanotherpeer,apeer an omputeanapproximationof the availability ofthe peer during theperiod (N

t

*P)by ountinghowmany bits are set to one. Note that, in fa t, the bit eld only provesthat the peer was online when the pulses were propagated and not during omplete hours. However,weshowinoursimulationsthat sending thepulse at arandom time inthe urrentperiodprovidesaverygoodapproximationoftherealavailability. 4.3 Veri ation Proto ol

It is in the interest of some peers to lie about their uptime, espe ially to get more resour es than they deserve. We thus provide a veri ation s heme to allow apeer to verify that the bit eld re eived from another peer is orre t. Morespe i ally,to hallengeagivenpeer,apeersele tsonebitsetto1inthe bit eld re eived from this peer (this an be easily generalizedto several bits hallengedat on e). It sendsaspe ialrequest Challenge(

i

,non e), ontaining

i

,theperiodofthebittobeveried,andnon e,whi hisarandomlygenerated shortstringtomakethe hallengeunique. Onre eptionofChallenge(

i

,non e), thepeerrepliesProof(

i

,non e,K

i

pub

,S

i

,reply)whereK

i

pub

andS

i

arerespe tively thepubli keyandthesignaturefrompulseT

i

,andreplyisthesignatureofnon e and the hashesof theidentities of the twopeers by the theprivate keyK

i

priv

( ode Fig.5). On re eption of Proof(

i

, non e, K

i

pub

, S

i

, reply), thepeer an verifythatreplyisthesignaturewiththe orre tkeyK

i

priv

,usingthekeyK

i

pub

fromthemessage,and analsoverifythatK

i

pub

isthepubli keyfromthepulse T

i

usingthesignatureS

i

andthewellknownKS

pub

key( odeFig.6).

The a tions to be taken when a peer fails to provide a orre t reply to a Challenge message is out of the s ope of this paper sin e it mostly depends

Tra e Size Length Sessions Availability AbsoluteError peers days

<

1

h 

>

1

d

<

25%



>

75%

<

1%



>

3%

SkypeSuperpeers[11 ℄ 2081 29 15% 80% 5% 60% 22% 18% 95% 4% 1% Mi rosoftDesktops[3℄ 51663 10 0% 85% 15% 15% 15% 70% 97% 3% 0% OvernetClients[1 ℄ 1469 7 39% 61% 0% 49% 44% 7% 58% 32% 10% Synth. Uniform 100000 20 30% 68% 2% 20% 60% 20% 20% 70% 10% Synth. Exponential 100000 20 27% 61% 2% 80% 15% 5% 20% 78% 2% Figure7: Weransimulationsusingafewavailabilitytra es olle tedfordierent

workloads. Ex ept the Skype workload, su h systems are not representative of real appli ations for Pa emaker: Mi rosoft network exhibits a very small hurn,typi alof ompanynetworks, whereason the ontrary,Overnet's hurn isveryhigh,evenforale-sharingappli ation(a tuallymu hhigherthanre ent observationsontheEdonkeynetwork).

on the appli ation using our measurement system. However, our proto ol is designed so that it is possible to propagate both the Availability and the Challenge messages to other peers in the network. Hen e, other peers are allowed to use a not replied Challenge message to hallenge the same peer again. Toavoidfalse laims, the peers mightuse theAvailabilitymessages to he k that themessagewasindeed signed byaselsh peer. If somepulses aredamagedonapeerduetoafailure,thuspreventingveri ation,thepeeris expe tedto learthe orrespondingbitsfromitsavailabilityhistory.

5 Evaluation Road-map

Themain goalofourevaluationstudyistoillustratethat Pa emakeris: ˆ S alable: It an a ommodate the growth of the system; it is able to

workwithmillionsofpeers onne tedtogether.

ˆ A urate: Theerrorbetweenthemeasuredandrealavailabilityofapeer isnegligible.

ˆ Low- ost: It is less expensive than other systems providing a similar measure.

ˆ Se ure: The measure still ree ts the reality even though selsh peers maytrytomodifyit. Furthermore,itisabletodete tlyingnodestimely. ˆ Easy-to-deploy: It anbeimplemented easilyand deployedwith

mini-mal ongurationonthepeers.

WestudiedPa emakerviaa ombinationofanalysis,simulationsusingreal andsyntheti tra esand animplementation on a170 nodePlanetlab testbed. Intheremainderofthis paper,werstpresenttheperforman e ofPa emaker asanavailabilitymeasurementsystemusingsimulationresultsonthesyntheti tra es. Although,wealsoransimulationswithrealtra es(seeFig.7),weomit these results for the sake of brevity, espe ially be ause syntheti tra es allow us to evaluate Pa emaker on larger-s alenetworks (100,000 peers) with more extremeavailabilitydistributions(uniformandexponential).

Next,wepresentresultswherePa emakeroperatesinthepresen eofselsh peers that try to heat the system by advertising a higher availability. Our goal in these experiments is to show that Pa emakeris able to provide a u-rate availability measurements in an e ient manner even in the presen e of su h selsh peers. Finally, we on lude with a dis ussion on implementation details. Essentially,thePlanetlabexperimentsillustratetheeaseofdeployment ofPa emakerinarealisti setting.

6 Availability Monitoring with Pa emaker

Therstgoalof ourevaluation studyis toprovethes alabilityofPa emaker, the a ura y of its availability measurements and the negligible load it adds to the system. To this end, our simulation setup onsists of two parts: (1) theavailabilitypatternsofpeersand(2)theunstru turedoverlaynetwork(the mesh) onne ting peers and the server. In this se tion, we rst present this setupindetailandnext,theperforman eresultsin omparisontoaping-based availabilitymeasurementsystem.

6.1 Simulating the Availabilityof Peers

Inthesyntheti tra esusedbyoursimulations,availabilityfollowseithera uni-form orexponentialdistribution. Whileworkingwiththeuniformdistribution allowsusto spanallvaluesof availability,theexponentialdistribution ismore representativeofrealpeer-to-peersystems[1℄. Basedonthesetwodistributions, theavailabilityofapeer

y

,

a

y

,is al ulatedas:

a

y

=

(

0.02 + 0.98 · U (0, 1)

ifuni.

max(0.02, min(1, e

1−ln (2+65·U(0,1))

₎₎

ifexp.

(1)

Additionally, the number of dis onne tions per day,

d

y

for ea h peer follows a uniform distribution:

d

y

= U (0, 10)

. Using

a

y

and

d

y

, the probabilitiesto swit hbetweenON(i.e.,onlineandavailable)andOFF (i.e.,oineandnot available) statesare omputedasfollows.

λ

y

=

24×60

d

y

(2)

µ

y

=

a

y

×

λ

y

1−a

y

(3) Using these two probabilities, the Markov hain depi ted in Fig. 8 drives the state hanges. Additionally, to a ount for the ee t of the timezones, this Markov hain is modied to obtain a diurnal pattern: during the day, peers havetwi e their normalprobability ofswit hing to ONand half their normal probabilityofswit hingtoOFF.

TheresultingavailabilitypatternsarepresentedinFig.9andFig.10,whi h depi tthenumberofonlinepeersandthesessionlengthsofpeers,respe tively, for uniform and exponential distributions. Fig. 9 shows that the number of available peers per round is lower with exponential distribution ompared to uniformdistribution. Essentially,whilewithexponentialdistribution,the num-ber of available peers per round is approximately 25,000 during the day and 10,000 during thenight, for theuniform distribution, thenumber ofavailable

ON

OFF

1 − λ

j

1 − µ

j

λ

j

µ

j

Figure8: Onlineandoinetimesofapeerare omputedusingaMarkov hainwith probabilities

λ

and

µ

.

0

10000

20000

30000

40000

50000

60000

70000

80000

0

5000

10000

15000

20000

25000

Number of available peers

Time (minutes)

100,000 peers, unif. dist.

100,000 peers, exp. dist.

Figure9: Numberofpeersonlineovertime. Sin ethetimezonesofpeersareonlyon 12hours,thenumberofpeersfollowadiurnalpattern, whi hwouldnotbethe ase ifthedistributionwasover24hours.

1

10

100

1000

10000

0

1e+06 2e+06 3e+06 4e+06 5e+06 6e+06 7e+06

Session Length (minutes)

CDF of sessions

100,000 peers, exp. dist

100,000 peers, unif. dist

1 day

1 hour

Figure10: CDFofSessionslengths. Themediansessionlengthisaroundtwohours.

peersis65,000duringthedayand35,000duringthenight. Fig.10showsthat, as expe ted from Fig.9, there are a highernumber of sessions in the ase of uniformdistribution omparedtoexponentialdistribution. Thesessionlengths rangefromaminutetoafewdaysforbothdistributionsandthemediansession lengthisaroundtwohours.

6.2 Overlay Network Setup

Weuse amesh that onne tsallpeersand asingleserver. Theserverdiuses pulses throughthis mesh every hour (i.e., unit time P = 1hour) to measure theavailabilityofpeers. Inoursimulations,themeshisformed asfollows: the server has an out-degree of 10 ( alled hildren in the sequel), and ea h peer hasaout-degree ( hildren)andin-degree (parents)of5. Althoughmanyother approa hes ouldbe onsideredtobuildthemesh,weusedthesimplefollowing proto ol: to onne tto themesh,apeer rstsendsanAskRootmessagetothe server,whi hreplieswithalistofits hildreninthegraph. Thepeerthensends AskParent messagesto the hildren. Every hildeither a eptsthepeer asa hild, orsends a random hild among its hildren. The pro ess iterates until thepeeris onne tedto 5dierentparents.

Weaddedthefollowinglo aloptimizationsto improvethemesh:

ˆ At every round, if a peer hasa free hild slot, it hooses among all the hild andidatestheonewiththebestmeasuredavailability. Tothisend, in oursimulations, ea h peer delays itsresponse to AskParentmessages by1minutetobeableto hoosethebest andidate.

ˆ Tode reasethediameter ofthenetwork,apeer dis onne tsthe hildren thatare atthe samedistan efrom theserver. Thedistan e information islearnedeither from theAskParentorthe Distan eupdate messages, whi h are sent by a peer ea h time its distan e to the server hanges.

1

10

100

1000

10000

0

5000

10000

15000

20000

25000

Messages per Minute

CDF of Minutes

100000 peers, unif. dist.

100000 peers, exp. dist.

10000 peers, unif. dist.

10000 peers, exp. dist.

1000 peers, unif. dist.

1000 peers, exp. dist.

Figure 11: The numberof AskRoot messages re eived by the server per minute in oursimulations. Themean rateis 1/50 ofthe numberof nodes, i.e. 100,000 nodes onsume33messagesperse ond. Eveninoursimpliedmesh,afewservers aneasily handleafewmillionsofpeers.

Obviously,forpeersthatdonothaveaparent,thedistan etotheserver isinnity. Otherwise,peersadvertisetheirminimumdistan etotheserver throughtheir urrentparents. Notethatthewell-known ount-to-innity problemmighto urduringtheDistan eupdatesandisresolvedsimilar to[12℄ by hoosingasmall number(e.g.,10)forinnity.

Whilewe hosethisspe i meshgenerationproto olhereforitssimpli ity, themesh ouldbebuilt dierentlybasedon theappli ation requirementsand theservi edesired. Pa emakeronlyrequirestheunderlyingmeshtobeableto diuse thepulses su essfully, whi h is areasonable expe tation. Sin e Pa e-makerrelies ondiusion ofpulses,thes alabilityande ien yofourproto ol and the a ura y of the availability measurement depends on the underlying mesh. Therefore, werst showthes alabilityof themesh usedin our simula-tionsby ountingthenumberofAskRootmessagesre eivedbytheserverfrom neworre onne ting peers (see Fig.11). Weevaluate thenumberof AskRoot messagesforbothuniformandexponentialavailabilitydistributionswith1,000, 10,000and 100,000 peers. As expe ted, asthenumber ofpeers in reases,the number of AskRoot messages also in reases. Furthermore, with uniform dis-tribution, sin e thenumber of sessions is higher, we observe ahigher number ofmessagessentperminute. This is be ause thepeers in oursimulationsare memorylessandhen e,ea htimethey omeba konline,theyneedtoredis over parents. Nevertheless,evenwiththis property,Fig.11showsthatthe meshis s alable: thenumberofAskRootmessagesgrowstoonly1000whenthenumber ofpeersin reasesto100,000. Notethatthisbasi allytranslatestolessthan10 kB/stra loadontheserver,whi hisveryreasonable.

Tounderstandthes alabilityofthe onstru tedmeshfurther,Fig.12plots themaximalhop-distan e to theserver. As expe ted, itgrowslogarithmi ally

0

2

4

6

8

10

12

14

16

18

0

5000

10000

15000

20000

25000

Maximal distance from server

CDF of minutes

100,000 peers, unif. dist.

100,000 peers, exp. dist.

10,000 peers, unif. dist.

10,000 peers, exp. dist.

1,000 peers, unif. dist.

1,000 peers, exp. dist.

Figure12: Maximaldistan e tothe serverovertime. Notethat thediameterof the networkisnottoohigh.

5

10

15

20

25

1

10

100

1000

10000 100000 1e+06

1e+07

Time to becoming a child (minutes)

CDF of connections

100,000 peers, unif. dist., 5,908,795 conns

100,000 peers, exp. dist., 1,853,893 conns

10,000 peers, unif. dist., 609,272 conns

10,000 peers, exp. dist., 190,549 conns

Figure13: Timespentbetweenthebeginningofapeersessionandthe onne tionto itsrstparent. Onlyinsomerare ases,itisabove7minutes. Sin einoursystem,we fo usonlongsessiontimes(mediansessionlengthistwohours),thisdelayisnegligible.

0

200

400

600

800

1000

0

5000

10000

15000

20000

25000

30000

Mean availability (1/1000)

CDF of minutes

peers at distance 1

peers at distance 2

peers at distance 3

peers at distance 4

peers at distance 5

peers at distance 6

Figure14: Mean availability of onne ted peers dependingon their distan e to the server. Thepeers losertotheserverhaveahigheravailabilitythatfartherpeers.

with the number of peers in the network. More spe i ally, the maximum numberofhopsrangebetween5and12dependingonthenumberofpeersand the availability distribution. The ee t of this is also seen in Fig. 13, whi h shows the time it takes for a new peer to nd its rst parent. This delay is morethan10minutesonlyforlessthan1/1000ofthe onne tions. Essentially, these delays, whi h are on the order of a few minutes, an be onsidered as negligible, sin e we are interested in measuring availability for long sessions (i.e.,themedian sessionlengthistwohours).

Finally, theee t of ourlo al optimizationsare depi tedin Fig.14, where themeanavailabilityofpeersversustheirdistan etotheserverisplotted. The gureshowsthatpeers losetotheserverhaveahigheravailabilitythanpeers farther away. This is due to prioritizing peers with higher availability when sele ting hildren. Without su h a prioritization, we would not observe this ee tandthemeshwouldbelessstable.

6.3 Results on A ura y

In this se tion, we present resultsin terms of a ura y and ost-ee tiveness of Pa emaker. We simulate 100,000peers for 20 days (i.e., 28,800 minutes). Everyroundin thesimulationtakesoneminute. Froma ommuni ationpoint of view, this puts some timeout on messages, whi h allows us to dete t peer dis onne tions(forinstan e, TCPkeepaliveis30se onds).

For ea h peer in the network, we ompute the a ura y as the dieren e betweenitsrealavailabilityandtheavailabilitymeasuredbyoursystem(i.e.the availability thatit isableto provetoother peers). Essentially,this represents the absolute error, plotted on Fig. 15 for both the uniform and exponential distributions. Our measuredavailabilitymat hes the realavailability of peers whenthepulse period,P, is1hour. Uniformrandomdistribution exhibitsthe

0.0001

0.001

0.01

0.03

0.1

0.15

0

20000

40000

60000

80000

100000

Absolute Error on Availability

CDF of peers

abs. error, unif. dist.

abs. error, exp. dist.

Figure15: Measuredavailability omparedtorealavailabilityforuniformdistribution. Notethatthedieren eisnegligible.

0.0001

0.001

0.01

0.1

0

20000

40000

60000

80000

100000

Error on measured availability

CDF of peers

period 10 minutes

period 20 minutes

period 30 minutes

period 60 minutes

period 120 minutes

Figure16: TheerrorinmeasuredavailabilityversusPforexponentialdistribution

worst ase: for70%ofthepeers,theabsoluteerrorislessthan1%,forthenext 20%ofthepeers,itislessthan3%,andneverex eeds10%(oftime).

Obviously,if weredu e thepulse period P,we ana hievebettera ura y but at ahigher ost. Fig.16 and Fig.17 show thea ura y of Pa emakeras P rangesbetween10minutesto 2hours. Forbothdistributions,while P =2 hourswoulda hievethebest ost,italsoprovidesthelowesta ura y(i.e.,the errorissigni antlyhigher). WhiletheerrorimmediatelyimproveswithP=1 hour, forlowerP values,Pa emakerperformswith omparablea ura y. This

0.0001

0.001

0.01

0.1

0

20000

40000

60000

80000

100000

Error on measured availability

CDF of peers

period 10 minutes

period 20 minutes

period 30 minutes

period 60 minutes

period 120 minutes

Figure17: TheerrorinmeasuredavailabilityversusPforuniform distribution

0.001

0.01

0.1

1

0

20000

40000

60000

80000

100000

Absolute Error

CDF of peers

pacemaker error (P=1h)

pings with 5 observers

pings with 10 observers

pings with 15 observers

Figure18: A omparisonbetweenPa emakerandping-basedsystemforuniform dis-tribution.Pa emakerisequivalenttoaping-basedsystemthatuses5and10observers forea hpeer.

shows that Pa emaker a hieves a good trade-o between a ura y and ost. Notethat in oursimulations,peersa eptunorderedpulses (i.e.,apulse fora giventimewouldbea eptedevenifapulsefor alater timehasalreadybeen re eived). Notfollowingthispoli y andegradethea ura yofthemeasurefor smallvaluesofPasthepeerdistan e totheserverin reases.

We believe that even in the worst ase, the a ura y of Pa emaker is a - eptableforamajorityoftheappli ationssin e1-5%errordoesnot hangethe

0.001

0.01

0.1

1

0

20000

40000

60000

80000

100000

Absolute Error

CDF of peers

pacemaker error (P=1h)

pings with 5 observers

pings with 10 observers

pings with 15 observers

pings with 20 observers

pings with 25 observers

pings with 30 observers

Figure 19: A omparison between Pa emaker and the ping-based system for expo-nential distribution. Pa emakerperforms better thana ping-based system with 20 observersforea hpeer.

availability lassofapeer(asdis ussedinSe tion3). Furthermore,we ompare Pa emaker with asystem where ea h peer in the network is monitoredevery minute using pings by a small set of randomly sele ted observers (similar to AVMON[19℄). Fig.18showsthatfortheuniformdistribution,theping-based system a hieveshigh a ura y (i.e., negligible error) with only a low number (5-10)observers. However,thisdoesnotholdforthemorerealisti exponential distribution,asseeninFig.19. Inthis ase,theping-basedsystemmustusea higher numberof observers(e.g., 20observers) to rea h the samea ura yas Pa emaker. Ontheother hand,in theworst ase, Pa emakerhasa ostof 10 Pulsemessagesperpeer(if themeshdegreeis10)and perhour(iftheperiod P is onehour). Furthermore, note that in aping-based system, no measure-ments anbetaken if theobserversare down, whi h is often the asefor the exponentialdistribution. Therefore, Pa emakerprovides morea ura y asits availabilitymeasurementdoesnotdepend onaxedsetofobservers.

7 Pa emaker against Selsh Peers

WhilePa emakera hievesgooda ura yinenvironmentswherenoselshpeers are present, itis essentialto maintainsimilarperforman e whenpeers exhibit selsh behavior. In thefollowingse tion,weevaluatehowPa emakerhandles dierent selsh behaviors,whi h are identied in Se tion 3. These behaviors, translatedintoPa emaker ontext,arenamely:

ˆ Lazy peers: These peers do not propagate pulses so that other peers havealowermeasuredavailability.

ˆ Opportunisti peers: Thesepeersonly onne tto themeshto re eive thepulsesandimmediately dis onne tafterward.

0.0001

0.001

0.01

0.1

1

10

0

20000

40000

60000

80000

100000

Error on measured availability

CDF of peers

90% lazy peers

50% lazy peers

40% lazy peers

20% lazy peers

0% lazy peers

3%

Figure20: Error onmeasuredavailability dependingonthe numberoflazypeers in thesystem. Althoughlazypeersdonotpropagatethepulses totheirneighbors, the error is under10% with 50% of lazy peers inthe network, and almost not ae ted with30%oflazypeers.

0.0001

0.001

0.01

0.1

1

10

0

20000

40000

60000

80000

100000

Error on measured availability

CDF of peers

90% lazy peers

50% lazy peers

40% lazy peers

20% lazy peers

0% lazy peers

3%

Figure21: Erroronmeasuredavailabilitywithlazypeerswithauniformdistribution

ˆ Lying peers: Thesepeerslieabouttheiravailability. 7.1 Lazy Peers

Lazypeers preventother peers from beneting from the systemby not prop-agating the pulses to their neighbors. Basi ally, lazy peers do not followthe proto ol when the proto ol a tions have a ost and no dire t benet. As an

0

0.2

0.4

0.6

0.8

1

0

20000

40000

60000

80000

100000

Availability

CDF of peers

real availability (uniform)

random diffusion

trusted diffusion (5 min. conns.)

trusted diffusion (10 min. conns.)

trusted diffusion (30 min. conns.)

trusted diffusion (50 min. conns.)

Figure 22: Availabilities measuredusing eitherrandom he ksor by requiring some onne tion length. It shows that sending the pulse randomly (ata random minute duringthepulsehour)performsmu hbetterthanrequiringaminimalsessionlength (from5to50minutes)froma hildtopropagatethepulse.

additional onsequen e,thesepeersmayimprovetheirratingsinthesystemby de reasingthemeasuredavailabilityofotherpeersratherthan laimingahigher availabilitylikethelying peers. Toevaluate theimpa t oflazy peers, we ran-domly sele tedpeers aslazypeersin oursimulations. Theresults,depi tedin Fig.20,showthatthea ura yoftheavailabilitymeasureisnotae tedmu h (i.e.,theerrorremainsunder 10%)untiltheper entageoflazypeershits50%. This goodresultsareadire tpropertyof thedegree hosenforoursimulation mesh. Hen e,we on ludethat:

ˆ Whenthere isalownumberoflazy peers (i.e.,upto30%oflazypeers), their impa t is negligible on the measured availability of other peers. Hen e,lazypeersdonotsu eedinredu ingtheavailabilityoftheir neigh-bors.

ˆ Whenthere isahigh numberoflazy peers(i.e., greaterthan50%), their impa tismoresigni antbuttheirmeasuredavailabilityisalsoas dimin-ishedastheoneof ollaborativepeerssin etheyarealsoae tedbyother lazypeersthat donotpropagatepulses.

7.2 Opportunisti Peers

Opportunisti peers tryto heatthesystemby onne tingtothenetworkonly to getpulsesto in reasetheirper eivedavailability. Inoursystem,su hpeers would onne tatxed times,dependingonthes heduleof pulsediusion. To avoidopportunisti behavior,weproposetwodierentpoli ies:

ˆ Random diusion: Within ea h period P, the server starts the pulse diusionatarandomtime. Hen e,theopportunisti peers annotfore ast whento onne ttothenetwork.

ˆ Trusted diusion: When a peer re eives a pulse, it only propagates thepulse to hildrenwhi h havebeen onne tedforalongtime. Hen e, opportunisti peersneverre eivepulses.

Fig.22plotstheimpa tofthesepoli iesonthea ura yofmeasured avail-ability. It shows that random diusion performs mu h better as soon asthe requiredsession lengthfor hildrenbe omes too long. Essentially, trusted dif-fusionrequires guessingtheaverageneighborsessionlengthto avoidpunishing good neighbors that do not havelong session length. This might bedi ult sin ethesessionlengthsexhibitahigh varian e. Hen e,weusedrandom diu-sionin alloursimulations.

7.3 Lying Peers

In ontrastto lazy and opportunistpeers, lying peers try to a hieveahigher statusin the networkbyadvertising falseavailabilityinformation. In our sys-tem, this is simplydone by swit hinga 0bit to 1 in the availability bit eld. Pa emaker provides peers with the ability to hallenge peers based on their advertised availability. Using this s heme, the probability that a hallenger

y

dis oversthat

x

islyingin agiventryisdeterminedbytwofa tors:

ˆ How manybits

x

liedabout(i.e.,howmanybits areswit hed from 0to 1)

ˆ Howmanybits

y

hallenges

Inthispaper,wedidnot onsiderthe aseof hallengingtheentirebiteld duetothehigh omputationaloverheadandthegrowthinmessagesize. Hen e, in agiventry, onlya xednumber of bits, denoted as

i

, are hallenged. Our goal is to al ulate the probability that a hallenge sent to peer

x

su eeds when

i

bits are hallenged. Giventhe numberofswit hed bits,

n

switched

, and thenumberof orre tbits,

n

correct

,(whi hadduptothetotalnumberof1-bits inthebiteld)

p(x, 1)

anbe al ulatedas:

p(x, 1) =

n

switched

(x)

n

correct

(x) + n

switched

(x)

(4) This anbe generalizedto

i

bits asfollows. A hallenge would not su eed if andonlyifallthe hallenged

i

bits are orre t. Hen e,

p

(x, i) = 1 − Π

i−1

k=0

n

correct

(x) − k

n

correct

(x) + n

switched

(x) − k

(5) Fig.23showshow

p(x, i)

in reaseswith

i

whentheper entageof

n

switched

bits amongthetotalnumberof1-bitsis5,10,25and 50%,respe tively. Notethat whenthe lyingper entageislow,

p(x, i)

is also low- even thoughitimproves with in reasing

i

. On the other hand, at a given hallenge, when the lying per entageis high, it ismoreprobableto dete t liarseven when

i

is low. For instan e,for

i

= 3

,

p(x, i) = 0.89

whenlyingper entageis50%.

0

0.2

0.4

0.6

0.8

1

1.2

0

2

4

6

8

10

p(i)

Number of challenged bits (i)

5% switched bits

10% switched bits

25% switched bits

50% switched bits

Figure23:

p(x, i)

when5,10, 25and50%ofthe1-bitsareswit hed.

p(x, i)

in reases asthelyingper entageandthenumberof hallengedbitsin rease.

However,theprobabilityofdete tingalyingpeerdoesnotonlydependon

p(x, i)

but also on how and when the hallenges are sent by the peers. Our proto oldoesnotexpli itlyspe ifywhen hallengesshouldbesenttopeersand how the system should rea t when a peer fails to reply to a hallenge sin e these arestri tlyappli ation-dependent. However,in thisse tion,weoutlinea generi strategyfordealingwithlyingpeersandbasedonthisstrategyanalyze theprobabilityandthetimetodete tthemin agivensystem.

Weassumethatea hpeerisworkingwith

m

peersonaveragefortheneeds of the appli ation. A peer

y

an hallenge apeer

x

as astep of one of their onne tions,andthatasu essfulresponseismandatoryforanyintera tion af-terwards. Hen e,bothpeersshouldbeawake(whi hisgovernedbytheMarkov haindepi tedinFig.8). Theprobabilitythat

y

isinONstateis

p

y

ON

=

µ

y

λ

y

+µ

y

. Similarly,theprobabilitythat

x

isONis

p

x

ON

. Sin etheseprobabilitiesare in-dependent, the probability that

x

an be onne ted to and hallenged by

y

,

p

c

(x, y)

, is

p

c

(x, y) = p

x

ON

· p

y

ON

(6)

Let's assume

y

hallenges

x

with a known frequen y,

f

. Depending on this frequen y, theprobability that thelying peer

x

is dete tedby

y

,

p

detect

(x, y)

, during thesystemtime

t

s

is:

p

detect

(x, y) = 1 − (1 − p

c

(x, y) · p(x, i))

f ·t

s

(7) Inotherwords,alyingpeerwillonlybenot dete tedifallthesu essfullysent hallengeshaveasu essfulresponseduring

t

s

. Notethat

p

detect

(x, y)

issimply the probability of nding the swit hed bits in the bit eld of

x

. The a tual dete tionhappenswhen

x

annotrespondtothe hallenge(e.g.,bynotsending aprooforsendingafalseproof).

With

m

peers working with

x

and thus hallenging

x

, the probability of dete tinganlyingpeer

x

,

p

detect

(x)

,is:

p

detect

(x) = 1 − Π

m

y=1

(1 − p

detect

(x, y))

(8) Again,

x

willonlybenot dete ted,ifall

m

peersfailtodete titslie.

Inadditiontodete tionprobability,averagedete tiontimeisalsoimportant asitae ts howfast we anre overfrom availability measurementerrors. To al ulatetheaveragedete tiontime,

t

detect

,werstneedto al ulatethe prob-abilityofdete tingalyingpeer

x

onthe

n

th

try. Wedenotethisprobabilityas

p

n

detect

(x, y)

and al ulateitas:

p

n

_detect

(x, y) = (1 − p

c

(x, y) · p(x, i))

n−1

· p

c

(x, y) · p(x, i)

(9) Sin ethisfollowsageometri distribution,themeannumberoftriesne essary for

y

todete t

x

is

1

p

c

(x,y)·p(x,i)

. Sin ethereare

m

peers hallenging

x

,

x

willbe dete tedwheneveroneofthesepeersdis oversitslie. Hen e,averagedete tion time of

x

depends on the minimum of the average number of tries ne essary among

m

peers. Sin ethe timebetweenea h hallengeis

1

f

, theaveragetime todete talyingpeer,

t

detect

is:

t

detect

=

1

f

· min

y

1

p

c

(x, y) · p(x, i)

(10) TounderstandifPa emaker andete tlyingpeerse iently,westudya net-workwhereea hpeeris hallengedby

m

= 5

peers. Peerssenda hallengeon e everyday(i.e,

f

= 1

)forasystemtime,

ts

= 15

days. Notethatunderuniform distribution,theaverageavailabilityofa hallengingpeeris 51%,whereasthis is8%forexponentialdistribution. Furthermore,basedonFig.23,

i

is sele ted as3. We nextanalyze the

p

detect

(x)

when the lying peer

x

is 5%, 10%, 25% and50%availableinthesystem. Moreover,weassume

x

liesuniformlyrandom basedonitsavailability. Inotherwords,ifitis5%availableittriestoimprove its availability by U(1%,95%). Based on this lying behaviour, the following (availability,averagelying)valuesareanalyzed: (0.05,0.48),(0.1,0.46),(0.25, 0.38)and (0.5, 0.26). Given this setting, using Eqs. 5-8, we plot

p

detect

(x)

in Fig.24. As expe ted, due to the low average availabilityof peers, the proba-bility of dete tinga lying peer is lower for exponential distribution ompared touniform distribution. However,note that, forboth ases,theprobabilityof dete tionishighifthelyingpeersareonline 50%. A tually,itisimportantto at hthesepeerssin elyingpeerswithlessthan10%availabilityarenotusing thesystem anyway. Similarly, Fig. 25showsthat under uniform distribution, thelyingpeersareexpe ted tobe aughtfasterthanexponentialdistribution. However,as alyingpeer'spresen ein reasesin thesystem,thedete tiontime de reasesa ordinglyforbothdistributions.

Ouranalysisresultsarealso onrmed bysimulationresults,whi hare de-pi tedin Fig.26andFig.27foruniformandexponentialdistributions, respe -tively. In our simulations, 5% of the population onsists of lying peers. We wait for ve daysto rea h a stable network before sending hallenges (this is thereasonwhythenumberoflyingpeersstaysatin bothguresupto5days andthenstartsde reasing). Theresultsshowthatthelyingpeersaredete ted in a ordan ewiththe analysis: fasterfor uniformdistribution and slowerfor

0

0.2

0.4

0.6

0.8

1

(0.05,0.48)

(0.1,0.46)

(0.25,0.38)

(0.5,0.26)

p

detect

(Real availability, Average lying)

unif. dist

exp, dist.

Figure24:

p

detect

(x)

withdierentavailabilityandlying hara teristi s

0

50

100

150

200

250

300

(0.05,0.48)

(0.1,0.46)

(0.25,0.38)

(0.5,0.26)

t

detect

(Real availability, Average lying)

unif. dist

exp, dist.

0.4

0.42

0.44

0.46

0.48

0.5

0.52

10

5

0

2000

4000

6000

8000

10000

System Availability

Number of Liars

Time (days)

real availability, exp. dist.

measured availability (1 challenges, 5 observers)

measured availability (2 challenges, 5 observers)

measured availability (3 challenges, 5 observers)

number of liars (1 challenges, 5 observers)

number of liars (2 challenges, 5 observers)

number of liars (3 challenges, 5 observers)

Figure 26: The measured and real availability, and the remaining numberof lying peersinthesystemwithtime(foruniformdistribution).

0.1

0.12

0.14

0.16

0.18

0.2

10

5

0

2000

4000

6000

8000

10000

System Availability

Number of Liars

Time (days)

real availability, exp. dist.

measured availability (1 challenges, 5 observers)

measured availability (2 challenges, 5 observers)

measured availability (3 challenges, 5 observers)

number of liars (1 challenges, 5 observers)

number of liars (2 challenges, 5 observers)

number of liars (3 challenges, 5 observers)

Figure 27: The measured and real availability, and the remaining numberof lying peersinthesystemwithtime(forexponentialdistribution).

exponential distribution. More spe i ally, aspredi ted from Fig.25, almost alllyingpeersare aughtafter the

10

th

day(i.e., in5days)foruniform distri-bution. On theother hand,for exponential distribution,almost 20%liarsare waiting to bedete ted after 15days(see Fig.27). However, wesee that on e the system starts dete ting and removing lying peers from the network, the measured systemavailability approa hesthe real system availability for both distributions.

8 Implementation

Asarststeptowardsrealdeployment,weranPa emakerover170nodesofthe Planet-Lab network for onemonth sin e September 24, 2008. In this se tion, wepresentthedetailsofourimplementationandinitialresults.

8.1 Implementation Details

Pa emaker was implemented asa single program, written in Obje tive-Caml. It uses openSSL for ryptography andnetwork libraries forpeer-to-peer om-muni ationsfrom ourpreviouswork (MLdonkey[14℄, Peerple [8℄), spe i ally, for marshaling messages and establishing ommuni ations. One of the most important features of Pa emaker is its ease of deployability: It took a single programmer less than a week to implement and fully deploy it (in luding an additionalauto-upgradefeature).

Inourimplementation, an optionis usedto de idetherole of thenodein thesystemat start-up. Theserolesare:

ˆ Client: Astandardpeerinthenetwork.

ˆ Server: Aserver peer,whi hdiusespulsesin thenetworkperiodi ally. ˆ Master: A logger peer, whi h doesn't run the Pa emaker proto ol but instead,allpeers onne ttoiteveryhourtouploadtheirlogs. Thisisdone tobeabletoanalyzetheselogstoevaluatetheperforman eofPa emaker. Theentiresystem anbedividedintothreeparts: (1)ourPa emaker proto- ol,(2)ameshproto olforbuildingthenetworkand(3)alesharingproto ol for logging and software update purposes. All of these proto ols ontain 12 messagesintotal,aslistedbelow:

ˆ Pa emaker: 4 messages, whi h are Pulse, Availability, Challenge andProof,andtheirhandlershavebeenimplemented. However,sin ewe arenotrunninganyrealappli ationandhavenoselshnodes,onlyPulse messagesaresentinourexperiments.

ˆ Mesh: The mesh proto ol builds the underlying network using 3 mes-sages. TheAskParentandAskParentReplymessagesare usedto estab-lishpermanentlinksbetweenpeersandpropagateotherparent andidates. TheDistan emessagehelpsde reasingthediameterofthenetwork. ˆ File sharing: Finally,weuse5messagesto transferles betweenpeers

andsyn hronizedire tories. These messagesallow:

 Logging: The log dire tory of every peer is syn hronized with the master. Inea hsyn hronizationonlynewormodiedlesare trans-fered.

 Softwareupdates: Todiuseanewbinaryinthenetwork,oneofthe lientsisupdated,whi hin turnstartsagossipofthisupdate. Thenalprogramis2000linesofObje tive-Caml ode,where400linesare messagedes riptions(amongwhi h 140lines areforPa emaker), 700lines are handlers(amongwhi honly70linesareforPa emaker;thelesharingfeature isthemostverbose)and250linesareforthemainfun tionality.

0.001

0.01

0.1

1

120

125

130

135

140

145

150

155

160

165

Absolute Availability Error

CDF of peers

pacemaker, one-hour pulse

pings, 5 observers

pings, 20 observers

3%

Figure28: AvailabilityErrorinPlanet-Labevaluation,165nodes

8.2 Deployment Details

TodeployPa emaker,weusedthree omputersinourlab. Oneoftheseserved astheserver andalsopropagatedsoftwareupdatesinthenetwork. These ond omputera tedas anormal lient to enablelo al debuggingofproblems. The third omputer wasthe master. Next, wegot a ess to onePlanet-Lab sli e, wherewerststartedwith10nodes,then 50nodesaftertwodays,andnally deployedPa emakeron170nodes. However,dueto arestartingproblemwith rond daemon after node rashes, the number of nodes running Pa emaker wasobserved to go down as lowas 145before the ronddaemon is restarted manually. Therefore, the resultspresented in the nextse tion areon oneday in luding165nodes.

8.3 Results

The initial results with our Planet-Lab deployment are depi ted in Fig. 28. Theseresultsshowthat usingPa emaker,theerror inavailabilitymeasure re-mainedbelow1%for90%of thepeers. Onlyforthe6%ofthepeers,theerror washigher than3%. Wealso deployed aping-basedavailabilitymeasurement systemto beableto ompareitagainstPa emaker. Thegureshowsthatthe ping-basedsystembothwith5and20observersperformsimilarlytoPa emaker intermsofa ura y. However,notethatPa emakera hievesthisa ura ylevel withalower ost. Furthermore,thesimilarityin a ura yperforman e isalso notsurprising be ausetheavailabilityofnodesin oursli edidnotshowmu h variation. Comparingping-basedsystemwith5observersagainst20observers also onrmsthis as thein rease in the numberof observersdid notimprove theavailabilitymeasure. Inthefuture,weplantousePa emakerinareal peer-to-peer ba kup storage system to evaluate its performan e in more dynami settings.

8.4 Dis ussion

OurgoalwithaPlanet-Labimplementationwastoshowtheease-of-deployment of Pa emaker. However, sin e there were no selsh peers in the network, we were notable to test the availabilitynoti ations and the veri ation of peer availability. Nevertheless, it is important to evaluate the ost of Pa emaker when Availability,Challengeand Proofmessagesare sent. Espe ially the ostofbandwidthneedstobe onsideredsin eitisusuallytherarestresour e in peer-to-peernetworks.

InPa emaker, bandwidth is mainly onsumed during theex hange of sig-natures andkeysin pulsemessages. With RSA,these signaturesandkeysare typi ally 256 bytes long. This ost an further be redu ed by using ellipti urves,whi ha hieveagoodlevelofse uritywitharound15bytes. Therefore, wedonotexpe tPa emakerto in urhigh osts. Forinstan e, ifweuseRSA, thesizeofthemessagessentbyPa emaker anbe al ulatedapproximatelyfor ayear(i.e,

N

t

= 8760

)as:

ˆ Pulse : 800B(2 keysand 1signature)

ˆ Availability : 1400B(biteld[

N

_

t

℄+1signature) ˆ Challenge : 70B(a non eof64B)

ˆ Proof : 900B(non e+1key+2signatures)

Note that the pulse message is sent on e per period P to all the neighbors. Other messages would probably only be sent on e a day between two peers working together. Consequently, weexpe t the bandwidth ostof Pa emaker tobenegligible.

9 Related Work

Inthis se tion,werstpresent urrentresear honpeer-to-peernetworksthat reliesonavailabilityinformation. Su hsystemsserveasourmainmotivationto provideavailabilityinformationse urelyinthepresen eselshpeers. Next,we dis ussrelatedworkonavailabilitymeasurement,fo usingspe i allyontheir operationinthepresen eofselshpeers.

9.1 Uses of AvailabilityInformation in P2P Networks The majority of the resear h on building peer-to-peer networks heavily relies on information about stability oravailability of peers. Indeed, many systems relyonastable oreorsuper-peers,whi haresele tedfortheirhighavailability in the system. Forinstan e, [1℄ and[4℄ report that, in Gnutellaand Overnet, respe tively, the peers with higher availability tend to be more stable than other peers. Based on this result, [1℄ proposes a proto ol that builds amore stablenetworkbysele tingpeerswithhigheravailability. However,theproposed solution annot opewithselshpeersthatmightlieabouttheirrealavailability togetabetterstatusinthesystem. Similarlyin[21℄,agradienttopologyisbuilt sothatthemoststablepeersareatthe ore ofthenetworkandthelessstable peers stay onthe border. However,selsh peers an laim higheravailability to bein ludedin the ore, andthen refusetoserverequests eventhoughthey

Pa emaker AVMON Ar hite ture Pulses Pings Durability Unlimited Limited( hurn)

Supported Topologies

Internet Yes Yes Ad-ho Networks Yes No So ialNetworks Yes No FirewalledPeers Yes No

Vulnerabilities

SelshPeers Resilient NotTreated ColludingPeers Not Treated Vulnerable

Figure29: ComparisonwithAVMON[19℄. Seese tion9.2fordetails.

benetfromtheirgoodpositioninthenetwork.In[9℄,availabilityinformationis usedtosele tsuper-peersinthenetworktobuildatop-levelChordDistributed Hash Table (DHT) over another less stable DHT. Again, selsh peers might managetobein ludedinthetop-levelDHT,andusetheirpositiontode rease theirload.

In addition to help build more stable networks, availability information is alsouseful forrepairingthe network. Forinstan e, using availability informa-tion, repla ement poli ies de rease the ee t of hurn in a peer-to-peer sys-tem [10℄. It was shown that, although performing well, the performan e of arandom repla ement annot rea h the performan e of arepla ement poli y basedon hoosingpeerswithmaximumavailability. Similarly,in[7℄, availabil-ityinformationisusedtoproa tivelyrepairfragmentsinapeer-to-peerstorage system based on an estimation of the failure rate. Finally, in [22℄ an obje t repli amaintenan esystemisstudiedundertemporaryandpermanentfailures fordierentpeer-to-peersystems. Itisshownthatdatatendstoa umulateon nodes with high availabilityand unlimited apa ity. Essentially, ifthe apa -ity islimited,the performan e degradeswhenthenodeswithhigh availability be omesaturatedasthenodeswithlowavailabilitytriggermanyrepairs. Ob-viously, iftheavailabilityinformationis ompromisedin anyofthese systems, repairswouldnotbepossible.

9.2 Comparison with AVMON

Availability measurement systems an be lassiedinto two ategories: lo k-basedsystems,wheremeasuresarebasedonthelo al lo kandping-based sys-temswhere measuresaredonebyhellomessages. Pa emakerintrodu esanew ategory,pulse-basedsystems,wheremeasuresarebasedonpulsesoodedinthe network. Clo k-basedsystems su h as [18℄ are obviously vulnerable to selsh peers.

Weintrodu edandevaluatedping-basedsystemsinSe tion6.3. Ourresults show that, to get the same level of a ura y asPa emakerin a realisti sys-tem, apeer in aping-based systemneeds to send 25 ping messagesperhour, while a peer in Pa emakeronly needs to send 5 diusion messages per hour. Indeed, in ping-based systems, peers an onlymonitor availability when they are online, so moremonitorsare needed to ope with hurn. Moreover, their

measures be omeunavailable assoonasthey leavethe system. Finally, peers behindrewalls annotbemonitored,whereasPa emaker anstillrea hthem. Therefore,measurementsarelessa urate, lesse ientandlessdurable.

Tothe best of ourknowledge, AVMON [19℄ isthe only ping-basedsystem designed with se urityin mind. Byusing ahash fun tion to mat h observers andobservedpeers,AVMONtries toavoidthat peers laimhigheravailability thanthe realityby olludingwith otherpeers. AVMONsuers fromboththe drawba ksof ping-basedsystemsandthedrawba ksofits hash-baseds heme, asdetailed in [13℄. Fromase uritypoint-of-view,selsh observers anstill lie abouttheavailabilityofthepeersthattheyaresupposedtomonitor. Moreover, thehashme hanismisvulnerableto ollusion: peers an hangetheirlistening portuntiltheyarea eptedasobserversforthepeerswithwhomtheywantto ollude.

9.3 Colluding Peers

As other availability measurement systems, Pa emaker annot yet ope with olludingpeers,but itis,tothebestofourknowledge,theonlyoneresilientto selshpeers. Nevertheless,wethinkthat,rst, ollusionishardertoimplement thanselshness,andse ond,itshouldbepossibletoextendPa emakerto ope with ollusion.

Indeed, selshnessrequires only small modi ations of thesoftware (to lie on bitmaps) or of the environment (to lter out pings or requests in ping-based systems). On the ontray, ollusion requires deep modi ationsof the software(extensionoftheproto ol)and ompli ityofotherpeersthatneedto bedis overedonthenetwork.Fortherstpart,thereisagame-theori in entive not to diuse su h modiedsoftware for olluding: the bigger the number of peers olluding, the smaller the benet for ea h olluding lient. Moreover, lientstryingtodis overother olluding lientsopenly anbedete tedby honey-pots, i.e. lients a epting to ollude only to dete t olluders, and able to bla klistthemonasystem-wides ale. Consequently, ollusion anbeexpe ted tobelimitedtoafewmanually reatedgroupsofmodied lientstrustingea h other.

Thereare alsoseveralapproa hestoextendPa emakerto opewith ollu-sion. A rstapproa hwouldbetoinsertin thepulsethepathofIP addresses followedduringitsdiusion. Su has hemewouldhelpbla klisting lientsthat keepdiusingoldpulsestootherpeerstodisruptthesystem. Anotherapproa h wouldbetotra kmodi ationsofthehistoryof pulsesofa lient, todete tif anoldpulseisadded,tolimitthetimeduringwhi h ollusion anhappen. Fi-nally,wearealsoinvestigatingamoreinterestingapproa h, losertoPa emaker spirit, basedontheuseofMerkletrees[17℄,to a tuallyen odethepresen eof apeer inthesystemdire tlyinthepulse.

10 Con lusion

Inthispaper,wehavepresentedasimplebut e ientwayofmonitoring avail-ability in peer-to-peer systemsin the presen e of selsh peers. Our proto ol, Pa emaker,usesasetofserverstopropagate ryptographi pulsemessagesina meshofpeers,allowingthemtomeasureand he kthehistoryofavailabilityof

otherpeerseasilyat anytime. Pa emakerisresistantto selshbehaviors,and inparti ular to lyingpeers,whi hlie abouttheiravailabilityto gaina ess to moreresour esin thesystem.

We have evaluated Pa emakerthrough analysis,simulationsand deployed the proto ol on a Planet-Lab testbed. Our results show that Pa emaker is a urate,ableto dete t selshbehaviors,lessexpensivethan ompetitorsand easyto deploy. Furthermore,the lowoverheadindu ed byPa emakerenables nottohamperthes alabilityofthepeer-to-peeroverlaynetwork.

Pa emakeralsointrodu esanewnetworkar hite ture,pulse-basedsystems, that,wethink, ouldhavemultipleappli ations inself-organizingsystems. We arenowinvestigatingsomeoftheseappli ations, forexamplein the ontextof sensor networks. As dis ussed in se tion 9.3, weare alsoworking ondierent approa hestoextendPa emakerto opewith olludingpeers.

Referen es

[1℄ Ranjita Bhagwan, Stefan Savage, and Georey Voelker. Understanding availability. InIPTPS,Int'lWork.on Peer-to-PeerSystems,2003. [2℄ Ranjita Bhagwan, KiranTati, Yu-Chung Cheng, Stefan Savage, and

Ge-oreyM.Voelker. Totalre all: systemsupportforautomatedavailability management. In NSDI, Symp. on Networked Systems Design and Imple-mentation,2004.

[3℄ William J. Bolosky, John R. Dou eur, David Ely, and Marvin Theimer. Feasibility of a serverless distributed le system deployed on an existing set of desktop p s. In SIGMETRICS, Int'l Conf. on Measurement and Modelingof ComputerSystems,2000.

[4℄ Fabian E. Bustamante and Yi Qiao. Friendships that last: peer lifespan and its role in p2p proto ols. In Int'l workshop on Web ontent a hing anddistribution,2004.

[5℄ FrankDabek,M. FransKaashoek,David Karger,RobertMorris, andIon Stoi a. Wide-area ooperativestoragewith CFS. InSOSP,ACM Sympo-sium onOperating SystemsPrin iples,2001.

[6℄ T.DierksandE.Res orla.RFC4346: TheTransportLayerSe urity(TLS) proto olversion1.1. IETF,April2006.

[7℄ AlessandroDuminu o,ErnstWBiersa k,andTaoukEn Najjary. Proa -tive repli ation in distributed storage systems using ma hine availability estimation. InCoNEXT,Int'lConf.onemergingNetworkingEXperiments andTe hnologies, 2007.

[8℄ Anh-Tuan Gai, Fabri e Le Fessant, and Laurent Viennot. http://www.peerple.net/,2007.

[9℄ L. Gar ias-Eri e, E.W.Biersa k,P.A. Felber,K.W. Ross,and G. Urvoy-Keller. Hierar hi al peer-to-peer systems. In Euro-Par, Intl. Conf. on Parallel andDistributedComputing,2003.