Entropy: a Consolidation Manager for Clusters

HAL Id: inria-00320204

https://hal.inria.fr/inria-00320204v2

Submitted on 11 Sep 2008

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

Fabien Hermenier, Xavier Lorca, Jean-Marc Menaud, Gilles Muller, Julia

Lawall

To cite this version:

Fabien Hermenier, Xavier Lorca, Jean-Marc Menaud, Gilles Muller, Julia Lawall. Entropy: a

Consol-idation Manager for Clusters. [Research Report] RR-6639, INRIA. 2008. �inria-00320204v2�

a p p o r t

d e r e c h e r c h e

N

0

2

4

9

-6

3

9

9

IS

R

N

IN

R

IA

/R

R

--6

6

3

9

--F

R

+

E

N

G

Thème COM

Entropy: a Consolidation Manager for Clusters

Fabien Hermenier — Xavier Lorca — Jean-Marc Menaud — Gilles Muller — Julia Lawall

N° 6639

Unité de recherche INRIA Rennes

Fabien Hermenier

∗

, Xavier Lor a

∗

, Jean-Mar Menaud

∗

, Gilles Muller

†

, JuliaLawall

‡

ThèmeCOMSystèmes ommuni ants ProjetOBASCO

Rapportdere her he n° 6639Septembre200823pages

Abstra t: Clustersprovidepowerful omputingenvironments,butinpra ti e mu hofthispowergoestowaste,duetothestati allo ationoftaskstonodes, regardless of their hanging omputationalrequirements. Consolidation is an approa h that migrates taskswithin a luster astheir omputational require-ments hange,bothtoredu ethenumberofnodesthatneedtobea tiveandto eliminatetemporaryoverloadsituations. Previous onsolidationstrategieshave reliedontaskpla ementheuristi sthatuseonlylo aloptimizationandtypi ally donottakemigrationoverheadintoa ount. However,heuristi sbasedononly lo aloptimizationmaymissthegloballyoptimalsolution,resultingin unne es-saryresour eusage,andtheoverheadformigrationmaynullify thebenetsof onsolidation.

Inthis paper, we propose the Entropy resour emanager forhomogeneous lusters, whi h performs onsolidation based on onstraint programming and takesmigrationoverheadintoa ount. Theuseof onstraintprogramming al-lowsEntropytondmappingsoftaskstonodesthatarebetterthanthosefound byheuristi sbasedonlo aloptimizations,andthatarefrequentlyglobally opti-malinthenumberofnodes. Be ausemigrationoverheadistakenintoa ount, Entropy hoosesmigrationsthat anbeimplementede iently,in urringalow performan eoverhead.

Key-words: Virtualization,Consolidation, Cluster,Re onguration, Migra-tion

∗

DépartementInformatique,É oledesMinesdeNantesINRIA,LINA,CNRS rst-name.lastnameemn.fr

†

É oledesMinesdeNantesINRIAGilles.Mulleremn.fr

‡

Résumé : Les grappes de serveurs fournissent un environnement de al ul puissant. Cependant, une partiede ette puissan eest perdue parune allo a-tion statique des tâ hes sur les n÷uds de al uls qui ne tient pas ompte de la variations de leurs besoins. En regroupant es tâ hes dynamiquement, la onsolidationpermet de réduire lenombre de n÷uds né essaires àl'exé ution des al uls, tout en éliminant les situations de saturations temporaires. Les stratégiesde onsolidationa tuellesefo alisentsurune optimisationlo aledu pla ementdestâ hesetnetiennentpas omptedel'impa tdesmigrations. Ces heuristiquesmanquentlanotiond'optimalitéglobalequiimpliqueune onsom-mationderesour esquin'estpasné essaire. Deplus,l'absen ede onsidération desmigrationsréduitdemanièrenotablelesperforman esdelagrappe,limitant ainsil'interêtdela onsolidation.

Cetarti leprésenteEntropy,ungestionnairede onsolidationpourgrappes homogènesutilisantuneappro hebaséesurlaprogrammationpar ontrainteset tenant omptedel'impa tdesmigrations. Notreappro hepermetlaréalisation d'un agen ement des tâ hes globalement meilleur par rapport aux appro hes lassiques à base d'heuristiques. De plus, en tenant ompte des migrations destâ hessur lagrappe,l'impa t de la onsolidationsur lesperforman es est diminuée.

Mots- lés : Virtualisation, Consolidation,Grappe,Re onguration, Migra-tion

1 Introdu tion

GridandCluster omputingarein reasinglyusedtomeetthegrowing ompu-tationalrequirementsofs ienti appli ations. Inthissetting,auserorganizes ajobasa olle tionoftasksthatea hshouldrunonaseparatepro essingunit (i.e,anentirenode,aCPU, ora ore)[6℄. Todeploythejob,theusermakesa requesttoaresour ebroker,spe ifyingthenumberofpro essingunitsrequired and the asso iated memoryrequirements. If therequested CPU and memory resour esareavailable,thejobisa epted. Thisstati strategyensuresthatall jobs a eptedinto the lusterwillhavesu ientpro essingunits andmemory to omplete their work. Nevertheless, it an lead to awaste of resour es, as manys ienti omputationspro eed in phases,notallofwhi huse allofthe allo atedpro essingunitsatalltimes.

Consolidationisawell-knownte hniquetodynami allyredu ethenumber ofnodesusedwithin arunning lusterbyliberatingnodesthatarenotneeded bythe urrentphaseofthe omputation. Liberatingnodes anallowmorejobs to be a epted into the luster,or anallow powering down unused nodesto saveenergy. Tomake onsolidationtransparent,regardlessoftheprogramming language,middleware,oroperatingsystemusedbytheappli ation,itis onve-nienttohostea htaskin avirtualma hine(VM), managedbyaVMMonitor (VMM)su hasXen[1℄,forwhi he ientmigrationte hniquesareavailable[5℄. Consolidationthen amountsto identifying ina tiveVMsthat anbemigrated to other nodes that havesu ient unused memory. A VMthat is ina tiveat onepointintimemay,however,laterbe omea tive,possibly ausingthenode thatishostingittobe omeoverloaded. A onsolidationstrategymustthusalso moveVMsfromoverloadednodestounderloadedones.

Severalapproa hesto onsolidationhavebeenproposed[3,7,11℄. These ap-proa hes,however,havefo usedon howto al ulate anew onguration, and havenegle tedtheensuingmigrationtime. However, onsolidationisonly ben-e ialwhentheextrapro essingunittimein urredformigrationissigni antly lessthantheamountofpro essingunittimethat onsolidationmakesavailable. Whilemigrating asingleXen VM anbeverye ient,in urring anoverhead ofonlybetween6and26se ondsin ourmeasurements,itmaynotbepossible tomigrateaVMtoits hosendestinationimmediately;insteadotherVMsmay rst haveto be movedoutof the way tofree su ientmemory. Delayingthe migrationofanina tiveVMonly ausesunne essarynodeusage. Ontheother hand,delayingthemigration ofan a tiveVM that isrunningon apro essing unit overloadedwith

n

otherVMsdegradestheperforman eofthose VMsfor a period of timeby afa tor of

n

as ompared to a non- onsolidatedsolution, in whi h ea h VM alwayshas itsown pro essing unit. In reasing thenumber ofVMsthatneedtomigrateas omparedtotheamountofavailable resour es only exa erbatesthese problems. Thus, itis essentialthat onsolidationbeas e ientandrea tiveaspossible.

Inthispaper,weproposeanewapproa hto onsolidationinahomogeneous lusterenvironmentthat takesintoa ountboththeproblem ofallo atingthe VMstotheavailablenodesandtheproblemofhowtomigratetheVMstothese nodes. Our onsolidationmanager,Entropy,worksintwophasesandisbasedon onstraintsolving[2,14℄. Therstphase,basedon onstraintsdes ribingtheset ofVMsandtheirCPUandmemoryrequirements, omputesapla ementusing theminimumnumberof nodesandatentativere onguration planto a hieve

thatpla ement. These ondphase,basedonarenedsetof onstraintsthattake feasiblemigrationsintoa ount,triestoimprovetheplan,toredu ethenumber ofmigrationsrequired. Inourexperiments,usingtheNASGridben hmarks[6℄ on a luster of 39 AMD Opteron 2.0GHz CPU unipro essors, we nd that a solutionwithout onsolidationuses24.31 nodesperhour, onsolidation based onthepreviously-usedFirstFitDe reasing(FFD)heuristi [3,17,18℄uses15.34 nodesperhour,and onsolidationbasedonEntropyusesonly11.72nodesper hour,asavingsofmorethan50%as omparedtothestati solution.

The remainder of this paper is organized as follows. Se tion 2 gives an overview of Entropy. Then, Se tion 3 des ribes how Entropy uses onstraint programmingtodeterminetheminimumnumberofnodesrequiredbya olle -tionofVMs,andSe tion4presetnshowEntropyuses onstraintprogramming tominimizethere ongurationplan. Finally,Se tion5evaluatesEntropyusing experimental results on a luster of the Grid'5000 experimental testbed, Se -tion6des ribesrelatedwork,andSe tion7presentsour on lusionsandfuture work.

2 System Ar hite ture

A luster typi ally onsistsofasinglenodededi ated to luster resour e man-agement, a olle tionof nodes that anhostuser tasks, and other spe ialized nodes,su hasleservers. EntropyisbuiltoverXen3.0.3[1℄andisdeployedon thersttwo. It onsistsofare ongurationenginethatruns onthenodethat provides luster resour emanagement and aset of sensors that run in Xen's Domain-0onea hnodethat anhostusertasks,i.e., VMs.

Thegoalof Entropyis toe ientlymaintainthe lusterina onguration, i.e. a mapping of VMs to nodes, that is (i) viable, i.e. that gives everyVM a esstosu ientmemoryandeverya tiveVMa esstoownpro essingunit, and(ii)optimal,i.e. thatusestheminimumnumberofnodes. Forthis,the En-tropyre ongurationengineiteratively1)waitstobeinformedbytheEntropy sensorsthat aVM has hangedstate, froma tiveto ina tiveorvi e versa, 2) triesto omputeare ongurationplanstartingfromthe urrent onguration that requiresthe fewestpossiblemigrations and leavesthe luster in aviable, optimal onguration, and 3) if su essful, initiates migration of the VMs, if thenew ongurationuses fewernodes thanthe urrentone,orifthe urrent onguration is not viable. The re onguration engine then waits 5 se onds before repeating theiteration, to a umulate new information about resour e usage. In this pro ess, the Entropy sensors periodi ally send requests to the HTTPinterfa eoftheXenhypervisoronthe urrentnodetoobtaintheCPU usageofthelo alVMs,andinferstate hangesfrom thisinformation. An En-tropysensoralsore eivesamessagefromthere ongurationenginewhenaVM shouldbemigrated,andsendsrequests totheXenhypervisorHTTP interfa e toinformitwhi hVMshould bemigratedandtowhi hnode.

Previousapproa hestoa hievingaviable, ongurationhaveusedheuristi s inwhi halo allyoptimalpla ementis hosenforea hVMa ordingto some strategy[3, 7, 11, 17℄. However, lo al optimization does notalwayslead to a globallyoptimalsolution,andmayfail toprodu eanysolutionatall. Entropy insteadusesConstraintProgramming(CP),whi h isabletodeterminea glob-ally optimalsolution, if oneexists, by using amore exhaustive sear h, based

//Instantiatinganewproblem 1

Problempb=newProblem(); 2

3 //De larationofthevariablesandtheirasso iateddomains 4 IntDomainVarx=pb.makeEnumIntVar("x",0,10); 5 IntDomainVary=pb.makeEnumIntVar("y",0,10); 6 IntDomainVarz=pb.makeEnumIntVar("z",0,10); 7 8

//De larationofthe onstraint 9

IntExpexp=pb.plus(x,y ); 10

Constraint =pb.eq(exp,z); 11

12 //The onstraintispluggedintotheproblem 13

pb.post( ); 14

15

//Westartsolving. 16

pb.solve(); 17

Figure1: Java odeusingtheCho olibraryforndingvaluesofvariables

x

,

y

, and

z

intherange0to10, su hthat

x

+ y = z

onadepth rstsear h. TheideaofCP isto deneaproblem bystating on-straints(logi alrelations)that mustbesatisedbythesolution. AConstraint Satisfa tionProblem(CSP)isdenedasasetofvariables,asetofdomainsthat representtheset ofpossiblevaluesthat ea hvariable antakeonandaset of onstraintsthatrepresentrequiredrelationsbetweenthevaluesofthevariables. A solution fora CSP isavariable assignment(a valuefor ea h variable)that simultaneouslysatisesthe onstraints.TosolveCSPs,EntropyusestheCho o library[10℄, whi h ansolveaCSPwhere thegoalistominimizeormaximize thevalueofasinglevariable. Figure1showsanexampleofCho o ode,whi h solvesthe problem of ndingvalues ofvariables

x

,

y

, and

z

in the range0to 10,su hthat

x

+ y = z

.

Be ause Cho o an only solve optimization problems of a single variable, there ongurationalgorithmpro eedsintwophases. Therstphasendsthe minimumnumber

n

ofnodesthatarene essarytohostallVMs. Werefertothis problem as theVirtual Ma hine Pa king Problem (VMPP).These ond phase minimizes the re onguration time, given the hosennumber of nodes

n

. We refer to this problem as the Virtual Ma hine Repla ement Problem (VMRP). Solvingtheseproblems maybetime- onsuming. Whilethere onguration en-ginerunsonthe lusterresour emanagementnode,andthusdoesnot ompete withVMsforCPUandmemory,itisimportanttoprodu eanew onguration qui klytomaximizethebenetof onsolidation. Thus,welimitthetotal om-putation time for bothproblems to 1minute, of whi h the rst phase hasat most15se onds,andthese ondphasehastheremainingtime. Thesedurations are su ienttogiveanontrivialimprovementin thesolution,as omparedto theFFDheuristi ,asshowninSe tion5. Furthermore,the onstraintsolveris implementedsu hthat ifthe omputationtimesoutwithoutthesolverhaving found a solution that has been proved to be optimal, then the best solution foundsofarisreturned.

3 The Virtual Ma hine Pa king Problem

The obje tive of the VMPP is to determine the minimum number of nodes that an host the VMs, given their urrent pro essing unit and memory re-quirements. Werstpresentseveralexamplesthatillustratethe onstraintson theassignmentof VMs to nodes, then onsider how to express the VMPP as a onstraintsatisfa tionproblem,andnally des ribesomeoptimizationsthat weuseinimplementingasolverforthisproblemusing Cho o.

3.1 Constraints on the assignment of VMs to nodes Ea h node in a luster provides a ertain amount of memory and number of pro essing units, and ea h VM requires a ertain amount of memory, and, if a tive, apro essingunit. These onstraintsmustbesatisedby aviable on-guration. Forexample,ifeverynodeisaunipro essor,thenthe onguration inFigure2(a)isnotviablebe auseitin ludestwoa tiveVMsonnode

N

1

. On theotherhand,the ongurationinFigure2(b)isviablebe auseea hVMhas a esstosu ientmemoryandea hnodehosts atmostonea tiveVM.

(a) Non-viable onguration

(b) Viable on-guration

Figure2: Non-viableandviable ongurations.

VM

2

and

VM

3

area tive

(a) A minimal viable ongura-tion (b) Another minimal viable onguration

Figure3: Viable ongurations.

VM

2

and

VM

3

area tive

Toa hieve onsolidation,wemust ndaviable ongurationthatusesthe minimumnumberofnodes. Forexample,the ongurationshowninFigure2(b) isviable,but itisnotminimal,be ause,asshownin Figure3(a),

VM

2

ouldbe hosted onnode

N

2

, using onefewernode. The problem ofnding aminimal, viable onguration is redu tibleto theNP-Hard 2-Dimensional Bin Pa king Problem [15℄, where thedimensions orrespondtothe amountof memoryand numberofpro essingunits.

The VMPP may have multiple solutions, as illustrated by Figures 3(a) and 3(b), whi h both use two nodes. These solutions, however, may not all entailthesamenumberofmigrations. Forexample,ifweperform onsolidation

withFigure2(b)astheinitial onguration,weobservethatonly1migrationis ne essarytorea hthe ongurationshowninFigure3(a)(moving

VM

2

onto

N

2

), but2arene essarytorea hthe ongurationshowninFigure3(b)(moving

VM

3

onto

N

2

and

VM

1

onto

N

3

).

3.2 ExpressingtheVMPPasa onstraintsatisfa tion prob-lem

ToexpresstheVMPPasaCSP,we onsiderasetofnodes

N

andasetofVMs

V

. Thegoalistondaviable ongurationthatminimizesthenumberofnodes used. Thenotation

H

i

,dened below,isusedtodes ribea onguration. Denition3.1 Forea hnode

n

i

∈ N

,thebitve tor

H

i

= hh

i1

, . . . , h

ij

, . . . , h

ik

i

denotesthesetofVMsassignedtonode

n

i

(i.e.,

h

ij

= 1

ithenode

n

i

ishosting the VM

v

j

).

We express the onstraints that a viable onguration must respe t ea h VM's pro essing unit and memory requirements as follows. Let

R

p

be the ve torof pro essing unit demand of ea h VM,

C

p

be the ve tor of pro essing unit apa ityasso iatedwithea hnode,

R

m

betheve torofmemorydemand of ea h VM, and

C

m

be the ve tor of memory apa ity asso iated with ea h node. Then,thefollowinginequalitiesexpressthepro essingunitandmemory onstraints:

R

p

· H

i

≤ C

p

(i)

∀n

i

∈ N

R

m

· H

i

≤ C

m

(i) ∀n

i

∈ N

Giventhese onstraints,ourgoalistominimizethevalueofthevariable

X

, denedasfollows,wherethevariable

u

i

is1ifthenode

i

hostsatleastoneVM, and0otherwise.

X

=

X

i∈N

u

i

,

where

u

i

=

(

1, ∃v

j

∈ V | h

ij

= 1

0,

otherwise (1) Welet

x

vmpp

denotethissolution.

Thesolverdynami allyevaluatestheremainingfreepla e(intermsofboth pro essing unit and memory availability) on ea h node during the sear h for a minimumvalue of

X

. This is done by solvingMultiple Knapsa k problems usingadynami programmingapproa h[16℄.

3.3 Optimizations

Inprin iple,the onstraintsolvermustenumerate ea h possible onguration, he kwhether itis viable,and omparethenumberofnodesto theminimum foundsofar. Inpra ti e,thisapproa hisunne essarilyexpensive. Our imple-mentationredu esthe omputation ostusinganumberofoptimizations.

Cho oin rementally he kstheviabilityandminimalityofa ongurationas itisbeing onstru tedanddis ardsapartial ongurationassoonasitisfound to benon-viableorto use morethan theminimum numberofnodes foundso far. Thisstrategyredu esthenumberof ongurationsthatmustbe onsidered.

Itfurthermoretriesto dete tnon-viable ongurationsasearlyaspossible,by using a rst fail approa h [8℄ in whi h VMs that are a tive and have greater memory requirementsare treated earlier than VMs with lesser requirements. Thisstrategyredu esthe han eof omputinganalmost omplete onguration andthen ndingthattheremainingVMs annotbepla edwithin the urrent minimumnumberofnodes.

Inprin iple,the domainof thevariable

X

istheentire set ofnon-negative integers. We an, however,signi antly redu e the sear h spa e and improve theperforman e ofthe solverbyidentifying lowerand upper bounds that are losetotheoptimalvalueandareeasyto ompute. Asalowerbound,wetake thenumberof a tiveVMsdividedbynumberof pro essingunitsavailableper node(Equation 2). If we ndasolutionusing this numberofVMs, then itis known to be optimal with no further tests. As anupperbound, we takethe value omputed by the First Fit De reasing(FFD) heuristi , whi h has been usedinotherworkon onsolidation[3,17,18℄(Equation3). TheFFDheuristi assignsea hVMtotherstnodeitndssatisfyingtheVM'spro essingunitand memoryrequirements,startingwiththe VMsthat requirethe biggestamount ofmemory. This heuristi tendsto provideagood value,in averyshort time (less than a se ond) but the result is not guaranteed to be optimal and the heuristi mayindeed notndanysolution. Inthelatter ase,theupperbound istheminimumofthenumberofnodesandthenumberofVMs.

X

≥

min













X

v

i

∈V

R

p

(i)

C

p

(j)













, n

j

∈ N

(2)

X

≤

(

x

d min

(|N |, |V|),

otherwise (3)

Furthermore, we observe that some nodes or VMs may be equivalent, in termsof theirpro essingunit and memory apa ityordemand,andtryto ex-ploitthisinformationtoimprovethepruningofthesear htree. Iftheresour es oeredbyanode

n

i

are notsu ient tohostaVM

v

i

, thentheyarealso not su ienttohostanyVM

v

j

withthesamerequirements. Furthermore,theVM

v

i

annot behosted byanyother node

n

j

withthesame hara teristi sas

n

i

. Theseequivalen esaredenedasfollows:

∀n

i

, n

j

∈ N | n

i

≡ n

j

⇔ C

p

(i) = C

p

(j) ∧

C

m

(i) = C

m

(j)

(4)

∀v

i

, v

j

∈ V | v

i

≡ v

j

⇔ R

p

(i) = R

p

(j) ∧

R

m

(i) = R

m

(j)

(5)

4 The Virtual Ma hine Repla ement Problem Thesolutionto theVMPPprovidestheminimumnumberofnodesrequiredto hosttheVMs. However,as illustrated in Se tion 3.1, foragiven olle tionof

VMs, there anbe multiple ongurations that minimize the number of used nodesandthenumberofmigrationsrequiredtorea hthese ongurations an vary. Theobje tive of theVirtual Ma hine Repla ement Problem (VMRP) is to onstru tare ongurationplanforea hpossible ongurationthatusesthe number of nodes determined by the VMPP, and to hoose the one with the lowest estimated re onguration ost. In the rest of this se tion, we onsider howto onstru ta re ongurationplan, howto estimate its ost, and howto ombine thesestepsintoasolutionfortheVMRP.

4.1 Constru ting a re onguration plan

The onstraintof viabilityhasto betakenintoa ountbothin the nal on-guration andalsoduring migration. A migrationisfeasibleifthe destination node has a su ient amount of free memory and, when the migrated VM is a tive,ifthedestinationnodehasafreepro essingunit. However,toobtainan optimalsolutionitisoftenne essaryto onsidera ongurationin whi hsome migrations are notimmediately feasible. We identify twokinds of onstraints onmigrations: sequential onstraintsand y li onstraints.

A sequential onstraint o urs when one migration an only begin when anotheronehas ompleted. Asanexample, onsiderthemigrationsrepresented bythere ongurationgraphshown inFigure 4. Are ongurationgraphisan oriented multigraph where ea h edge denotes the migration of aVM between twonodes. Ea h edge spe ies the virtualma hine to migrate,the amountof memory

R

m

required to hostitand its state

A

(a tive) or

I

(ina tive). Ea h node denotes a node of the luster, with its urrent amount of free memory

C

m

and its urrent free apa ity for hosting a tive virtual ma hines

C

p

. In the example in Figure 4, it is possible to onsolidate the VMs onto only two nodes, by moving

VM

1

from

N

1

to

N

2

and moving

VM

2

from

N

2

to

N

3

. But these migrations annothappeninparallel,be auseaslongas

VM

2

ison

N

2

, it onsumesall ofthe available memory. Thus, themigration of

VM

1

from

N

1

to

N

2

anonlybeginon ethemigrationof

VM

2

from

N

2

to

N

3

has ompleted.

N

1

C

m

=400,

C

p

=0

N

2

C

m

=0,

C

p

=1

N

3

C

m

=400,

C

p

=0

VM

1

R

m

=200,

A

VM

2

R

m

=400,

I

Figure4: Asequen eofmigration

A y li onstrainto urswhenasetofinfeasiblemigrationsformsa y le. An example is shown in Figure 5(a), where, due to memory onstraints,

VM

1

anonlymigratefrom node

N

1

to node

N

2

when

VM

2

hasmigratedfrom node

N

2

,and

VM

2

anonlymigratefromnode

N

2

tonode

N

1

when

VM

1

hasmigrated fromnode

N

1

. We anbreaksu ha y lebyinsertinganadditionalmigration. Apivot nodeoutsidethe y leis hosentotemporarilyhostoneormoreofthe VMs. Forexample,in Figure5(b),the y lebetween

VM

1

and

VM

1

isbrokenby migrating

VM

1

tothenode

N

3

,whi hisusedasapivot. Afterbreakingall y les of infeasiblemigrations in this way,anorder anbe hosenfor themigrations asin thepreviousexample. Thesemigrations in lude moving theVMsonthe pivotnodesto theiroriginaldestinations.

N

1

C

m

=0,

C

p

=-1

N

2

C

m

=0,

C

p

=1

VM

1

R

m

=256,

A

VM

2

R

m

=256,

I

(a) Inter-dependant migrations

N

1

C

m

=0,

C

p

=-1

N

3

C

m

=512,

C

p

=1

N

2

C

m

=0,

C

p

=1

VM

1

R

m

=256,

A

VM

1

R

m

=256,

A

VM

2

R

m

=256,

I

(b)Abypassmigrationbreaksthe y le

Figure5: Cy leofnon-feasiblemigrations

Taking the aboveissues into a ount, the algorithm for onstru tinga re- ongurationplanisasfollows. Startingwithare ongurationgraph,therst step is to identify ea h y le of infeasible migrations, identify a node in ea h su h y lewhere theVMsto migratehavethe smallesttotalmemory require-ment,andsele tapivot nodethat ana omodatetheseVMs'pro essingunit andmemoryrequirements. Theresultis anextended re ongurationgraphin whi h for ea h su h hosen VM, the migration from the urrent node to the destinationnode inthedesired ongurationisrepla ed byamigrationtothe pivotfollowedbyamigrationtothedestinationnode. Subsequently,thegoalis totryto doasmanymigrations inparallel aspossible,so thatea hmigration will takepla e with theminimum possibledelay. Thus, themigration plan is omposedofasequen eofsteps,exe utedsequentially,wheretherststep on-sistsofallofthemigrationsthatareinitiallyfeasible,andea hsubsequentstep onsistsofallofthemigrationsthat havebeenmadefeasiblebythepre eding steps. As an example, Figure 6 shows a re onguration graph that has been extendedwithamigrationof

VM

5

rsttonode

N

2

andthentonode

N

3

tobreak a y le ofinfeasible migrations. From this re ongurationgraph, we obtaina three-stepre onguration plan. Therststepmigrates

VM

1

,

VM

3

,

VM

4

and

VM

5

(to the pivot

N

2

). Then the se ond step migrates

VM

2

and

VM

7

. Finally, the thirdstepmigrates

VM

5

toitsnaldestination.

N

2

C

m

=512,

C

p

=1

N

4

C

m

=512,

C

p

=1

N

5

C

m

=768,

C

p

=0

N

3

C

m

=512,

C

p

=0

N

1

C

m

=640,

C

p

=0 I.

VM

4

R

m

=256,

I

I.

VM

3

R

m

=256,

I

III.

VM

5

R

m

=256,

A

II.

VM

7

R

m

=384,

A

II.

VM

2

R

m

=512,

A

I.

VM

1

R

m

=256,

I

I.

VM

5

4.2 Estimating the ost of a re onguration plan

The ost of performing a re onguration in ludes boththe overhead in urred by themigrations themselvesand thedegradationin performan e thato urs whenmultiple a tiveVMsshareapro essingunit,aso urswhenamigration is delayeddue to sequentialor y li onstraints. Thelatter isdetermined by thedurationofpre edingmigrations. Inthisse tion,werstmeasurethe ost anddurationofasinglemigration,andthenproposea ostmodelfor omparing the ostsofpossiblere ongurationplans.

Migration ost Migrating a VM from one node to another requires some CPUandmemorybandwidthonboththesour eanddestinationnodes. When there is an a tive VM on either the sour e or destination node, it will have redu eda esstotheseresour es,andthuswilltakelongerto ompleteitstask. Inthisse tion,weexaminethese ostsinthe ontextofahomogeneous luster. Figure7showstheset ofpossible ontextsin whi hamigration ano ur, depending on the state of the ae ted VMs, in the ase where ea h node is a unipro essor. Be ause a migration only has an impa t on the a tive and migrated VMs, we ignore the presen e of ina tive, non-migrated VMs in this analysis. Anina tiveVM anmovefromanina tivenodetoanodehostingan a tiveVM(Ina tiveToA tive,orITA),from anodehostingana tiveVMto anina tivenode(Ina tiveFromA tive, orIFA), orfrom onenodehostingan a tiveVMtoanother(Ina tiveFromA tiveToA tive,orIFATA).Similarly,an a tiveVM anmovetoanina tivenode(A tiveToIna tive,orATI)ortoan a tivenode(A tiveToA tive,orATA),althoughthelatterisneverinteresting inaunipro essorsettingasaunipro essornodeshouldnothostmultiplea tive VMsatonetime.

(a)ITA (b)IFA ( ) IFATA

(d)ATI (e)ATA

Figure7: Dierent ontextsforamigration.

VM

2

isa tive

Inordertoevaluatetheimpa tofamigrationforea h ontext,wemeasure both the duration of the migration and the performan e losson a tive VMs. Tests areperformedon twoidenti alnodes,ea h with asingleAMD Opteron 2.4GHzCPUand4GbofRAMinter onne tedthrougha1Gblink. Weusethree

0

5

10

15

20

25

512

1024

1536

2048

Migration time in sec.

Memory used by the migrated VM, in MB

IFATA

ITA

ATI

IFA

Figure8: DurationofVMmigration

0

5

10

15

20

25

30

512

1024

1536

2048

Migration overhead in sec.

Memory used by the migrated VM, in MB

IFATA

ITA

ATI

IFA

Figure9: Impa t ofmigration onVMperforman e

VMs:

VM

1

,whi his ina tive,and

VM

2

and

VM

3

, whi hare a tiveand exe utea BT.WtaskembeddedinaNASGRIDEDben hmark[6℄. TheVMsarepla ed onthe nodesa ordingto the IFATA, ITA, ATI, and IFA ongurations. We varytheamountofmemoryallo atedtothemigratedVMfrom512to2048MB. Figure 8shows theaverage durationof the migration in termsof the amount of memory allo ated to the migrated VM. Figure 9showsthe in reaseof the durationoftheben hmarkduetothemigrationofaVMusingagivenamount ofmemory.

Weobserverstthat theduration ofthemigration mostlydependsonthe amount of memory used by the migrated VM. Se ond, the performan e loss variessigni antlya ordingto the ontextof themigration. Forthe ontext IFA, theonlyoverhead omes from readingthememory pagesonnode

N

1

, as writingthepagesontheina tivenode

N

2

doesnothaveanyimpa tonana tive VM.Forthe ontextATI, itis thea tiveVMthat migrates;in this situation, themigrationisalittlemoreexpensive: be auseXenusesanin remental opy-on-writeme hanismtomigratethememorypagesofaVM[5℄, multiplepasses are needed to re opy memory pages that are updated by the a tivity of the VM during the migration pro ess. The ontext ITA in urs an even higher overhead,aswritingthememorypages of

VM

1

onnode

N

2

usesupmostofthe CPUresour esonthat node,whi hare thennotavailable to

VM

2

. Finally,the

ontext IFATA in urs the highestoverheadas themigrations a ton boththe sour eandthedestinationnode. Thisoverheadis omparabletothesumofthe overheadof ontextsIFA andITA.

Thisevaluationofthe ostofmigrationsshowsthatmigratingaVMhasan impa t on both thesour e and destination nodes. The migration redu es the performan e of o-hosteda tivevirtual ma hines for aduration that depends on the ontextof the migration. Inthe worst ase, the performan e lossof a omputationaltaskisaboutthesameasthedurationofthemigration. Although theoverhead anbeheavyduringthemigrationtime,themigrationtimeisfairly short,andthushaslittleimpa tontheoverallperforman e. Nevertheless,these numberssuggestthatthenumberofmigrationsshould bekeptto aminimum. Migration ostmodel Figures8and9showthat theoverheadfor asingle migrationandthedelayin urredforpre edingmigrationsbothvaryprin ipally in termsof the amount of memoryallo ated to themigrated VMs. Thus, we base the ostmodelonthisquantity.

The ost fun tion

f

is dened as follows. The estimated ost

f

(p)

of a re ongurationplan

p

isthesumofthe ostsofthemigrationsofea hmigrated VM

v

(Equation6). Theestimated ost

f

(v)

ofthemigrationofaVM

v

isthe sumoftheestimated ostsofthepre edingsteps,plustheamountofmemory allo ated to

v

(Equation 7). Finally, the estimated ost

f

(s)

of a step

s

is equalto the largestamountof memoryallo atedto anyVM that is migrated in step

s

. This estimated ost onservativelyassumes that one step anonly beginwhenallof themigrationsof thepreviousstephave ompleted. Forthe re ongurationplanshowninFigure6,theestimated ostofstepIIis512,the estimated ost of the migration of

VM

2

is 768, and the estimated ost of the wholere ongurationplan is

4224

.

f

(p) =

X

v∈p

f

(v)

(6)

f

(v) = R

m

(v) +

X

s∈

prevs

(v)

f

(s)

(7)

f

(s) =

max

(R

m

(v)), v ∈ s

(8)

4.3 Implementing and optimizingthe VMRP

ToexpresstheVMRPasaCSP,weagainusethe onstraintsthata onguration must be viable, asdes ribed in Se tion 3.2, and additionally spe ify that the numberofnodesusedin a ongurationisequaltothesolutionofthe VMPP (Equation 9):

X

i∈N

u

i

= x

vmpp

(9)

For ea h ongurationthat satises these onstraints, the solver onstru tsa re ongurationplan

p

,ifpossible. Theoptimalsolution

k

istheone that mini-mizesthevariable

K

,denedasfollows(Equation 10):

K

= f (p)

(10)

Minimizingthe ostofare ongurationprovidesaplanwithfewermigrations and steps, and a maximum degree of parallelism, thus redu ing the duration andtheimpa tofare onguration.

Thelowerboundfor

K

isthesumofthe ostofmigratingea hVMthatmust migratei.e. whenmultiplea tiveVMsarehostedonthesamenode. Theupper bound orrespondstothe ostofthere ongurationplan

p

vmpp

asso iatedwith the ongurationpreviously omputedbyVMPP:

(

X

v∈V

migrate

R

m

(v)) ≤ K ≤ f (p

vmpp

)

(11)

Like theVMPP, the VMRP usesequivalen es to redu e the time required tondviable ongurations. FortheVMRP, however,theequivalen erelation betweenVMshastobemorerestri tivetotakeintoa ounttheimpa toftheir migration. Indeed, migration of equivalent VMs must havethe same impa t on the re onguration pro ess. Thus, equivalent VMs must have the same resour edemandsandmustbehostedonthesamenodes. Inthissituation,the equivalen erelationbetweentwoVMsisformalizedbyEquation12.

∃v

i

, v

j

∈ V | v

i

≡ v

j

⇔ R

p

(i) = R

p

(j) ∧

R

m

(i) = R

m

(j) ∧

host(v

i

) = host(v

j

)

(12)

Entropydynami allyestimatesthe ostoftheplanasso iatedwiththe on-guration being onstru ted based on information about the VMs that have alreadybeenassignedtoanode. Then,Entropyestimatesaminimum ostfor the omplete future re onguration plan. Forea hVM that hasnotyet been assignedtoanode,thesolverlooksatVMsthat annotbehostedbytheir ur-rentnodeandin reasesthe ostwiththesefuturemigrations. Finally,thesolver determines whether the future onguration based on this partial assignment mightimprovethesolutionorwillne essarilybeworse. Inthelattersituation, thesolverabandonsthe onguration urrentlybeing onstru tedandsear hes foranotherassignment.

5 Evaluations

Entropyuses onstraintprogrammingin order tond abetterre onguration plan than that found using lo ally optimal heuristi s. Nevertheless, the more exhaustive sear h performed by onstraint programmingis only justied if it leadstoabettersolutionwithin areasonableamountof time. Inthisse tion, we rst evaluate the two phases of the re onguration algorithm of Entropy onsimulationdata,toillustrate therangeofbenetthat Entropy anprovide. Wethen useEntropy ona luster in theGrid'5000 experimental testbedon a olle tionofprogramsfromtheNASGridben hmarksuite[6℄.

5.1 Evaluation of the VMPP and VMRP

TheVMPPin ludesthenumberofnodesin the ongurationidentiedbythe FFDheuristi asaninitialupperbound,andthusneitheritssolutionnorthat oftheVMRPwilleverusemorenodesthantheFFDsolution. Inthisse tion, wemeasurethetimerequiredforour onstraint-basedre ongurationengineto signi antlyredu eboththenumberofnodesandthe ostofthere onguration plan,as omparedtothesolutionproposedbytheFFDheuristi ,onarangeof simulateddata. Wehaveusedtheseresultsasthebasisofthetimeouts hosen in Entropy, as des ribed in Se tion 2. In our evaluation, we onsider solving the VMPP and the VMRP using either FFD or Entropy. The FFD solution to theVMPP is thenumberof nodesin the onguration hosenbythe FFD heuristi , and the FFD solution to the VMRP is the minimal re onguration planthat produ esthis onguration.

We onsidertwo lassesofproblemsizes,ea husing64or128nodesandan equalnumberofVMs. Forea h lass,wehaverandomlygenerated100 ong-urations withthe followingproperties: Ea h VM needszerooronepro essing units, depending on its state, and 1 or 2 GB of memory. Nodes ea h have onepro essingunitand3GBofmemory. Thesame ongurationsareusedfor evaluatingthesolutionsof boththe FFDandEntropyimplementations ofthe VMPP and theVMRP. Thededi ated nodethat exe utesthe re onguration algorithmhasanAMDOpteron2.0GHzCPUand2GBofRAM.The re ong-urationalgorithm is implementedin Javaandruns onthe standardSun Java 1.5virtualma hine.

0

20

40

60

80

100

0

10

20

30

40

50

60

% of minimized configurations

Computation time in sec.

64VMs,64Ns

128VMs,128Ns

(a)Minimizationof

X

0

20

40

60

80

100

64/64

128/128

% of total

VMS/nodes

Equiv.

1 nodes

2 nodes

3 nodes

4 nodes

5 nodes

(b) Improvement wrt. FFD

Figure10: PropertiesofthesolutionoftheVMPPforvariousproblemsizes

Evaluation of the VMPP Figure 10(a)showstheper entageof problems in ea h lass for whi h the minimum number of nodes has been determined withinthegivenamountoftime. The omputationtimeforsolvingtheVMPP is prin ipally determined by the total numberof VMs and nodes and by the numberofequivalen e lasses,asidentiedin Se tion3.3. Forthetwo lasses, thesolverneedsfewerthan5se ondsto omputetheminimumnumberofnodes for90%ofthe ongurations.

Asshown inFigure 10(b),Entropyndsabetterpa kingbyupto 5fewer nodesfor47%ofthe ongurations.Contrarytotheheuristi thatstopafterthe rst ompleteassignmentof theVMs, Entropy ontinuesto omputeabetter solutionuntilittimesoutorprovestheoptimalityofthe urrentone.

Evaluation of the VMRP Figure 11(a)showstheprogressionin ndinga ongurationwith minimum ost,

K

. Be auseofthehigh ostof reatingand evaluating the re onguration plans, the solver is never able to prove that a ongurationhasthesmallestre onguration planin thetime allotted. Thus, we onsider asolutionto be minimal until onewith a 10%lower re ongura-tion ost is omputed.

1

The graphdenotes the per entageof solutionswhere there onguration ostasso iatedwiththe omputed ongurationisminimal, over time. Thene essary time for omputing a onguration with aminimal re onguration ostisprin ipallydeterminedbythenumberofVMsandnodes. After10se onds,90%ofthe ongurationswith64nodesareminimal. Cong-urationswith128nodesrequirea omputationtimeof20se onds.

Figure 11(b) shows the ee tiveness of the redu tion of

K

by omparing there onguration ostof theoriginal solution omputed by Entropy forthe VMPPwith the ostof thenal onguration. Thesolution produ edforthe VMRPusesthesamenumberofnodesasthesolutionprodu edfortheVMPP but has a re onguration ost that is up to 40%lower. Entropy redu es the re onguration ostfor93%ofthe ongurations.

0

20

40

60

80

100

0

10

20

30

40

50

60

% of minimized cost

Computation time in sec.

64VMs,64Ns

128VMs,128Ns

(a)Minimizationof

K

0

20

40

60

80

100

64/64

128/128

% of total

VMs/nodes

Equiv.

1-10%

11-20%

21-30%

31-40%

(b) Improvement wrt. VMPP

Figure11: PropertiesofthesolutionoftheVMRPforvariousproblemsizes

5.2 Experiments on a luster

We now apply Entropy on a real luster omposed of 39 nodes, ea h with a AMDOpteron 2.0GHzCPUand 2GBofRAM. Onenodeisdedi ated tothe re ongurationengineandthreenodesareusedasleserversthatprovidethe diskimagesfortheVMs. Theremaining35nodesruntheXenVirtualMa hine Monitorwith200MBofRAMdedi atedtoXen'sDomain-0. Thesenodeshosta totalof35VMsthatrunben hmarksoftheNASGridben hmarksuite[6℄. This ben hmarksuiteisa olle tionofsyntheti distributedappli ationsdesignedto ratetheperforman eandfun tionalitiesof omputationgrids. Ea hben hmark is organized as a graph of tasks where ea h task orresponds to a s ienti omputation that is exe uted on a single VM. Edges in the graph represent thetaskordering. Thisorderingimpliesthat thenumberof a tiveVMsvaries duringtheexperiment;thereare typi allyfrom10 to15a tiveVMs. Entropy, however,isunawareof thesetaskgraphs,insteadrelyingontheinstantaneous

1

Weusethethresholdof10%inthisguretoa ountforthefa tthatthere onguration ostfun tiononlyprovidesanestimate.

(a)ED (b)HC ( ) VP

Figure12: Computation graphsofNASGridBen hmarks

00:00

02:00

04:00

06:00

08:00

10:00

12:00

14:00

0

50000 100000 150000 200000 250000 300000 350000 400000

Reconfiguration duration in min.

Reconfiguration cost

FFD (9 reconfigurations)

Entropy (18 reconfigurations)

Figure13: Re ongurationplans omputedbyFFDandEntropy

des riptions provided by the sensors to determine whi h VMs are a tive and ina tive.

The35VMsareassignedto thevarious tasksoftheNASGridben hmarks ED, HC, and VP, whose omputation graphsare shown in Figure 12. Ea h set of VMs asso iated with a given ben hmark has its own NFS le server that ontains the VMs' disk image. The ED ben hmark uses 10 VMs with 512 MB of RAM ea h. It has one phase of omputation that on erns all of its VMs. The HC ben hmark uses5 VMs with 764 MB of RAM ea h. This ben hmark isfully sequentialand hasonly onea tive taskat atime. Finally, theVP ben hmarkuses 20VMs,with 512MBofRAM ea h. This ben hmark hasseveralphaseswherethenumberofa tiveVMsvaries. Beforestartingthe experiment,ea hVMisstarted inanina tivestate,in aninitial onguration omputedusing Entropy. This ongurationuses13nodesand orrespondsto amaximum pa king. All three ben hmarksare startedat thesametime. We testtheben hmarksusingFFDandEntropyasthere ongurationalgorithm. Figure13showstheestimated ostofea hre ongurationplansele tedusing FFDandEntropyand thedurationof itsexe ution. Therelationshipbetween the ost and the exe ution time is roughly linear, and thus the ost fun tion

f

is a reasonableindi ator of performan e for plans reated using both FFD and Entropy. Furthermore,we observethat re ongurationbasedon Entropy planstypi ally ompletesmu hfasterthanre ongurationbasedonFFDplans. Indeed,theaverageexe utiontimeforplans omputedwithFFDis about413

se onds whilethe averageexe ution time for plans omputedwith Entropyis only107se onds. With shortre onguration plans,Entropyis ableto qui kly rea ttothefrequent hangesinthea tivityofVMs,andthusqui klydete tsand orre tsnon-viable ongurations. Entropyperforms 18shortre ongurations overtheduration oftheexperiment, whiletheFFD-basedalgorithm performs 9longerones.

Figures14(a)and14(b)showthea tivityofVMswhilerunningthe ben h-markswithFFD andEntropy,in terms ofthenumberof a tiveVMsthat are satisedandunsatised. SatisedVMsarea tiveVMsthathavetheirown pro- essingunit. UnsatisedVMsarea tiveVMsthatshareapro essingunit. The averagenumberofunsatised VMsis1.75forFFDand1.05forEntropy. The numberofunsatisedVMsisasigni ant riteriontoratethebenetofa re on-guration algorithm. An unsatised VM indi ates anon-viable onguration, andthus aperforman eloss.

0

5

10

15

20

00:00

00:10

00:20

00:30

00:40

00:50

01:00

01:10

01:20

01:30

01:40

Active VMs

Time (hours)

Satisfied VMs

Unsatisfied VMs

(a)FFD

0

5

10

15

20

00:00

00:10

00:20

00:30

00:40

00:50

01:00

01:10

01:20

01:30

01:40

Active VMs

Time (hours)

Satisfied VMs

Unsatisfied VMs

(b)Entropy Figure14: A tivityofVMs

Whentheben hmarksstart,12VMs be omea tiveatthesametime. En-tropy qui kly remaps the VMs and obtainsa viable onguration by minute 6. FFD, onthe other hand, doesnot rea h aviable ongurationuntil mu h later. Thetotalnumberof a tiveVMs in reasesat minute 10,thusin reasing thenumberofunsatisedVMs. AsEntropyisnotinare ongurationstateat thattime, it omputesanew ongurationandmigratestheVMsa ordingly, to obtainaviable ongurationby minute 11. FFD, onthe otherhand, is in themidstofmigratingVMsat thepointoftherstpeakofa tivity,a ording to apreviously omputed,and now outdated, re ongurationplan. FFDonly rea hesaviable ongurationinminute18. Inthissituation,we onsider that aniteration ofthe re onguration pro ess using FFDtakestoomu h time as omparedtothea tivityoftheVMs.

Theaverageresponsetimeofare ongurationpro essmeasurestheaverage durationbetweendete tingthepresen eofunsatisedVMsandthenextviable onguration. It indi ates the apa ity ofthe re onguration pro essto s ale with thea tivity of VMs. Forthis experiment, the average response time for FFDis248se onds. ForEntropy,theaverageresponsetimeis142se onds.

Figure 14(b) shows that number of unsatised VMs is always zero after 1:00. This is due to the unequal duration of theben hmarks. At minute 50, theben hmarkHC endsits omputation. Then thea tivity ofVP hangesat minutes54and58andrequiresare onguration. Fortheremainingtime,there isnonewphasethat makesunsatisedVMs: Theend ofthelast phaseofVP

at 1:10doesnotrequireare onguration and thea tivityof thelast running ben hmark,ED,is onstant.

10

12

14

16

18

20

00:00

00:10

00:20

00:30

00:40

00:50

01:00

01:10

01:20

01:30

01:40

Used nodes

Time (hours)

FFD

Entropy

Figure15: NumberofnodesusedwithFFDandEntropy

Figure 15 shows the numberof nodes used to host VMs. Re onguration plans omputed with FFD require more migrations and thus tend to require more pivot nodes. Forthis experiment, the re onguration pro ess based on FFD requires up to 4 additional pivot nodes. This situation is parti ularly unfortunatewhen onsolidationisusedtosaveenergy,bypoweringdownunused nodes,asnodeshavetobeturnedonjusttoperformsomemigrations. Entropy, whi h reates smallerplans, requires at mostone additionalpivot nodes, and thusprovidesaenvironmentfavorabletotheshuttingdownofunusednodes.

0

10

20

30

40

50

60

70

80

90

100

ED

VP

HC

Runtime in min.

FFD

Entropy

wo. consolidation

Figure16: RuntimeComparison

Byminimizing theduration of non-viable ongurations, Entropyredu es theperforman elossdueto onsolidation. Figure16showstheruntimeofea h ben hmarkforFFD,Entropyandforanenvironmentwithoutany onsolidation. Inthelattersituation,ea hVMisdenitivelyassignedtoitsownnodetoavoid performan elossduetothesharingofpro essingunits. Inthis ontext,35nodes arerequired. Theglobaloverheadforallben hmarks omparedtoaexe ution without onsolidationis19.2%forFFD.Entropyredu esthisoverheadto11.5%. We an summarizethe resour eusage ofthe various ben hmarks in terms ofthenumberofnodesusedperhour. Withoutany onsolidation,runningthe ben hmarks onsumes53.01nodesperhour. ConsolidationusingFFDredu es this onsumptionto24.53nodesperhour. ConsolidationusingEntropyfurther redu esthis onsumptionto23.21nodesperhour. However,thesenumbersare ae tedbythedurationofea hben hmark. Whenallben hmarksarerunning, the onsolidationonly omes fromthere ongurationenginethat dynami ally mixesina tiveVMswitha tiveVMsinthedierentphasesoftheappli ations. When a ben hmark stops, it reates zombie VMs that still require memory

resour esbut should be turned o. Thus, to estimate the onsumption that onlyresultsfrom mixing ina tiveand a tivenon-zombieVMs,we onsiderthe onsumption until the end of the rst ben hmark to omplete, HC. In this situation,runningthethreeben hmarkswithout onsolidation onsumes24.31 nodesperhour, with FFD onsumes15.34 nodes per hour, andwith Entropy onsumesonly11.72nodesperhour.

6 Related work

Power-Aware VM repla ement Nathujiet al.[12℄ presentpowere ient me hanismsto ontroland oordinatetheee tsofvariouspowermanagement poli ies. This in ludes thepa kingofVMs throughlivemigration. Theylater extendedtheirwork tofo usonthetradeo betweentheServi e Level Agree-ments of the appli ations embedded in the VMs and the ne essity to satisfy hardware power onstraints[13℄. Entropy addressesthe re ongurationissues brought by the livemigration of VMs in a luster and provides asolution to pa kVMsintermsoftheirrequirementsforpro essingunitsandmemory,while minimizing thedurationof the re ongurationpro ess and itsimpa t on per-forman e.

Vermaet al. [17℄ proposean algorithm that pa ks VMsa ordingto their CPU needswhile minimizing the numberof migrations. This algorithm is an extensionoftheFFDheuristi and migratesVMslo atedonoverloadednodes tounder-exploitednodes. Restri tingmigrationstoonlythosefromoverloaded nodes to underloaded nodes hasthe ee t that allsele ted migrations are di-re tlyfeasible; the sequentialand y li onstraintsthat wehaveidentied in Se tion 4 annotarise. Nevertheless, thisimplies that theapproa h maymiss opportunitiesforsavings,in aseswhererearrangingtheVMswithinthe under-loaded nodeswould enableother, even morebene ial migrations. In this sit-uation,thisapproa hfails,potentiallyviolatinganyServi eLevelAgreements, evenifthereisapossiblesolution. EntropyexploitsalargersetofpossibleVM migrationsbyaddressingsequentialand y li onstraints,andthus anbeused tosolvethemore omplexre ongurationproblemsthat ano urinahighly loadedenvironment.

Performan eManagementthroughrepla ement Khannaetal.[11℄ pro-pose are onguration algorithm that assigns ea h VM to a node in order to minimizetheunusedportionofresour es. VMswithhighresour erequirements are migrated rst. Bobroet al. [3℄ base their repla ement engineon a fore- astservi ethat predi ts,for thenextfore astinterval,theresour edemands of VMs, a ording to their history. Then the repla ement algorithm, whi h is based on an FFD heuristi , sele ts a node than an host the VM during thistimeinterval. Toensuree ien y,thefore astwindowtakesintoa ount thedurationofthere ongurationpro ess. However,thisassignmentdoesnot onsider sequentialand y li onstraints, whi h impa t the feasibility of the re ongurationpro essanditsduration.

VMs repla ement issues Grit et al. [7℄ onsider some VMs repla ement issuesforresour emanagementpoli iesinthe ontextofShirako[9℄,asystemfor on-demandleasingofsharednetworkedresour esinfederated lusters. Whena

migrationisnotdire tlyfeasible,duetosequen eissues,theVMispausedusing suspend-to-disk. On e thedestination nodeisavailableformigration, theVM isresumedonit. Entropyonlyuseslivemigrationsin ordertopreventfailures in theuserenvironmentduetosuspendingpartofadistributed appli ation.

Sandpiper [18℄ is a re onguration engine, based on an FFD heuristi , to relo ateVMsfrom overloadedto under-utilized nodes. Whena migration be-tween two nodes is not dire tly feasible, the system identies a set of VMs to swap in order to free a su ient amount of resour es on the destination node. Then the sequen eof migrations is exe uted. This approa h is ableto solvesimplerepla ementissuesbutrequiressomespa efortemporarilyhosting VMs oneither thesour e orthedestination node. Byidentifying pivot nodes andbypassmigrations,Entropy anresolve y leswithoutperformingmultiple swap operations that in rease the number of migrations thus the duration of there ongurationpro ess.

7 Con lusion and Future Work

Previousworkhasreje tedtheuseof onstraintsinimplementing onsolidation as being too expensive. In this paper, we have shown that the overhead of onsolidationisdeterminednotonlythetimerequiredto hooseanew ong-uration, but also by thetime required to migrate VMs to that onguration. Our onstraint-programming based approa h, whi h expli itly takes into a - ountthe ost of the migration plan, an indeed redu e thenumber of nodes and themigrationtime signi antly,as omparedto resultsobtainedwiththe previouslyusedFFDheuristi . Wehaveimplementedthisapproa hinour on-solidation managerEntropy, andshownthat it an redu ethe onsumption of lusternodesperhourfora olle tionofNASGridben hmarksbyover50%as omparedtostati allo ationandbyalmost25%as omparedto onsolidation usingFFD.

The ongurations onsideredinthispaperarefairlysimple,be auseinthe lustersavailable intheGrid'5000experimentaltestbed,everynodehasonlya singlepro essorandallnodeshavethesameamountofmemory. Ourapproa h, however,is dire tlyappli ableto lustersprovidingmultipro essorsand nodes with non-homogeneousmemoryavailability, be ausethe numberofpro essors and theamountofmemoryavailablearesimplyparametersoftheVMPP and VMRPproblems. Wewillextendourresultstosu h lusterswhentheybe ome available tous.

In future work, we plan to onsider the problem of admission ontrol for lusters providing onsolidation. We expe t that simulationresults, like those des ribedinSe tion5.1, anhelptoidentifythenumberoftasksthata luster providing onsolidation ana ept. Wealsoplan to onsider theappli ability oftheapproa htootherkindsofsoftwarethans ienti omputations,su has e- ommer e.

A knowledgments

Experimentspresentedinthispaperwere arriedoutusingtheGrid'5000 exper-imental testbed[4℄,aninitiativefromtheFren hMinistryofResear hthrough

theACIGRIDin entivea tion,INRIA,CNRSandRENATERandother on-tributingpartners.

Availability

TheprototypeEntropyisavailableonourwebpage: http://www.emn.fr/x-info/entropy/

Referen es

[1℄ P. Barham,B.Dragovi ,K.Fraser,S. Hand, T.Harris,A. Ho,R. Neuge-bauer, I. Pratt, and A. Wareld. Xen and the art of virtualization. In Pro eedings ofthe19thACMSymposiumonOperatingSystemsPrin iples, pages164177,BoltonLanding,NY,USA,O t.2003.ACM Press. [2℄ F.Benhamou,N.Jussien,andB.O'Sullivan,editors. TrendsinConstraint

Programming. ISTE,London,UK,May2007.

[3℄ N. Bobro, A.Ko hut, andK. Beaty. Dynami pla ement ofvirtual ma- hines for managing SLA violations. Integrated Network Management, 2007. IM '07. 10th IFIP/IEEE International Symposium on, pages 119 128,May2007.

[4℄ R.Bolze,F.Cappello,E.Caron,M.Daydé,F.Desprez,E.Jeannot,Y. Jé-gou, S. Lantéri, J. Ledu , N. Melab, G. Mornet, R. Namyst, P. Primet, B.Quetier, O.Ri hard,E.-G. Talbi, andT.Iréa. Grid'5000: alarges ale andhighlyre ongurableexperimentalgridtestbed.International Journal of High Performan e ComputingAppli ations, 20(4):481494,Nov.2006. [5℄ C. Clark,K. Fraser,S. Hand, J.G.Hansen, E.Jul, C.Limpa h, I.Pratt,

andA.Wareld. Livemigrationofvirtualma hines. InPro eedingsof the 2ndACM/USENIX SymposiumonNetworkedSystemsDesign and Imple-mentation(NSDI'05), pages273286,Boston,MA,USA,May2005. [6℄ M. Frumkin andR. F.V. derWijngaart. NAS grid ben hmarks: A tool

forgridspa eexploration. ClusterComputing, 5(3):247255,2002. [7℄ L.Grit,D.Irwin,A.Yumerefendi,and J.Chase. Virtualma hinehosting

fornetworked lusters: Buildingthefoundationsfor"autonomi " or hestra-tion. InVirtualizationTe hnologyinDistributedComputing,2006.VTDC 2006. FirstInternational Workshopon,pages18,Nov.2006.

[8℄ R. Harali kandG.Elliott. In reasingtreesear he ien y for onstraint satisfa tionproblems. Arti ialIntelligen e,14(3):263313,O tober1980. [9℄ D. Irwin,J.Chase,L.Grit,A.Yumerefendi,D.Be ker,andK.G.Yo um. Sharingnetworkedresour eswithbrokeredleases. InATEC '06: Pro eed-ingsoftheannual onferen eonUSENIX'06AnnualTe hni alConferen e, pages1818,Berkeley,CA, USA,2006.USENIX Asso iation.

[10℄ N. Jussien, G.Ro hart,and X. Lor a. TheCHOCO onstraint program-mingsolver.InCPAIOR'08workshoponOpen-Sour eSoftwarefor Integer andContraintProgramming(OSSICP'08),Paris,Fran e,June2008. [11℄ G. Khanna, K.Beaty, G. Kar, and A. Ko hut. Appli ation performan e

managementin virtualized serverenvironments. Network Operations and ManagementSymposium,2006.NOMS2006.10thIEEE/IFIP,pages373 381,2006.

[12℄ R.NathujiandK.S hwan.VirtualPower: Coordinatedpowermanagement invirtualizaedentreprisesystems.In21stSymposiumonOperatingSystems Prin iples (SOSP),O t.2007.

[13℄ R. Nathuji and K. S hwan. VPM tokens: virtual ma hine-aware power budgetingin data enters. InHPDC '08: Pro eedings of the 17th interna-tional symposium on High performan edistributed omputing, pages 119 128,NewYork,NY, USA,2008.ACM.

[14℄ F.Rossi,P.vanBeek,andT.Walsh.HandbookofConstraintProgramming (Foundations of Arti ial Intelligen e). Elsevier S ien e In ., NewYork, NY,USA,2006.

[15℄ P.Shaw. A onstraintforbinpa king. InPrin iples andPra ti eof Con-straintProgramming(CP'04), volume3258 ofLe tureNotes inComputer S ien e,pages648662.Springer,2004.

[16℄ M. Tri k. A dynami programmingapproa h for onsisten yand propa-gationforknapsa k onstraints. InPro eedings of the ThirdInternational WorkshoponIntegrationofAIandORTe hniquesinConstraint Program-mingfor CombinatorialOptimizationProblems (CPAIOR-01),pages113 124,2001.

[17℄ A. Verma, P. Ahuja, and A. Neogi. Power-aware dynami pla ement of HPCappli ations. InP.Zhou,editor,ICS,pages175184.ACM,2008. [18℄ T. Wood, P. J. Shenoy, A. Venkataramani, and M. S. Yousif. Bla k-box

IRISA, Campus universitaire de Beaulieu - 35042 Rennes Cedex (France)

Unité de recherche INRIA Futurs : Parc Club Orsay Université - ZAC des Vignes

4, rue Jacques Monod - 91893 ORSAY Cedex (France)

Unité de recherche INRIA Lorraine : LORIA, Technopôle de Nancy-Brabois - Campus scientifique

615, rue du Jardin Botanique - BP 101 - 54602 Villers-lès-Nancy Cedex (France)

Unité de recherche INRIA Rhône-Alpes : 655, avenue de l’Europe - 38334 Montbonnot Saint-Ismier (France)

Unité de recherche INRIA Rocquencourt : Domaine de Voluceau - Rocquencourt - BP 105 - 78153 Le Chesnay Cedex (France)

Unité de recherche INRIA Sophia Antipolis : 2004, route des Lucioles - BP 93 - 06902 Sophia Antipolis Cedex (France)

Éditeur

INRIA - Domaine de Voluceau - Rocquencourt, BP 105 - 78153 Le Chesnay Cedex (France)

http://www.inria.fr