HAL Id: hal-00560819
https://hal.archives-ouvertes.fr/hal-00560819
Submitted on 30 Jan 2011
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of
sci-entific research documents, whether they are
pub-lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Selection Criteria
Jacques Julliand, Pierre-Alain Masson, Régis Tissot, Pierre-Christophe Bué
To cite this version:
Jacques Julliand, Pierre-Alain Masson, Régis Tissot, Pierre-Christophe Bué. Generating Tests from
B Specifications and Dynamic Selection Criteria. FAC, Formal Aspects of Computing, 2011, pp.3–19.
�hal-00560819�
Generating Tests from B
Spe i ations and Dynami Sele tion
Criteria 1
J. Julliand,P.-A.Masson, R. Tissot and P.-C. Bue LIFC,UniversitedeFran he-Comte
16,routedeGrayF-25030Besan onCedexFran e fjulliand,masson, tissot, bueglif .univ-f omte. fr
Abstra t. This paper is about generating tests from dynami sele tion riteria alled test purposes, in addition to stru tural tests, obtained from stati sele tion riteria. We present a method that re-uses a behavioral model and an abstra t test on retizationlayer developed for stru tural testing, and relies on additionaltestpurposes.Wepropose,intheBframework,apro essoftestgenerationthatusesthesymboli animationme hanismsofLTG(LeiriosTestGenerator)basedon onstraintsolving,andguidedbythetest purposes. Webuild forthat aBmodelthat is thesyn hronizedprodu tof abehavioralBabstra t model andatestpurpose des ribedasalabelledtransitionsystem.Weprovethe orre tnessofthis method, and show someexperimental resultsobtained onthe IAS ase study. IASis an industrial smart- ardplatform dedi atedtotheoperationsofIdenti ation,Authenti ationandele troni Signature.Ourexperimentsshow thatthetestsobtainedfromtestpurposesare omplementarytothestru turaltests.
Keywords:Model-BasedTesting, Test Purpose,IASCaseStudy.
1. Introdu tion
Bmodelsarewellsuitedforprodu ingfun tionaltestsofanimplementationbymeansofamodel-basedtesting approa h [BJK
+
05, UL06℄. This approa h,as isdes ribed in Se . 5and illustratedbyFig. 8,pro eeds by writingaformalbehavioralmodel (M)oftheexpe tedfun tionalitiesofasystem.Thismodelisanabstra tion ofanyrealimplementation,andissupposedtoprovideareliableviewoftheimplementationundertest(IUT). Byapplyingsele tion riteria,atestgeneration tool anautomati allyextra t testsfromthemodel.These tests are parti ular \exe utions"of the model. They are sequen esof operation alls, with valuesof their parametersandtheirresultsaspredi tedbythemodel.Thetestsareabstra tsin etheyhavethesamelevel of abstra tionasthe model. Theyare on retizedby a on retization layer (CL) to be ome exe utableon theIUT.ComparingtheresultsreturnedbytheIUTwiththeonespredi tedbythemodelallowsdelivering averdi tofthetests.
Stru turaltesting uses stati (synta ti ) sele tion riteria, essentially providing ontrol ow and data
1
Resear h partiallyfundedbythe Fren h NationalResear h Agen yANR (POSE ANR-05-RNTL-01001) and the Region Fran he-Comte.
overageofthemodel.Thetestsexer isethefun tionalitiesofthesystembydire tlya tivatingand overing the orresponding operations. Industrialstudies haveproven the eÆ ien y of the method to dete tfaults in animplementation(seeforexample[EFHP02,BLLP04℄). WritingM andCLisanimportante ort,but the ostisjusti ed bythepossibilitytoautomati ally omputeagreatnumberofsmarttest ases,withM asanora le.Nevertheless, stati sele tion riteriaappear to beinsuÆ ientto exer isetheIUTin tortuous situations.Wethinkforexampleofsomes enariosofatta kofsystemsrequiringstrongse urityguarantees. Ourobje tiveisto bene tfromM andCLto omputesomeadditionalteststhat useaparti ulars enario asasele tion riterion.
Thes enario anbedes ribedbymeansofatestpurpose(TP),whi hwe onsiderasadynami (semanti ) sele tion riteriathat or hestratesthesu essive alls of the operationsof the model. The testsextra ted fromthemodelbymeansofatest purposearesequen esofoperation alls orrespondingtothes enario.
The ontext of this work is the test generation from B models 2
. We use LTG (Leirios Test Genera-tor)[JL07℄,thetestgeneratorfrom Smartesting
3
,toautomati allyextra tabstra ttestsfrom abehavioral model written in B. LTGuses a onstraint solverfor omputing the tests. LTG produ esstru tural tests byapplyingstati riteriato overallthepathsofthe ontrolstru tureofeveryoperation.Moreover,it is possibleto assistthe generationof testsby providing LTG with sequen esofoperation alls that des ribe theshapeoftheexpe tedtests. Wehavevalidated ourapproa honIAS, anindustrial standardforsmart ards.
Ourmain ontributioninthispaperistode neintheBframeworkapro essthatusesLTGforgenerating abstra t tests, but with adynami sele tion riterion, provided to LTGin theshapeof aset of sequen es ofoperations,des ribedbyaTP.Also,wehaveperformedexperimentsthat showthat thesetests arenew testsw.r.t.theonesobtainedfrom stati riteria.
We give in Se . 2 some preliminary de nitions to our work. IAS, the ase study on whi h we have experimentedourapproa h, isdes ribedinSe .3.Se tion4de nestestpurposes,andproposesalanguage dedi atedtotheirexpression.Themodel-basedtestingpro esswithstati riteriausingLTG,aswellasour pro essbasedondynami riteria,areintrodu edinSe .5.Se tion6des ribeshowto ombineabehavioral modelandatestpurposetoobtainaBmodelforthetestgeneration.Ourexperimentalresultsaregivenin Se .7.We on lude, ompareourproposition torelatedworksandexposesomefutureworksinSe .8.
2. Preliminaries
This se tiongivestheba kgroundrequiredforreading thepaper,withrespe t toBin parti ular. Wegive generalnotionsaboutBabstra tma hines.Wede nethenotionsofBtra eandBexe ution.Wealsode ne therestri tionsdueto thetargetedappli ation lassandtothe ontextoftestgeneration.
Firstintrodu edbyJ.-R.Abrial[Abr96℄,aBabstra tma hinede nesanopenspe i ationofasystem byaset ofoperations.Intuitively,anoperationhasapre onditionandmodi estheinternalstatevariables byageneralizedsubstitution.Anoperationisprovidedwithalistofparametersand anreturnresults.
We address a parti ular lass of spe i ations. Our spe i ations are defensive, i.e. we assume that an operationterminates wheneverit is invoked with well typed parameters.That meansthat we onsider environmentsthatrespe ta ontra t:theyalways alltheoperationswithwelltypedparametervalues.We also assume that any operationreturns a status word (the term is borrowed from the smart ard world) that odi esareportofitsexe ution.Thereforein theremainderofthepaper,operationsarede ned asin Def.1.
De nition1(Operation). LetS i
beasubstitution.Letsw i
beastatuswordandp i bealistofparameter names. Let T i (p i ) be a typing predi ate on p i . An operation named op i is de ned as sw i op i (p i ) = PRET i (p i )THENS i END.
Forde ning aBabstra t ma hine,weneedto remindthe readerof thenotionsof Bpredi atesand B generalizedsubstitutions.Bpredi atesonasetofvariablesx aredenoted byP(x),R (x),I(x),T(x),... In theremainderofthispaper,thepredi ateI(x)denotesaninvariantandT(p)denotesatypingpredi ateon theparametervariablesp.Whenthereisnoambiguityonx,wesimplydenotethepredi atesbyP,R ,I,... 2
Thispaperisarevisedandextendedversionofapaper[JMT08b ℄previouslypresentedattheABZ'08 onferen e. 3
WedenotebyStheBgeneralizedsubstitutionsandbyE,F,... theBexpressions.Expressionsaretypedas natural,boolean,set,fun tionorrelation.RelationsbetweentwosetsAandCaredenotedasA$C.Total andpartialfun tionsarerespe tivelydenotedasA!C andA!7 C.Apairofelementsrelatedbyarelation orafun tionisdenotedasa7! .GivenasubstitutionS andapost- onditionRweareableto omputethe weakestpre onditionP,su hthatifP issatis ed,thenR issatis edaftertheexe utionofS.Theweakest pre ondition,de ned in[Abr96℄, isdenoted by[S℄R .WedenotebyhSiRtheexpression:[S℄:R ,intuitively meaningthatifhSiRissatis ed,thena omputation ofS existsterminatingin astatesatisfyingR .Given aBsubstitutionS,aparti ularpredi atedenotedbyprd
x
(S)de nestherelationbetweenthevaluesofthe statevariablesx before theexe utionofS andthe valuesofthestatevariable x
0
after theexe utionofS. prd
x
(S)isthebefore-afterpredi ateofS.Itisde nedinDef.2.Babstra tma hinesarede nedasinDef.3. De nition2(Before-after predi ate). Let S be a substitution. The before-after predi ate prd
x (S) is de nedasprd x (S)=hSi(x=x 0 ).
De nition3(B Abstra t Ma hine). ABabstra tma hineMisatuplehx;I;Init ;OPiwhere x isaset ofstatevariables,
I isaninvariantpredi ateoverx, Initisasubstitution alledinitialization, OP isaset ofoperationde nitionsasin Def.1.
WedenoteasX M
(whereX2fx;I;Init;OPg)a omponentoftheBmodelM.Ifthereisnoambiguity onthemodelthatis onsidered,wesimplydenoteitbyX.AmodelMde nesasetA
M
ofoperationnames andasetPred
M
ofBpredi atesoverthestatevariablesx ofM.
The test ases are nite exe utions. We rst de ne the notion of B tra e of a B abstra t ma hine in Def.4.Intuitively,aBtra eisa nitesequen eofoperationnamesstartingafter theinitialization.
De nition4 (BTra e). Let M= hx;I;Init ;OPi be aB abstra t ma hine. A tra e is a nite sequen e M =Init;op 1 ;op 2 ;:::;op n whereop i
isthenameofanoperation(2A M
)de ned inOP asinDef.1. Severalexe utions anbeasso iatedtoaBtra ebe ause,foranyoperationop
i
,therearepossiblyseveral parametervaluesv
i ofp
i
that satisfythetypingpredi ateT i
(p i
). As anbeseenin Def.5,anexe ution is aninstan eofaBtra ewithparametervaluesforeveryoperation allthatsatisfythepre onditionT
i (p
i ). De nition5(B Exe ution). Let M=hx;I;Init ;OPi beaB abstra t ma hine.Let
M = Init;op 1 ;op 2 ; :::;op n beatra eofM. M =(op 1 (v 1 );w 1 );(op 2 (v 2 );w 2 );:::;(op n (v n );w n )isanexe utionasso iatedto M , denoted by M 2Exe B (M; M
),ifthereisasequen eofstatevariablevaluesu 0 ;u 1 ;u 2 ;:::;u n ,asequen e ofstatuswordsw 1 ;w 2 ;:::;w n
and asequen eofparametervaluesv 1 ;v 2 ;:::;v n su hthat [x 0 :=u 0 ℄prd x (Init), foranyi21::n:[p i :=v i ℄T i (p i )^[x;x 0 ;sw i ;p i :=u i 1 ;u i ;w i ;v i ℄prd x (S i ).
Sin eweassumeourspe i ationstobedefensive(i.e.thepre onditionsarelimitedtotypingpredi ates), there isat least oneexe utionasso iated to aBtra eifT
i (p
i
)is asatis abletyping predi ate.Thanksto that,weassumethat theexe utionsrespe t thepre onditions,i.e.theenvironment(simulatedbythetest generator)always alls theoperationswithwell-typedparametervalues.Inother words,thetestgenerator hooses parameter values that satisfy the pre ondition, i. e. the typing predi ate T
i (p i ). Moreover, the operation allop i (v i
)fromthestateu i 1
givesthenewstatevariablevaluesu i
andreturnsthestatusword w i .u i 1 ,u i ,w i andv i
satisfythebefore-afterpredi ateofS i
.
3. IAS Case Study
ThisworkwasdoneintheframeworkoftheRNTLPOSEproje t,thatbringstogetherindustrial(GEMALTO, SMARTESTING,SILICOMP/AQL)anda ademi (LIFC/INRIACASSISproje t,LIG)partners.Theaim of the proje t was thevalidation of the onformityof a systemto its se urity poli y, espe ially for smart ards.
DF: file_02
DF: file_01
MF: (root)
PIN: pin_02
DF: file_03
EF: file_04
KEY: key_01
Fig.1.AsampleIAStreestru ture
testshadbeen omputedandexe utedonanIASimplementationbyGEMALTO.Wehaveextendedthese testswithtests omputedfromdynami sele tion riteria.
IASisastandardforSmartCardsdevelopedasa ommonplatformfore-AdministrationinFran e,and spe i edin [GIX04℄by GIXEL. IAS provides servi esto the other appli ations running on the ard. IAS onformstotheISO7816standard.
The lesystemofIASisillustratedwithanexampleinFig.1.Filesin IASareeitherElementary Files (EF), orDire tory Files (DF), e.g. file01 and file02 in Fig. 1.The le system is organizedasa tree stru turewhoserootisdesignedasMF (MasterFile).
TheSe urity DataObje ts (SDO) areobje tsof anappli ation that ontainhighly sensitivedata su h asPIN odes(e.g. pin02inFig.1) or ryptographi keysthat prote tanotherdata. They anbeusedto restri tthea essto someoftheappli ationdata.
Thea esstoanobje tbyanoperationinIASisprote tedbyse urityrulesbasedonse urityattributes. Thea essrules an possiblybeexpressed asa onjun tionofelementarya ess onditions,su hasNever (whi histherulebydefault,statingthatthe ommand annevera esstheobje t),Always(the ommand analwaysa esstheobje t), orUser (userauthenti ation:theusermustbeauthenti atedbymeansofa PIN ode).
Letuspresentthevariables ofthemodel thatweuse inaforth omingexampleofatestpurpose given in Se . 4.3. Let XID be a set of X identi ers, where X is either DF, PIN, OBJ or SDO. The variable urrentDF (2DF ID) storesthe urrent sele tedDF. The variable pin02dfParent (2PINID!7 DFID) is a partial fun tion that asso iates to a PIN the DF where it is lo ated. The variable rule2obj (2 SDOID[falways;neverg$OBJID) is arelation that asso iates to aSDO the obje tthat it prote ts. If theobje tis always(resp. never) a essible,then theSDO is repla ed by thevaluealways(resp.never). Thevariablepinauthenti ated2df(2PIN ID$DFID)isarelationthatasso iatesaPINwiththeDF wheretheownerof thePINisauthenti ated.
Consider for example the data stru ture shown in Fig. 1. The predi ate pin02 7! file01 2 pin02 dfParentistruesin ethePINobje tpin02islo atedintheDFfile01.Thepredi atepin027! file022rule2 objistrueifthea esstotheDFfile02isprote tedbyauserauthenti ationoverthe SDO pin02.If pin027!file022pinauthenti ated2dfis true,then thea essto theDF file02 isauthorized,otherwiseitisforbidden.
Theservi esprovidedbytheIASplatform anbeinvokedbymeansofvariousAPDU 4
ommands.Some of these ommands allow the reation of obje ts: for example, PUTDATAOBJPINCREATE reates a PIN ode, CREATEFILEDF reatesaDF, ... Someareused to navigatethroughthe le system,su hasSELECTFILEDF -PARENTorSELECTFILEDFCHILD.Somesetthevaluesofattributes:forexample,RESETRETRYCOUNTERisforresetting thePINtry ounter toitsinitial value,CHANGEREFERENCEDATA isfor hangingaPIN odevalue,VERIFY setsa validation ag totrueorfalsedepending onthesu essof anauthenti ationoveraPIN ode,... Other ommandsare for hangingthelife y le stateof les, su h asDEACTIVATEFILE, ACTIVATEFILE, TERMINATEFILE, orDELETEFILE, ...
Ina ordan ewith APDU ommands, theIAS platform respondsto a ommand bymeans ofastatus word(i.e.a odi ednumber),whi hindi ateswhethertheAPDU ommandhasexe uted orre tlyornot.If not,thestatuswordindi atesthenatureoftheproblemthatpreventedthe ommandfromendingnormally.
4
OP ::= operationname j "$op"
j "$opnf" OPLIST "g" OPLIST ::= operationname
j operationname "," OPLIST SP ::= statepredi ate
Fig.2.Synta ti RulesfortheModelLayer CHOICE ::= "j" j ""
Fig.3.Synta ti RulefortheTestGenerationDire tiveLayer 4. Test Purpose
We see a test purpose as a means to exer ise the system in a parti ular situation, for example w.r.t. a property.Basedonhisknow-how,anexperien edse urityengineerwillimaginepossibles enariosin whi h he thinks the property mightbe violatedby an erroneousimplementation. Hedes ribesthe s enarioasa testpurpose.
Wehavede nedin[JMT08a℄alanguagetoexpresssu htestpurposes.Itisbasedonregularexpressions andallowstheengineerto on eiveitss enariosintermsofstatestoberea hedandoperationstobe alled. WepresentthelanguageinSe .4.1.Thestartingnon-terminalofitsgrammarisSEQ.Wegiveitssemanti s inSe .4.2,andshowatestpurposeexampleinSe .4.3.
4.1. Language for Test Purposes Des ription
Wedesigned thelanguageto beasgeneri aspossiblew.r.t. themodelling languageused to formalizethe system.Thelanguageisstru turedasthreedi erentlayers:model,sequen e,andtestgenerationdire tive.
Themodel layer isfordes ribingtheoperation allsandthestatepropertiesin thetermsofthe behav-ioral model M. This layer onstitutes theinterfa e between M and thetest purposes, and is theonly one that is modelling language dependent. The sequen e layer is based on regular expressions and allows the des riptionoftheshapeoftest s enariosassequen esofoperation alls leadingto statesthatsatisfysome stateproperties.Thetest generation dire tivelayer isusedto dealwith ombinatorialissues,byspe ifying somesele tion riteriaintendedforthetest generationtool.
Wegivethesyntaxof ea h layer.An exampleofatest purposeissued from theIAS asestudy anbe seenin Se .4.3.
4.1.1. Model Layer
ThesyntaxofthemodellayerisgiveninFig.2.TheruleSPdes ribes onditionsasstatepredi atesoverthe variablesof M.TheruleOPallowsfordes ribingtheoperation alls,eitherbyanoperationnameindi ating whi h operation is alled,or by the token $opmeaning that any operation is alled,or by$opnfOPLISTg meaningthatanyoperationis alled ex eptonefromthelistOPLIST.
4.1.2. TestGeneration Dire tive Layer
Thispartof thelanguageis givenin Fig.3.It allowstospe ifyguidelinesfor thetestgeneration step.We proposeonedire tiveaimedat redu ingthesear hforinstantiationsofthetestpurposes.
TheruleCHOICEintrodu estwooperatorsdenoted asjandfor overingthebran hesofa hoi e. Let S
1 and S
2
betwotestpurposes. ThenS 1
jS 2
spe i esthat thetest generatormust generatetestsforboth S 1 andS 2 .S 1 S 2
spe i es thatthetestgeneratormustgeneratetestsforeitherS 1
orS 2
.Thisdire tiveis takenintoa ountbytheunfoldingfun tionthat willbeshowninFig.10andexplainedin Se .5.2. 4.1.3. Sequen e Layer
SEQ ::= OP j "(" SEQ ")" j OP " (" SP ")" j SEQ "." SEQ
j SEQ REPEAT j SEQ CHOICE SEQ REPEAT ::= "*" j "+" j "?"
j "f" num "g" j "f" num ",g" j "f," num "g" j "f" num "," num "g" Fig.4.Synta ti RulesfortheSequen eLayer
aregularexpression.
Astepofasequen eiseitheranoperation allasdenotedbyOP(seeFig.2)orasubsequen eofoperation allsthat leadstoastatesatisfyingastatepredi ate,asdenoted byOP (SP).
Sequen es anbe omposed bythe on atenationoftwosequen es, therepetition ofa sequen eorthe hoi ebetweentwosequen es.Weusetheusualregularexpressionrepetitionoperators(*forzeroormany times,+foroneormanytimes,?forzerooronetime),augmentedwithboundedrepetition operators(fng meansexa tlyntimes,fn,gmeansatleastntimes,f,mgmeansatmostmtimes,andfn,mgmeansbetween n and m times). Noti e that using the operators * and + possibly de ne in nite sets of tests. To be of pra ti alinterest, they will haveto beinstantiated bythe test engineer asexpli it numberssometime in thepro ess.Usingtheseoperatorsin atestpurposeallowstheengineertopostponethisde ision.
4.2. Semanti s of the Test Purposes
Thesemanti sofatestpurposeexpressedinourlanguageisgivenasalabelledtransitionsysteminDef.6. The semanti sof a test purpose TP is bound to a B abstra t ma hine M that is the spe i ation of the system under test. We say that TP is de ned on M. We give a unique name to any transition in a set T = ft 1 ;t 2 ;:::;t n
g. The binding between the semanti s of TP and M is su h that the transitions of the semanti sofTParelabelledbythenamesoftheoperationsofMinA
M
,andastatepredi ateofPred M
on thevariablesxofMisasso iatedto anystateofthesemanti sofTP.
De nition6(Semanti s of a Test Purpose). Thesemanti sofatestpurposeonamodelMisatuple hQ;q
0
;T;; ;Q f
iwhereQisa nitesetofstates,q 0
2Qistheinitialstate,Q f
Qisthesetofterminating states, T 2 T ! (Q2
A M
Q) is a nite set of labelled transitions that are named and denoted by t i 7 !q i 1 opi ! q i , 2Q! Pred M
is atotalfun tion that asso iatesa statepredi ate, denoted by(q i
), witheverystate,and 2Q!7 fj;gisapartialfun tionthat asso iateswitheverysour estateofa hoi e expressionitskindofoperator.
Tolightenthevo abulary,intheremainderofthepaper,thewordtestpurposeisusedbothfordesigning atest purposeexpressedinourlanguage,andfordesigningitssemanti s.
De nition7(TP Tra e). A nite sequen eof transitions TP =t 1 ;t 2 ;:::;t n
isatra eofatestpurpose TPifthere areq i 2Qandop i 2A M
su hthat foranyi21::n,t i 7 !q i 1 op i !q i 2T andq n 2Q f . Givenatra e TP
,there arezeroormanyexe utionsof TP
ontheBabstra tma hineonwhi h TPis de ned.
De nition8(TP Exe ution). LetM=hx;I;Init ;OPi beaBabstra tma hine. Let TP =t 1 ;t 2 ;:::;t n be a tra e of a test purpose TP = hQ;q
0 ;T;; ;Q f i de ned on M. TP = (t 1 (v 1 );w 1 );(t 2 (v 2 );w 2 );:::; (t n (v n );w n )isanexe utionasso iatedto TP ,denoted by TP 2Exe TP (M; TP
),ifthereareasequen e of statevaluesofTPq 0 ;q 1 ;q 2 ;:::;q n
,asequen eofstatevariablevaluesofMu 0 ;u 1 ;u 2 ;:::;u n ,asequen eof statuswordsvaluesw
1 ;w
2 ;:::;w
n
andasequen e ofparametervaluesv 1 ;v 2 ;:::;v n su hthat: [x 0 :=u 0 ℄prd x (Init), foranyi21::n:t i 7 !q i 1 op i !q i 2T, foranyi21::n:[p i :=v i ℄T i (p i )^[x;x 0 ;sw i ;p i :=u i 1 ;u i ;w i ;v i ℄prd x (S i )^[x:=u i ℄(q i ).
Wehavede nedinDef.6thesemanti sofatestpurposeasalabelledtransitionsystem.Weobtainitas follows.We rstexpresstheregularexpressionsasnormalforms,basedonthethreefollowingbasi operators:
operatorsarerede nedfromthesethreebasi operators.Theinstan esofthe onstru tions"$op","$opnf" OPLIST"g"andOP" ("SP")"are olle tedastheyare,intoasetLofatomi symbols.Se ond,fromthese normalforms,we omputeanautomatonhQ;q
0 ;T 0 ; ;Q f iwhereT 0
isasetoflabelledtransitionsintheset Q(A
M
[L)Qand isapartialfun tioninQ!fj;7 g.Weapplytheusualtransformationrulesofaregular expressionintoanautomatontogetit.Thereishoweveralittledi eren ewiththeusualrulesduetoourtwo hoi eoperators:with ,welabelthestateonwhi hthe hoi eo urswiththe orresponding hoi eoperator. Third, assuming that the name t of every transition in T is unique, we transform the automata hQ;q
0 ; T 0 ; ;Q f
ithat wehaveobtainedinto transitionsystemshQ;q 0
;T;; ;Q f
iasfollows. Letops bean OPLIST,abeanoperationnamein A
M ,b beanoperationnamein A M [f$opgandsp beastatepredi ate: t7 !q AM !q 0 2T ifq $op !q 0 2T 0 orq $op (sp) ! q 0 2T 0 , t7 !q AMnfopsg ! q 0 2T ifq $opnfopsg ! q 0 2T 0 , t7 !q a !q 0 2T ifq a !q 0 2T 0 orq a (sp) ! q 0 2T 0 , foreverystateq
0 2Q,(q 0 )= V (q i 2Qandq i b (sp i ) ! q 0 2T 0 ) sp i ;otherwise(q 0 )=true.
AtestpurposeTPde nesasetof nitetra esthatrepresentsasetofsymboli test ases.We allea h tra ea TPtra e (see Def.7). A TP tra eis a nite sequen e of transitions that iswell formed w.r.t. the transitionrelationofTP.Tobepre ise,letusnoti ethatitisa tuallyoneofthesetofsetsof nitetra es, due to the test generation dire tive represented by the fun tion and the operator . For example, the semanti softheregularexpression(a j b):( d)isoneofthefourfollowingsets ofTPtra es:fa: ;b: g, fa:d;b:dg,fa: ;b:dgorfa:d;b: g.Thesesymboli test asesmustbeinstantiatedastest ases(nonsymboli ), alled TP exe utions (see Def. 8) by asymboli animator from abehavioral model M and some overage riteria. In Def. 8,an exe ution is a nite sequen e of pairsmade of anoperation all provided with the valuesof itsparameters,andtheexpe tedstatuswordvaluereturnedbytheoperation all.
Theexe utionsare easyto omputebyatestgeneratorwhentheTPtra esaresequen esoftransition names whoselabelshave allbeeninstantiated, i.e. in whi h there is no $oplabelon thetransition. Ba k-tra kingmaybene essaryto satisfy the onstraintsset by thepredi ates for thestates to rea h, and the enabling onditionsoftheoperations.
AsfortheBexe utions,severalTPexe utions anbeasso iatedtoaTPtra eforthesamereasons.But intheTPexe utions,everyoperation allop
i (v
i
)mustmoreoverleadtoastatethatsatis esthetargetstate predi ate (q
i
)whi h is asso iated to the targetstate q i
of the test purpose. For that, in Def. 8,wehave added thefollowing ondition for any i: [x:=u
i ℄(q
i
). Consequently,it is also possible that no exe ution is asso iated to a TP tra e if there is no sequen e u
1 ;u
2 ;:::;u
n
of state variable values that satisfy the sequen e(q 1 );(q 2 );:::;(q n
)oftargetstateproperties.
4.3. Test Purpose Example
Here,weexhibit oneof the test purposes written for theexperimentation of ourapproa h. Wewantedto test aproperty sayingthat \to a ess an obje t prote ted by aPIN ode, the PIN must be authenti ated". Wehavewrittenatestpurposethat ausesthelossofthePINauthenti ationinallpossibleways,andthen triestoa esstheobje t.Thetestpurposeisgivenintwostages:theinitializationstageandthe oretesting stage.
Figure 5 presents theinitialization stage of the test pattern in four steps, aiming at building thedata stru ture required on the ard to run the test. The DF file 01 and file02 and the PIN pin02 are names of obje ts that are de ned in the des riptionof the TP. Theirtypes are de ned from the types of parametersthat theyinstantiate.Noti ethat the targetstatepredi atesare expressed in thetestpurpose asB predi atesoverthe obje tsde lared in the TPand the state variablesof the Bmodel M (seeSe . 3 forthe explanationof thevariables usedin this example).Theaim ofthe rststepis to reate anewDF denoted file01.These ond stepaimsat reatingaPINobje tdenoted pin02into theDF file01and gaininganauthenti ationoverit.Theaimofthethirdstepisto reatetheDFfile02intotheDFfile01. Finally,thelast stepaims at settingthe urrentDFto file01 inorder to startthe oreof thetest. The resultingdatastru tureisthatofthedashed ir ledpartoftheFig.1:theDFfile02isprote tedbythe
CREATEFILEDF
(rule2obj[f le01g℄=falwaysg^ urrentDF= le01) //P1 .PUTDATAOBJPINCREATE.VERIFY
(PIN2dfParent(pin02)= le01
^ le012pinauthenti ated2df[fpin02g℄) //P2 .CREATEFILEDF
(rule2obj[f le02g℄=fpin02g^ urrentDF= le02) //P3 .SELECTFILEDFPARENT
( urrentDF= le01) //P4
Fig.5.Exampleofatestpurpose|initializationstage .(VERIFYjCHANGEREFERENCEDATA
j(RESET.SELECTFILEDFCHILD)jRESETRETRYCOUNTER j(SELECTFILEDFPARENT.SELECTFILEDFCHILD))
( urrentDF= le01^ le012=pinauthenti ated2df[fpin02g℄) //P5 .SELECTFILEDFCHILD
( urrentDF= le02) //P6
.CREATEFILEDFjDELETEFILEjACTIVATEFILEjDEACTIVATEFILE jTERMINATEFILEDFjPUTDATAOBJPINCREATE
Fig.6.Exampleofatestpurpose|exe utionstage
WehavegiveninFig.5andFig.6alabeltoea htargetstatepredi ates,sowe anrefertoitafterwards. Theselabelsappearasdoubleslashed ommentsontherighthandofea hpredi ate://P1,//P2,et .
Figure 6 shows the ore testing stage, des ribing the test purpose of asu essful authenti ation after allpossiblewaystoloseanauthenti ation.First,thepatterndes ribesthe vepossiblewaysforlosingthe authenti ation overthePIN pin02(forinstan e, afailure oftheVERIFY ommand orareset ofthe retry ounter).Theaimofthese ondstepistosele ttheDFfile02,withthe ommandSELECTFILEDFCHILD. The nalstepofthetestpatterndes ribestheappli ationof six ommands, withthe urrentdire tory le beingfile 02in ordertotestthe orre tnessofthea ess onditions.
The omplete test purpose is represented as an automaton in Fig. 7. The edges are labelled by the operationnamesofthepatternandthelabelsintheverti esrefertothetargetstatepredi atesPiofFig.5 andFig.6.Predi atetruedenotesastatethat isnot onstrained.
5. Model-Based Testing Pro esses
Thisse tion rstdes ribesamodel-basedbla k-boxtestingpro essusing stati stru turalsele tion riteria to omputetestsfrom amodel.Thenwe ompletethispro essbyusing adynami sele tion riterion(TP) instead of stati ones, to ompute additionaltests. This approa h is implemented within the Leirios Test
P 1 true P 2 P 3 P 4 P 5 P 6 true
CREATEFILEDF PUTDATAOBJPINCREATE VERIFY
CREATEFILEDF SELECTFILEDFPARENT
VERIFY . . .
CHANGEREFERENCEDATA
SELECTFILEDFCHILD CREATEFILEDF
. . .
Fig.8.Fun tionalModel-BasedTestGenerationPro ess
Generator(LTG)tool[JL07℄from Smartesting,that takesBmodels asinputs.TheLTGtest omputation algorithm,presentedin [CLP04℄,isbasedonstru tural overage riteriaoftheoperationsofthemodel.
5.1. Model-Based Testing with Stati Sele tion Criteria 5.1.1. Model-Based Testing Pro ess
The pro ess for omputingmodel-basedfun tional tests is summarized by Fig.8. The pro ess is made of threesteps.
TestGeneration.Asetoffun tionaltests is rststati ally omputedfromabehavioralfun tionalmodel Ma ordingtosomestati sele tion riteria.Inour ase,thetestgenerationisperformedbyLTG.The tool omputestesttargetsfromthemodela ordingto ontrol ow,de ision, onditionanddata overage riteria,asfurther detailedin Se .5.1.2andSe .5.1.3.
Con retization.Asthetests omputedhavetheabstra tionlevelofthefun tionalmodelM,theyhaveto betransformedinto on retetests,atthelevelof theimplementationunder test(IUT).Thissteprelies onthe on retizationlayerwhi h mapstheoperationsanddataof Mtotheoperationsanddata ofthe IUT,asfurther explainedin Se .5.1.4.
Exe ution.In this step theverdi t is given bythe omparison between theoutputs predi tedby M as in ludedinthe on retetests,andtheoutputsgivenbytheexe utionoftheIUTonthedataappearing in the on retetests(seeSe .5.1.4).
Thedashed ir ledpartsinFig.8showwhatinthepro esswillbereusedtogeneratetestsfromdynami sele tion riteria(TP),in additiontothefun tional ones.This willbeperformedbyrepla ingtheabstra t fun tional tests entering the right hand dashed ir led part by abstra t dynami tests generated from a fun tionalmodelMandaTPasitisshowninFig.10.
Thenextthree se tionsdetailthe ompositionofthetest ases,thegenerationof testtargetsby appli- ationofstati overage riteriaand nallythe on retizationoftestsequen esintoexe utables ripts. 5.1.2. TestCase Composition
The purpose of the model-based testing approa h of LTG is to a tivate the operations of the B model. More pre isely, it fo uses on a path- overage of the ontrol ow graph of the operations, in whi h ea h pathis alledabehavior. Thus,ea hoperationis overeda ordingtoitsstru ture,byextra tingitsnested behaviors.Ea h behavioris omposed oftwoelements:ana tivation onditionandane e tthat des ribes theevolutionofthestatevariablesifthea tivation onditionissatis ed.
Forea hbehavior,atesttargetisde nedasitsa tivationpredi ate( alledde ision).Thetests overing thebehaviorwill be onstituted ofapreamble that putsthe systemin astatethat satis es thea tivation predi ate of the behavior. To a hieve that, ustomized algorithms automati ally explore the state spa e de ned by the B model and nds one path from the initial state to a state verifying the target. LTG automati ally sele tsthe shortest preamble that rea hes the test target. It is equipped with a onstraint
preamble
00
00
00
00
00
11
11
11
11
11
00
00
00
00
00
11
11
11
11
11
00
00
00
00
00
11
11
11
11
11
00
00
00
11
11
11
00
00
00
11
11
11
00
00
00
11
11
11
postamble
identification
body
00
00
00
00
00
11
11
11
11
11
Fig.9.CompositionofaLTGtest ase
Apartfromthepreamble,atestisthus omposedofthe4elementsshowninFig.9.Thetestbody onsists of theinvo ationof thetested operation with theadequateparameterssothat the onsidered behavioris e e tively a tivated. The identi ation phaseis aset of user-de ned operation alls that are supposed to performthe observation of the systemstate.Their invo ationwhen playing the test ase on theIUT will makeitpossibleto omparethe on retelyobservedvaluesw.r.t.theirexpe ted values omputedfrom the model.Finally,atest aseisendedbyapostamblethatisanoptionalsequen eofoperations allsthatresets thesystemto itsinitialstatesoasto hain thetest asestogether.
5.1.3. Coverage Criteria for Test Target Generation
From thepreviousbasi de nition ofatest target,basedon the overageof thestru tureof theoperation model,twoothermodel overage riteria anbeapplied,namelypredi ateanddata overage.These riteria aresele tedbythevalidation engineer.
Predi ate overagemakesitpossibletoin reasethetesttargetsnumber,andpossiblytheirerror de-te tionabilities.Thisprovidesameanforsatisfying lassi alpredi ate overage riteriathatare:(i)De ision Coverage (DC)statingthatthetestsevaluatethede isions(ea ha tivation ondition)atleaston e,(ii) Con-dition/De isionCoverage (C/DC)statingthatea hbooleanatomi subexpression( alled a ondition) ina de isionhasbeenevaluatedastrueandfalse,(iii)Modi edDe ision/ConditionCoverage (MC/DC)stating that ea h ondition ana e t theresultofits en ompassingde ision,or(iv) MultipleConditionCoverage (MCC)stating that thetestsevaluateea hpossible ombination ofsatisfyingapredi ate.Inpra ti e, dif-ferentrewritingrulesareapplied onthedisjun tivepredi ateformof thede isions,soastore nethetest targetsinorder totakethis overage riteriaintoa ount(formoredetailssee[UL06℄).
Data overage makes itpossible to indi ate whi h of thetest data have to be omputed in order to instantiatethetests.Theoptions,appliedtooperationparametersand/orstatevariables,proposea hoi e between:(i)all thepossiblevaluesfor agivenvariable/parameterthatsatisfythe testtarget, (ii)asmart instantiationthat sele ts asinglevalue forea htest data, or(iii) boundaryvalue overage,fornumeri al data,that willbeinstantiatedtotheirextremavalues(minimalandmaximalvalues).
5.1.4. Exe utable S ripts and Verdi ts
On etheabstra t test aseshavebeen omputed,theyhaveto betranslatedinto thetestben hsyntaxso asto beautomati allyexe utedontheIUT.Thisisthe on retizationstep.
Toa hievethat,the validation engineerhas to provide two orresponden e tables. Oneof these tables mapsthe operationsignaturesoftheBmodelto the ontrolpointsofthetest ben h.Theother onemaps theabstra t onstantvaluesoftheBmodeltotheinternaldatavaluesoftheIUT.Byusinganappropriate translator,atests riptisautomati allygeneratedinto thesyntaxofthetestben h,readytoberunonthe IUT.The orresponden etablesandthetranslatorimplementthe on retizationlayer.
Forea htest,theverdi tisestablishedby omparingtheoutputsofthesysteminresponsetoinputssent asthesu essiveoperations.The on retizationlayerisin hargeofdeliveringtheverdi t,byimplementing fun tionsthatperformthe omparison.Inthis ontext,themoreobservationoperations(identi ationphase ofFig.9)areavailable,themorea uratetheverdi tis.
Limitations This approa haims at ensuring that the behaviors des ribed in the model also exist in the IUT, and their implementation onforms to the model. Nevertheless, this approa h su ers from several limitations.
Fig.10.Pro essforGeneratingandExe utingTestsfromaBmodelandaTestPurpose
of varietyin the omposition of these preambles,whi h may avoid revealing errors. Se ond,the preamble omputationisboundedin depthand/ortime.Thismaypreventatesttargetfrombeingrea hed.
To over ome these limitations, we now present amodel-based testing approa h that onsists of using dynami sele tion riteriato omputenewtestsw.r.t.theonesissuedfrom LTG.
5.2. Model-Based Testing with Dynami Sele tion Criteria
Ourpro essforgeneratingtestsusesatestpurposeTPassele tion riterionandaBbehavioralfun tional model M asora le.The omplete pro essis des ribed by Fig.10. Noti ethat thedashed ir led parts are thesameasinFig.8,showingwhatisreusedfromthepreviouspro ess.Herewerepla ethe omputationof abstra tfun tionaltestsbasedonstati sele tion riteria,bya omputationofabstra tdynami testsbased onaTP.Theabstra tdynami test omputation ismadein threesteps:
syn hronizeMandthesemanti sofTPin M TP
, omputethesetofTPtra es
TP
unfoldingthesemanti s ofTP, omputethesetofTPexe utions(abstra tdynami tests)fromM
TP
and theset ofTPtra es.
Computingtheabstra ttest asesisobtainedbyasymboli animationoftheTPtra esonaBma hine M
TP
thatis thesyn hronizedprodu tbetweentheBmodel Mandthe testpurposeTP. Thesyn hronized produ tbetweenMandTPis omputeda ordingtotheexpressionin BthatisgiveninSe .6.Theresult isaBma hineM
TP
whoseexe utionsarethepossibleexe utionsfromMthat onformtoTP.Besides,TPis unfoldedasa nitesetofTPtra es(seeDef.7)
TP
,i.e.assequen esoftransitionnames(ea honelabelled with anun-parameterized operation all) de ned a ordingto TP, but withoutthetarget states.This set omputesalltheTPtra eswhoselaststateisterminating,andwhoselengthislowerorequaltoamaximum lengthde nedbythetester.
WeuseLTGtoinstantiatetheTPtra es.LTGisalsoaBtra e(seeDef.4inSe .2)animator,usedby thetestengineertovalidateitsmodelsandmanually ompletethetestssequen es.ATPtra eisaBtra e ofM
TP
.LTGpro eedsbysymboli animation.Noti ethat anyothertoolwithsimilar apabilities ouldbe used for that purpose. The prin iple is to \guess"values forthe parameters of the operations that make itpossibleto exe utethesequen eofoperationsasdes ribedbyaparti ulartra e
TP
of thetestpurpose TP.Inother words,TPexe utionsare omputedbyLTGanimation apabilities from TPtra esand M
TP . The parametervaluesare omputed in LTGby a onstraintsolver, that nds somevaluesthat make the sequen esofoperationsof
TP
rea hthetargetstatesgivenin theTP. Noexe utionis omputedwhenthe target states are impossible to rea h. The status words are also omputed as expe ted by M
TP
for these parameters.Additionally,fromoneTPtra e
TP
,LTGwill tryto omputeaTPexe ution.
MACHINEM VARIABLESx INVARIANTI INITIALISATIONInit OPERATIONS ... sw i op i (p i )=
PRETi(pi)THENSiEND ... END MACHINEM TP INCLUDESM SETSQ=fq0;:::;qng VARIABLESCq INVARIANTCq2Q /*Cq: urrentstateofTP*/ INITIALISATIONCq:=q 0 OPERATIONS /*foranyt i 7 !q i 1 op i !q i 2T */ /*wede neanoperationti s.t.*/ ... sw i t i (p i )= PRET i (p i )THEN SELECTCq=q i 1 ^9(x 0 ;sw 0 i ) (prdx(S i )^[x:=x 0 ℄(q i )) THENsw i op i (p i )jjCq:=q i END END; ... END
Fig.11.CombinationofamodelMandatestpurposeTPonM
6. Combining a Model and a Test Purpose for Dynami Sele tion of Tests InFig.11,wede nehowtoexpressin Bthesyn hronizedprodu tM
TP
ofabehavioralmodelMdes ribed asaBabstra t ma hine,andatestpurposeTPonM.M
TP
in ludes theabstra tma hineMso thatit an read thestate variables x of M, andit an syn hronizeany transition t of TPwith a allto an operation of M labelled by t. The variable Cq represents the urrent state rea hed by the last transition exe uted in the test purpose TP. The initial state is q
0
. For any transition t i (su h that T(t i ) = q i 1 opi ! q i ), we de neanoperationalso alledt
i inM
TP
.Itsparametervaluesmustsatisfythetypingpredi ateT i (p i )ofthe operationop i thatis alled byt i
.This operationisenabledifthe urrentstateisq i 1
andiftherearestate variablevaluesx
0
and astatuswordvaluesw 0 i
aftert i
thatsatisfythebefore-afterpredi ateofthebody of theoperationop
i
andthetargetstatepredi ateofthetest purpose(q i
).Whenthese onditionshold, the operationt
i
alls theoperationof Mop i
andpla esthesysteminthetargetstateq i
ofthetestpurpose. Theorem1establishesthesoundnessofthemethod.ForaTPtra e
TP =t 1 ;t 2 ;:::;t n
(seeDef.7),anyB exe ution(seeDef.5)oftheB omposedabstra tma hineM
TP
fortheBtra e MTP =Init MTP ;t 1 ;t 2 ;:::;t n is aTPexe ution(see Def.8) of
TP
ontheabstra t ma hine M. Theorem 2establishes themethod om-pleteness.
Theorem1(Soundness). LetM TP
betheB ompositionofaBmodelMandatestpurposeTPonMas inFig.11,andlet
TP beaTPtra ethen, Exe B (M TP ;Init M TP ; TP )Exe TP (M; TP ):
Proof. Theproof relies on the fa t that,the di eren ebetweenthe Bexe utionsof the model M and the TP exe utionsof M, is that,thetarget predi ate (q
i
) holds in everytarget stateq i
of theTP exe ution. This ondition is also satis ed in the B exe ution of M
TP
sin eweadd this ondition in the guard of its operationst
i
(seeFig.11). Moreover,itis obviousthat theBexe utionsof M TP
andtheTPexe utions of M omputethesamesequen eofstatesasTP,andexe utethesamesequen eofoperation allsasM. Theorem2(Completeness). GivenaB ompositionM
TP
ofaBmodelM,atestpurposeTPonM and aTPtra e TP , Exe TP (M; TP )Exe B (M TP ;Init MTP ; TP ): Theproofisstraightforward.
OurimplementationwithLTG omputestheBexe utionofM TP
Testpurpose ℄operations ℄transitions ℄states
TP1 12 13 12
TP2 10 17 14
TP3 9 15 12
Table1.Testpurposesdes ription
7. Experimental Results
Inthispart,wereportand ommenttheresultsofanexperimentationdonewithase urity-basedBmodel ofIAS,whi his1032lineslongand ontains12Boperationsand19statesvariables.Thismodelfo useson a ess ontrol,andinparti ularonuserauthenti ationbymeansofaPIN ode.
InSe .7.1, wepresentthe goalofourexperiments. Wededu efrom thisobje tivethe riteriathat we mustevaluatetorea hit.Thenweproposeanexperimentalproto ol.InSe .7.2,wepresenttheexperimental results,andwe on ludein Se .7.3withtheanalysisoftheresults.
7.1. Goal, Means and Pro ess of Experimentation
Thegoalofourexperimentationsistoanswerthequestionofthe omplementarityofthetest asesgenerated from dynami sele tion riteria, w.r.t. the test ases generated from stati sele tion riteria. We have to addresstwopointstorea hthisgoal:
weneedsets oftest asesgeneratedeither withdynami orstati sele tion riteria, weneed overageevaluation riteriainorderto omparethedi erenttestsuites.
Asforthe rstpoint,wehavegeneratedfourtestsuites(seeTable2)named LTG,TP1,TP2andTP3. TheLTGtest suitehavebeengenerated usingC/DCstati sele tion riteriawith thetoolLTG.Thethree other test suites havebeengenerated using dynami sele tion riteriain the shapeof three test purposes named TP1, TP2 and TP3. Table 1 gives the number of operations, the number of transitions and the numberofstatesof ea htest purpose.The rsttestpurpose, thatisde ned in Se .4.3, aimsatprodu ing test sequen es ombining di erentways to lose the authenti ation overaPIN ode with the laun hing of di erent ommands prote ted by this PIN ode. The se ond test purpose aims at validating the orre t interpretation of an a ess rule, in a ontext where a onfusion ould o ur between two di erent PIN obje ts, due to the omplexity of the IAS obje t referen e me hanisms. The third test purpose aims at he kingthebehavioroftheappli ationwhenanauthenti ationoveraPINobje tis ombinedwith lelife y le hanges.
As for the se ond point, we havede ided to evaluate the overage of ea h test suite with respe t to a ommonframeofreferen e.Dire tlytakingtheIASmodelasareferen efor omparingthe overageofthe test ampaignswoulnothavebeenagood hoi efortworeasons: rst,thenumberofstatesandtransitions is toobig and se ond,thepart overedby aparti ular testpurpose wouldbetooweakto givesigni ant results.Thus,wehavede idedtogenerateanabstra tionofthemodelbyfo usingonvariablesgivingagood pointofviewofthestatesofthesystemtargetedinthetestpurposes.Thisabstra tionhasbeen omputed by the GeneSyst tool [BPS05℄. This tool omputes a symboli labelled state-transition system from a B modelandthedes riptionofthesymboli statesthat wewanttoobserve,i.e.thedomainde omposition of the hosenvariables.Inour ase,thegraphprodu edforIASwasmadeof18statesand497transitions.
In order to obtainan abstra tion whi h is relevant with respe t to theobservation ofthe system, and inparti ularthea ess ontrolbasedonuserauthenti ationbymeansofaPIN ode,wehave hosenthree variables.Thesevariablesare: urrent DFthatmodelsthelo ationofthe urrentdire tory;df2dfParent
5 thatrepresentsthestru tureofthedire torytree;andpinauthenti ated2dfthatindi atesthe authen-ti ationstatusofaPIN odeinside aDF.This hoi eofvariablesgaveusanabstra tionwellsuited tothe observationofthe overageofthetestsprodu edwiththetestpurposesTP1andTP2.Butthisabstra tion isnotwellsuitedtostudythe overageofthetestsgeneratedfromthetestpurposeTP3.Thisisduetothe fa t thatTP3aimsat testingthe ombinationoftheauthenti ationme hanismwith lelife y le hanges. Thevariablerepresentingthe lelife y lestatehasnotbeentakenintoa ounttoprodu etheabstra tion, 5
Tests ℄tests Averagelength Minlength Maxlength
LTG 65 2.5 1 5
TP1 35 9.4 9 10
TP2 66 9.5 8 11
TP3 88 6.9 5 8
Table2.Testgenerationresults
Tests ℄tests State overage Transition overage LTG 65 5/18=27.78% 33/497=6.64% TP1 35 9/18=50.00% 35/497=7.04% TP2 66 12/18=66.67% 52/497=10.46% TP3 88 5/18=27.78% 23/497=4.63% TP123 189 13/18=72.22% 87/497=17.51% Table3.Testsuites overagemeasures
be auseitresulted in toomanysymboli states.It ouldbeinteresting toprodu eanotherabstra tion to studythe overageresultsofthetests produ edwiththetest purposeTP3.
7.2. Results of Test Generation and Comparisonof the Test Suites
Tables2,3and4givetheresultsofourexperimentations.We onsiderthefollowingtestsuites:
the LTG test suite, where tests havebeen generated using behavior overage riteriawith overage of onditionsandde isions(C/DC)and overageofboundaryvaluesfortheoperationparameters; the three test suites TP1, TP2 and TP3, where tests havebeen generated using respe tivelythe test
purposesTP1,TP2andTP3asdynami overage riteria.
Table2indi atesforea htestsuitethenumberoftests omputed,theaveragenumberofoperation alls pertestsequen eandtheminimalandmaximalnumberofoperation allspertestsequen e.
Table3presentsthestateandtransition overagea hievedbyea htestsuiteaswellasbytheunionof thethreetestsuitesgeneratedusingthetestpurposes.
The omplementarity ofa testsuite e 1 w.r.t. atest suitee 2 isdenoted as omp(e 1 ;e 2 ).We measure it as the ratio between the number of transitions overed solely by e
1
(i.e. not by e 2
) and the full number of transitions overedby e
1
(possibly in ludingtransitions also overedbye 2
). If ov(e) is thenumber of transitions overedbyatestsuitee,then omp(e
1 ;e 2 )= ov(e1[e2) ov(e2) ov(e 1 ) .
Weneed additional overageresultsgivenin Table4to measurethe omplementarityofthetest suites issuedeither fromLTG orfromthetestpurposes:
TP1[ LTG,TP2 [ LTGand TP3[ LTG givethe overagea hievedby the unionof ea h test suite issuedfromthetestpurposeswiththeLTGtestsuite;
TP123[LTGgivesthe overagea hievedbytheunionofallthetestsuites.
Thelasttwo olumnsof Table4givetheper entageof transitionsthatarenotredundantly overedbythe testsuitesofLTGandbytheonesissuedfrom thetestpurposes.
Testsuite ℄tests State overage Transition overage omp(LTG;TP i ) omp(TP i ;LTG) TP1[LTG 100 9/18=50.00% 63/497=12.68% 28/33=84.8% 30/35=85.7% TP2[LTG 131 12/18=66.67% 83/497=16.70% 31/33=93.9% 50/52=96.2% TP3[LTG 153 6/18=33.33% 51/497=10.26% 28/33=84.8% 18/23=78.3% TP123[LTG 254 13/18=72.22% 109/497=21.93% 22/33=66.7% 76/87=87.4%
7.3. Report and Con lusion About the Results
The overage evaluation orroborates the fa t that the tests generated using test purposes as dynami overage riteria omplementthetestsgeneratedusingstati riteria.
Table2showsthattheaveragelengthofthetestsgeneratedfromthetestpurposesisbetween2.7and3.8 timeslonger thanthetests generatedfrom stati sele tion riteria.Table3showsthat thetestsgenerated fromthetestpurposes overuptotwi easmanystatesandtransitionsthanthetestsgeneratedfromstati sele tion riteria.
The rstpartof Table3shows thatthe test suitesobtainedfrom test purposes giveabetter overage ofthestatesandtransitions oftheabstra tion thanLTG,ex eptforthelasttestpurposeTP3.Thebetter overage{su h as66.67%ofstatesand10.46% oftransitionsforthetestsgeneratedusingTP2{isdueto thefa tthattestpurposesweredesignedtotestthea ess ontrol,andthattheabstra tionhasbeen hosen to fo usonit. Thepoor overageresultsobtainedwiththethird test purposeare dueto thefa t that the abstra tionwasnotsuitedto TP3(seeSe .7.1).
The resultsgiven in Table4 learlyshow that thereis littleredundan y betweenthe testsissued from LTG and theones issued from the test purposes. Nearly 85%and more of thetransitions overedby the LTG tests are not overed by the test purposes ones, and vi e-versa. There are two slightlylower ratios. \Only" 66.7%of theLTGtestsdi er from theunionof the onesissued from TP1, TP2andTP3. This is notsurprisingsin etheinterse tionsoftheLTG testswithea hofthethree testpurposesareput together bythismeasure.Wealsoseethatlessthan80%oftheTP3testsare omplementarytotheLTGones.This omes again from the abstra tion not well suited to TP3. Nevertheless, the ratio (78.3%) remains good. Finally,wethinkthatAll-Transition-Pairs overage riterion(everypairofadja enttransitionsinthestate transitionmodelmustbetraversedatleaston e),whi hhasnotbeenstudiedinthispaper, ouldalsoserve ourintentiontoshowthe omplementarityofthedi erenttestsuites.
These resultsshowthat wehavein reased the overageof thesystem{inparti ular, thea ess ontrol partwhi h isobservedbytheabstra tion{bygeneratingtest suitesfromthethree di erenttestpurposes. Theseresultsalsoshowthatthetestpurposesthatwedesignedleadto omplementarytestsequen esw.r.t. thetestsgeneratedfromstati sele tion riteria.
8. Con lusion
We have presented in the B framework amethod for generating tests from test purposes in a behavioral model-based testing ontext. Wehaveperformed experimentson the industrial smart ard platform IAS. This experimentation shows that the tests that we have generated are omplementary w.r.t. the stru -turalones [BLLP04, SLB05℄. Themethod makesuseof alreadyexisting material,written formodel-based stru tural testing:the behavioralmodel, the on retizationlayerand thetest exe utionenvironment.The approa h alsore-uses theset theory onstraint solversand thealgorithms forpreamblesear hing ofatest target. Additionally, test purposes are written by a test engineer to des ribe his test intentions.We have presented a languagededi ated to the expression of the test purposes. The language allowsthe tester to des ribeoperationstobe alledaswellasstatestoberea hed.Writingatestpurposeneedsgoodexpertise inthemodelofthesystemonbehalfofthetester.Hemustexpressthesetofexe utionsforwhi hhewishes atestsele tionbyatestpurpose.Buttheexpressivityofthelanguagethatweproposemakestheir des rip-tionseasier,thanksforexampletotheuseofregularexpressions.Ingeneral,itwould befarmorediÆ ult, if possible at all, to drivethe stati generatorby transformingthe behavioralmodel and/or adapting the stati sele tion riteria,in su hawaythatit ndssimilarteststotheones generatedfromtestpurposes.
Themethodeasilyensuresthetra eabilityof thetestsgeneratedtotheoriginaltestpurposes,sin ethe testsare omputedfromthem. Also,withthetra eabilityme hanismforfun tionaltestgenerationthat we use,weknowwhi hoperationbehaviorshavebeen overed.
AmongtheworksonModel-BasedTesting,someusestati (orstru tural)testsele tion riteria[EFHP02, BLLP04,UL06℄,appliedtothebehavioralmodel.Someotherworksapplydynami riteria.Ourworks tin thisse ond ategory,and ompleteatestgenerationenvironmentbasedonstati riteria.Dynami sele tion riteria target spe i lasses of exe ution of the system. The aim is to test dynami properties su h as safetyproperties,se urityproperties(a ess ontrol[DJM08,PMLT08℄,integrity,authenti ation,et .),and
riteriaaredes ribedasinput-outputlabelledtransitionsystems.Wehave alledtestpurposesthesedynami sele tion riteria.
Many other works use test purposes assele tion riteria to extra t tests from a model. The test pur-posesaredes ribedbytemporalpropertiesinatemporallogi [ADX01,TSL04℄,inputoutputLabelled(or Symboli )TransitionSystemsioLTS(ioSTS[JJRZ05,CIVDP07,FTW05℄),oruse ases[GHN93℄.
Asinalltheseapproa hes,ourmethodperformsthesyn hronizedprodu tbetweenthetestpurposeand abehavioralmodel. Twopointsmakeourmethod di erentfrom theapproa heswith properties expressed as temporal logi formulas. On onehand, the test purposes express atest intention from the tester by a ombinationofstatesequen ing(asintemporallogi )andoperation alls(whi hdoesnotexistintemporal logi ).Ontheotherhand,thetestgeneration te hnology isdi erent.Temporal logi basedapproa hesuse model- he kers,that generate tests by exhibiting ounter-examples. Our approa h uses onstraintsolving te hniquestoperformsymboli exe utions,onsymboli valuesoftheparametersoftheoperations.Thusitis possibletotreatin nitedatadomains,thankstostrategiesofstati sele tionof nitesetsofrepresentatives. Finally, theapproa h [ADX01℄ uses property mutation te hniques,basedon synta ti al transformation of operators.Inourapproa h,thetester ombinesatestneedwithaproperty,whi h anbeseenasasemanti mutationofaproperty.Ourmutationsintrodu emodi ationsinthesequen ingofoperation allswhilethe automati mutationstransformthepropositionalorrelationaloperatorsusedin theatomi onditions.
Our approa h di ers from approa hes su h as the oneadopted by TGV [JJ05℄ (resp. STG [JJRZ05℄) thatuseIOLTS(resp.IOSTS)expressingoperation alls,withnoinformationonthetargetedstates.These approa hes use onstraintsolving te hniques on data in integerand boolean s alar domains. We also use onstraintsolversonmore omplexdatastru turesofsettheorydomains,inordertofullytreatthe behav-ioralBmodellinglanguage(sets,fun tions,relationsandsequen es).Theapproa heswith IOSTS alsouse symboli exe ution te hniquesby abstra t interpretation, to redu e the size of the syn hronizedprodu t. Theunrea hablestates aresuppressedby over-approximation.This abstra tinterpretation allowstreating symboli models.Ourapproa husesasymboli modeltoevaluatethetests overage.
In[SML06℄, the authors present a test ase generation algorithm from B eventsystems and use ases byre nement.Therearethreemaindi eren eswithourapproa h.Ourmethodreusesabstra tBma hines anda on retizationlayerCLdedi atedtothefun tionaltestgeneration.Thereforewedonotre nethetest ases.Moreover,ourtestpurposesaremoreexpressiveuse asesthat ontaintarget statedes riptions.
Asadi eren ewiththepre edingapproa hes,wehaveshowninapreviouswork[MJP +
07℄howthetest purposes anbeautomati ally omputed,bymodellingsometestneedsassynta ti transformationrulesthat transformbehavioral properties. Weare urrently working at identifying and writingsu h transformation rules,basedontheIAS asestudy.Thisworkneedstobedevelopedbystudyingmanyother asestudies(for instan e,themini- hallengethatproposestodesignandverifyaPOSIX ompliant ash-basedsystem[JH07℄) in orderto produ erules suÆ ientlygeneri to beappli ableto avarietyofexamples.Rules ouldalsobe automati allydedu edfromthesynta ti expressionofaproperty,assuggestedby[BDGJ06℄forproperties expressedinJTPL, atemporal logi forJML.
Themethodthatwehavepresentedworkswell,andisappli abletoindustrialsizeappli ationsaslongas theTPsarenottoogeneri .Bythat,wemeanthatthe onstru tions$op
+ or$op
,althoughallowedbythe language,arenotusedbythetester.Ifno$opisusedatall,thenalltheoperation allsareexpli itlyde ned, andwe ndtheirparametervaluesbyanimationofthebehavioralmodelM.If$opisusedwithnorepetition operator,itisstilleasytoinstantiateitasanoperation all:thisisobtainedbytryingeveryoperationatmost on e.Butwhenthe onstru tions$op
+ or$op
areused, thevaluation be omesmore ompli ated.Indeed, every su h onstru tion hasto be instantiated, i.e. repla ed by a sub-sequen e of valuated and expli itly de ned operation alls.Thisimplies sear hing amongstall thepossibleinstantiations,onefor whi h there areparametervaluesthat ausethesub-sequen eto rea h thetargetedsymboli statespe i edin theTP. There is a ombinatorial explosion ofthe possibilities. Todealwith this situation,weplan to generate an abstra tion ofthe system,basedonvariablesand sub-domainsidenti edin the TP.We ouldsyn hronize thisabstra tion withtheTP.Wewouldthusobtainaviewofthesystemwhere thegeneri operation alls havebeeninstantiated.We ouldusethisviewtogeneratetestsfromastati sele tion riterion,su hasthe overage ofthestates,orof thetransitionsof thisview.These tests would besymboli tests,in the shape ofasequen eofoperation alls,providedwithsymboli valuesoftheirparameters.Theywouldhavetobe valuatedafterwardsfromthedetailedbehavioralmodel.We ouldalsousetheabstra tionsyn hronizedwith theTPasareferen emodeltoevaluatethetests overage.Thisapproa hraisestwote hnologi al hallenges. Ononehand,itisne essarytohaveatimeeÆ ientte hnologyofabstra tion,that anbeappliedinpra ti e.
sear h of avaluation of thesymboli tests will haveto nd sub-sequen esof operationsto insert between twosymboli alls.Butthis y le ombinationsear hishighly ombinatorial.Thus,theissuewillbeto nd in omplete,butpra ti allyeÆ ient,sear hte hniques.Thismeanste hniquesthatprovidereasonablygood overageratesfortheexamplestreated.
Referen es
[Abr96℄ Jean-RaymondAbrial.TheBBook:AssigningProgramstoMeanings.CambridgeUniversityPress,1996. [ADX01℄ P.Amman,W.Ding,andD.Xu.Usingamodel he kertotestsafetyproperties.InICECCS'01.IEEEComputer
So iety,2001.
[BDGJ06℄ F.Bouquet,F.Dadeau,J.Groslambert,andJ.Julliand. SafetypropertydriventestgenerationfromJML spe i- ations. InFATES/RV'06,volume4262ofLNCS,pages225{239.Springer,2006.
[BJK +
05℄ M. Broy,B.Jonsson, J.-P.Katoen, M. Leu ker, and A. Prets hner, editors. Model-Based Testing of Rea tive Systems,volume3472ofLNCS. Springer,2005.
[BLLP04℄ E.Bernard,B.Legeard,X.Lu k,andF.Peureux. Generationoftest sequen esfromformalspe i ations:GSM 11-11standard asestudy. Software:Pra ti eandExperien e,34(10):915{948,2004.
[BPS05℄ D.Bert,M.-L.Potet,andN.Stouls. Genesyst:atooltoreasonaboutbehavioralaspe tsofBeventspe i ations. InZB'05,volume3455ofLNCS,2005.
[CIVDP07℄ J.Calame,N.Ioustinova,andJ.VanDePol.Automati model-basedgenerationofparameterizedtest asesusing dataabstra tion.ENTCS,191:25{48,2007.
[CJMR07℄ C.Constant, T.Jeron,H.Mar hand,and V.Rusu. Integratingformalveri ationand onforman e testingfor rea tivesystems.IEEETransa tionsonSoftwareEngineering,33(8):558{574,August2007.
[CLP04℄ S.Colin,B.Legeard,andF.Peureux. Preamble omputationinautomatedtest asegenerationusingConstraint Logi Programming.TheJournal ofSoftwareTesting,Veri ationandReliability,14(3):213{235,2004.
[DJM08℄ J. Dubreil, T. Jeron, and H. Mar hand. Automati test generation for se urityproperties. Deliverable L3.3, INRIA/IRISAVerte sProje t,2008. PolitessProje t,ANR-05-RNRT-01301.
[EFHP02℄ E.E.Far hi,A.Hartman,andS.S.Pinter.Usingamodel-basedtestgeneratortotestforstandard onforman e. IBMSystemsJournal,41(1):89{110,2002.
[FTW05℄ L.Frantzen,J.Tretmans,andT.A.C.Willemse.Testgenerationbasedonsymboli spe i ations.InJ.Grabowski andB.Nielsen,editors,FATES2004,FormalApproa hestoSoftwareTesting,volume3395ofLNCS,pages1{15. Springer,2005.
[GHN93℄ J.Grabowski,D.Hogrefe,andR.Nahm.Test asegenerationwithtestpurposespe i ationbyMSCs.InSDL'93 -UsingObje ts,O tober1993.
[GIX04℄ GIXEL. Common IAS Platform for eAdministration, Te hni al Spe i ations, 1.01 Premium edition, 2004. http://www.gixel.fr.
[JH07℄ R.JoshiandG.Holzmann.Amini hallenge:buildaveri able lesystem.FormalAspe tsofComputing,19(2):269{ 272,June2007.
[JJ05℄ C.Jardand T.Jeron. TGV: theory,prin iplesand algorithms. Software Toolsfor Te hnology Transfert, 7(1), 2005.
[JJRZ05℄ T. Jeannet, T. Jeron,V. Rusu, and E. Zinovieva. Symboli test sele tion based on approximate analysis. In TACAS'05,volume3440ofLNCS,pages349{364.Springer,2005.
[JL07℄ E.Ja uelandB.Legeard.LEIRIOSTestGenerator:AutomatedtestgenerationfromBmodels.InB'2007,volume 4355ofLNCS,pages277{280.Springer,2007.
[JMT08a℄ J.Julliand,P.-A.Masson,andR.Tissot.Generatingse uritytestsinadditiontofun tionaltests.InAST'08,3rd Int.workshoponAutomationofSoftwareTest,pages41{44,Leipzig,Germany,May2008.ACMPress.
[JMT08b℄ J.Julliand,P.-A.Masson,andR.Tissot. GeneratingtestsfromBspe i ations andtestpurposes. InABZ'08, Int.Conf.onASM,BandZ,volume5328ofLNCS,pages139{152,London,UK,September2008.Springer. [MJP
+
07℄ P.-A.Masson,J.Julliand,J.-C.Plessis,E.Ja uel,andG.Debois.Automati generationofmodel-basedtestsfor a lassofse urityproperties. InA-MOST'07,pages12{22.ACMPress,2007.
[PMLT08℄ A.Prets hner,T.Mouelhi,andY.LeTraon.Model-basedtestsfora ess ontrolpoli ies.InICST'08,Int.Conf. onSoftwareTesting,Veri ation,andValidation,pages338{347.IEEEComputerSo iety,2008.
[SLB05℄ M.Satpathy,M.Leus hel,andM.Butler.ProTest:Anautomati testenvironmentforBspe i ations.InMBT'04, volume111ofENTCS,pages113{136,2005.
[SML06℄ M.Satpathy,Q.-A.Malik,andJ.Lilius. Synthesisofs enariobasedtest asesfromBmodels.InFATES/RV'06, volume4262ofLNCS,pages133{149.Springer,2006.
[TSL04℄ L.Tan,O.Sokolsky,andI.Lee. Spe i ation-basedtesting withlineartemporallogi . InIRI'2004, IEEEInt. Conf.onInformationReuseandIntegration,pages413{498,November2004.
