Model-Based Testing of GUI Applications Featuring Dynamic Instanciation of Widgets

(1)

HAL Id: hal-03099199

https://hal.archives-ouvertes.fr/hal-03099199

Submitted on 7 Jan 2021

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

Model-Based Testing of GUI Applications Featuring

Dynamic Instanciation of Widgets

Alexandre Canny, Philippe Palanque, David Navarre

To cite this version:

(2)

0RGHO%DVHG7HVWLQJRI*8,$SSOLFDWLRQV)HDWXULQJ

'\QDPLF,QVWDQFLDWLRQRI:LGJHWV

$OH[DQGUH&DQQ\ ICS-IRIT

Université Toulouse III Paul Sabatier 7RXORXVH)UDQFH

DOH[DQGUHFDQQ\#LULWIU

3KLOLSSH3DODQTXH ICS-IRIT

SDODQTXH#LULWIU

'DYLG1DYDUUH ICS-IRIT

QDYDUUH#LULWIU

Abstract²7KHWHVWLQJRIDSSOLFDWLRQVZLWKD*UDSKLFDO8VHU ,QWHUIDFH *8, LV D FRPSOH[ DFWLYLW\ EHFDXVH RI WKH LQILQLW\ RI SRVVLEOHHYHQWVHTXHQFHV,QWKHILHOGRI*8,7HVWLQJPRGHOEDVHG DSSURDFKHVEDVHGRQUHYHUVHHQJLQHHULQJRI*8,DSSOLFDWLRQKDYH EHHQ SURSRVHG WR JHQHUDWH WHVW FDVHV 8QIRUWXQDWHO\ HYLGHQFHV VKRZWKDWWKHVHWHFKQLTXHVGRQRWVXSSRUWVRPHRIWKHIHDWXUHVRI PRGHUQ*8,DSSOLFDWLRQV7KHVHIHDWXUHVLQFOXGHG\QDPLFZLGJHWV LQVWDQWLDWLRQ RU DGYDQFHG LQWHUDFWLRQ WHFKQLTXHV HJ PXOWL WRXFK ,Q WKLV SDSHU ZH SURSRVH WR EXLOG PRGHOV RI WKH DSSOLFDWLRQV IURP UHTXLUHPHQWV DV LW LV VWDQGDUG SUDFWLFH LQ 0RGHO%DVHG 7HVWLQJ 7R GR VR ZH LGHQWLILHG ,&2 ,QWHUDFWLYH &RRSHUDWLYH2EMHFWDVRQHRIWKHPRGHOOLQJWHFKQLTXHVDOORZLQJ WKHGHVFULSWLRQRIFRPSOH[*8,EHKDYLRU:HGHPRQVWUDWHWKDW WKLVQRWDWLRQLVVXLWDEOHIRUJHQHUDWLQJWHVWFDVHVWDUJHWLQJFRPSOH[ *8,DSSOLFDWLRQVLQDSURFHVVGHULYHGIURPWKHVWDQGDUG0RGHO %DVHG7HVWLQJSURFHVV

Keywords—GUI Testing, Model-Based Testing, User Interface Description Languages

, ,1752'8&7,21

7KHWHVWLQJRIDSSOLFDWLRQVZLWKD*UDSKLFDO8VHU,QWHUIDFHLV NQRZQ WR EH D FRPSOH[ DFWLYLW\ >@ HVSHFLDOO\ GXH WR WKH XQSUHGLFWDELOLW\RIWKHKXPDQEHKDYLRUDVZHOODVWRWKHLQILQLWH QXPEHU RI SRVVLEOH HYHQW VHTXHQFHV >@ 7R FRSH ZLWK WKHVH FKDOOHQJHV PRGHOEDVHG WHVWLQJ WHFKQLTXHV KDYH EHHQ GHYHORSHG WR WU\ WR JHQHUDWH UHOHYDQW WHVW VHTXHQFHV ZLWKRXW UHO\LQJ RQ PDQXDOVFULSWLQJ RU FDSWXUH DQG UHSOD\ RI WHVWHU¶V LQWHUDFWLRQV7KHVHPRGHOEDVHGWHFKQLTXHVPRVWO\UHOLHVRQWKH JHQHUDWLRQRIWHVWFDVHVIURPPRGHOVEXLOWG\QDPLFDOO\ZKLOH H[SORULQJWKHH[HFXWLRQ VSDFHRIWKH $SSOLFDWLRQ8QGHU7HVW $87 >@ LH PRGHOV EXLOW E\ UHYHUVH HQJLQHHULQJ 8QIRUWXQDWHO\HYLGHQFHVWKDWWKHVHDSSURDFKHVWHQGWREHOHVV HIIHFWLYH WKDQ OHVV VWUXFWXUHG RQHV ZLWK D VLJQLILFDQW UDQGRP DVSHFWLHUDQGRPDQGV\VWHPDWLFWHFKQLTXHV>@KDVEHHQ IRXQG IRU ERWK PRELOH >@ DQG GHVNWRS DSSOLFDWLRQV >@ )XUWKHUPRUHHYLGHQFHVWKDWWKHVHDSSURDFKHVGRHVQRWHQDEOH WKHWHVWLQJRILQFUHDVLQJO\FRPSOH[*8,DSSOLFDWLRQVKDYHDOVR EHHQSURYLGHGIRULQVWDQFHEHFDXVHWKH\GRQRWVXSSRUWDGKRF DQG QRQVWDQGDUG ZLGJHWV >@ RU DGYDQFHG LQWHUDFWLRQ WHFKQLTXHV>@>@HJPXOWLWRXFKRQHV ,ILQ0RGHO%DVHGDSSURDFKHVIRU*8,7HVWLQJPRGHOVDUHEXLOW G\QDPLFDOO\ZKLOHH[SORULQJWKHH[HFXWLRQVSDFHRIWKH$87 WKHSURFHVVRI0RGHO%DVHG7HVWLQJDVGHILQHGE\8WWLQJHWDO >@SURSRVHVWREXLOGWKHPRGHOVIURPWKHUHTXLUHPHQWVRIWKH DSSOLFDWLRQ2QHRIWKHPDLQFKDOOHQJHLQ0RGHO%DVHG7HVWLQJ LV WKDW LW UHTXLUHV PRGHOOLQJ WHFKQLTXHV WKDW DUH H[SUHVVLYH HQRXJK WR GHVFULEH WKH EHKDYLRU RI WKH $87 ,QFUHDVLQJO\ FRPSOH[*8,DSSOLFDWLRQVPD\IHDWXUHVPXOWLPRGDOLQWHUDFWLRQ WHFKQLTXHVHJ³SXWWKDWWKHUH´YRLFHJHVWXUHLQWHUDFWLRQ>@ G\QDPLFLQVWDQWLDWLRQRILQSXWGHYLFHVHJDGGLQJILQJHUVRQD WRXFKVFUHHQVFDOLQJG\QDPLFLQVWDQWLDWLRQRI*8,ZLGJHWV HWF(DFKRIWKHVHIHDWXUHVUDLVHVFKDOOHQJHVIRUERWKPRGHOOLQJ DQGWHVWLQJDQGWKLVSDSHUIRFXVHVRQWKHFKDOOHQJHVUHODWHGZLWK 0%7 RI DSSOLFDWLRQV IHDWXULQJ G\QDPLF LQVWDQWLDWLRQ RI *8, ZLGJHWV 7R RYHUFRPH WKHVH FKDOOHQJHV ZH LQYHVWLJDWHG WKH H[SUHVVLYHQHVV RI VHYHUDO 8VHU ,QWHUIDFH 'HVFULSWLRQ /DQJXDJHV 8',/V WR HVWDEOLVK ZKLFK RQHV DUH H[SUHVVLYH HQRXJK WR DOORZ WKH JHQHUDWLRQ RI WHVW FDVHV IRU VXFK DSSOLFDWLRQV :H LGHQWLILHG ,&2 ,QWHUDFWLYH &RRSHUDWLYH 2EMHFW DV WKH EHVW ILW IRU WKLV SXUSRVH :H LOOXVWUDWH WKH SRVVLELOLWLHV RIIHUHG E\ WKLV QRWDWLRQ E\ PRGHOOLQJ DQ $LU 7UDIILF &RQWURO $SSOLFDWLRQ $GGLWLRQDOO\ ZH SURSRVH D SURFHVVGHULYHGIURP>@WRHQDEOHERWKWKHLPSOHPHQWDWLRQ DQGWKHWHVWLQJRID*8,DSSOLFDWLRQIURPWKH,&2PRGHOV7KLV SURFHVVUHOLHVRQUDQGRPFRQVWUDLQHGVLPXODWLRQRIWKHPRGHOV WR EXLOG UHOHYDQW WHVW FDVHV ZKLOH DOORZLQJ WR FRSH ZLWK WKH SRWHQWLDOO\ LQILQLWH QXPEHU RI HYHQW VHTXHQFHV WKDW FDQ EH JHQHUDWHG

7KLV SDSHU LV VWUXFWXUHG DV IROORZ VHFWLRQ ,, SUHVHQWV WKH FKDOOHQJHV LQ *8, WHVWLQJ DQG WKH FXUUHQW VWDWH RI WKH DUW LQ PRGHOOLQJRI*8,DSSOLFDWLRQV6HFWLRQ,,,SUHVHQWVWKH $7& DSSOLFDWLRQZHXVHDVDFDVHVWXG\DQGVHFWLRQ,9SUHVHQWV,&2 WKHQRWDWLRQZHLGHQWLILHGDVWKHPRVWH[SUHVVLYHIRUGHVFULELQJ *8, DSSOLFDWLRQV IHDWXULQJ G\QDPLF LQVWDQWLDWLRQ RI *8, ZLGJHWV 6HFWLRQ 9 GHPRQVWUDWHV WKH DELOLW\ WR PRGHO WKH EHKDYLRUDQGWKHSUHVHQWDWLRQRIWKH$7&DSSOLFDWLRQZLWK,&2 DQGVHFWLRQ9,SUHVHQWVRXUSURFHVVIRUEXLOGLQJDQGWHVWLQJWKH $87IURP,&2PRGHOV8OWLPDWHO\VHFWLRQ9,,FRQFOXGHVWKLV SDSHULQDGGLWLRQWRGLVFXVVLQJIXWXUHZRUN ,, &+$//(1*(6$1'&855(1767$7(2)7+($57,17(67,1* $1'02'(//,1*2)*8,$33/,&$7,216

(3)

HQFRXQWHUHGLQ*8,WHVWLQJEHIRUHGLVFXVVLQJKRZH[SUHVVLYHD GHVFULSWLRQWHFKQLTXHPXVWEHWRHQDEOHWHVWFDVHVJHQHUDWLRQ IURPPRGHOVEXLOWSULRUWRGHYHORSPHQW

A. Challenges and Current State of the Art in GUI Testing *8,7HVWLQJLVGHILQHGE\%DQHUMHHHWDODVWKHWHVWLQJRIDQ DSSOLFDWLRQ WKDW KDV D *UDSKLFDO 8VHU ,QWHUIDFH VROHO\ E\ SHUIRUPLQJVHTXHQFHRIHYHQWVHJFOLFNRQEXWWRQHQWHUWH[W RQ *8, ZLGJHWV HJ EXWWRQV WH[WILHOGV >@ 7KH NH\ LVVXH KHUHLVWKDWWKHVSDFHRIDOOSRVVLEOHHYHQWVHTXHQFHVWKDWPD\ EHH[HFXWHGLVH[WUHPHO\ODUJHLQSULQFLSOHLQILQLWH >@7KXV WHVWFDVHJHQHUDWLRQLVPDGHGLIILFXOWDQGPDQXDOO\JHQHUDWHG WHVWVFULSWVXVXDOO\RIIHUVOLWWOHFRYHUDJHRIWKHHYHQWVHTXHQFHV >@ DV LW ZRXOG EH WRR ORQJ DQG FRVWO\ IRU WHVWHUV WR FRPSUHKHQVLYHO\ H[SORUH WKH H[HFXWLRQ VSDFH RI WKH $87 7KHVHPDQXDOWHVWVFULSWVDUHXVXDOO\SURGXFHGXVLQJVFULSWLQJ ODQJXDJH HJ $EERW _{6HOHQLXP :HE 'ULYHU} _RU

&DSWXUH5HSOD\WHFKQLTXHVZKHUHXVHULQWHUDFWLRQDUHORJJHG IRU EHLQJ UHSOD\HG ODWWHU LQ UHJUHVVLRQ WHVWLQJ HJ *8,&DW >@7HVW$XWRPDWLRQ);

$XWRPDWLF*8,7HVWLQJWHFKQLTXHVDLPVDWVROYLQJVRPHRIWKH LVVXHVEURXJKWE\PDQXDO*8,WHVWLQJDSSURDFKHVE\RIIHULQJ EHWWHU FRYHUDJH RI WKH HYHQW VHTXHQFHV WKURXJK DXWRPDWHG H[SORUDWLRQRIWKHH[HFXWLRQVSDFH9DULRXVWHFKQLTXHVHQDEOHV DXWRPDWLF*8,WHVWLQJVXFKDV>@

x 5DQGRP:DON HJ 7HVWDU5DQGRP >@ 8,$SSOLFDWLRQ([HUFLVHU0RQNH\_{ZKLFKUDQGRPO\}

SOD\V HYHQWV RQ DYDLODEOH *8, ZLGJHWV DQG LV SDUWLFXODUO\ UHOHYDQW IRU FUDVKWHVWLQJ PDNLQJ VXUH WKDW QR HYHQW VHTXHQFH OHDG WR WKH $87 VWRS UHVSRQGLQJFUDVKHV x 0RGHO%DVHG7HFKQLTXHVHJ$%7>@*8,7$5 >@ 7HVWDU4/($51,1* >@ WKDW GULYH WKH JHQHUDWLRQRIWHVWFDVHVZLWKDPRGHORIWKH*8,RIWKH $870RGHOEDVHGWHFKQLTXHVXVHJUDSKPRGHOVEXLOW G\QDPLFDOO\ ZKLOH H[SORULQJ WKH H[HFXWLRQ VSDFH RI WKH$87>@

7KHODWHUDSSURDFKLVUDWKHURULJLQDODV0RGHO%DVHG7HVWLQJ WHFKQLTXHVXVXDOO\UHOLHVRQPRGHOVEXLOWIURPWKHUHTXLUHPHQWV VSHFLILHGIRUWKH$87>@7KLVPD\H[SODLQZK\HYHQWKRXJK 0RGHO%DVHG7HVWLQJLVRIWHQSUDLVHGIRULWVHIIHFWLYHQHVV*8, 7HVWLQJ XVLQJ 0RGHO%DVHG WHFKQLTXHV LV OHVV HIIHFWLYH WKDQ UDQGRP WHFKQLTXHV >@>@ 8QIRUWXQDWHO\ PRVW RI WKH H[LVWLQJ*8,DSSOLFDWLRQV ZHUHQRWEXLOWIURP PRGHOV7KXV WKHVH WHFKQLTXHV PDNHV VHQVH HVSHFLDOO\ LI ZH FRQVLGHU WKDW EXLOGLQJPRGHOVDOORZVWRUHDVRQRQWKHDSSOLFDWLRQEHKDYLRU <HW ZKLOH PRGHOV RI DSSOLFDWLRQV VWLOO DUH QRW WKH QRUP WKH 8VHU&HQWHUHG'HVLJQSURFHVVLQ+XPDQ&RPSXWHU,QWHUDFWLRQ +&,LQYROYHVWKHSURGXFWLRQRIPRGHOVRIWKHXVHULQWHUDFWLRQV ZLWK WKH DSSOLFDWLRQV FDOOHG 7DVN0RGHOV 7KHVH PRGHOV DUH PRUHOLNHO\WRH[LVWIRUPRGHUQDSSOLFDWLRQVDQGWHFKQLTXHVKDV EHHQSURSRVHGIRUEXLOGLQJWHVWFDVHVIURPWKHP>@%H\RQG +&, FXUUHQW WUHQGV WRZDUGV SXVKLQJ 0RGHO%DVHG 6RIWZDUH

_{KWWSDEERWVRXUFHIRUJHQHWGRFRYHUYLHZVKWPO} _{KWWSVVHOHQLXPGHYGRFXPHQWDWLRQHQZHEGULYHU} _{KWWSZZZWHVWDXWRPDWLRQI[FRP}

_{KWWSVGHYHORSHUDQGURLGFRPVWXGLRWHVWPRQNH\}

(QJLQHHULQJ 0%6( LQ 6RIWZDUH (QJLQHHULQJ >@ PDNHV LW FRQFHLYDEOHWKDWLQDIRUHVHHDEOHIXWXUHPRUHDQGPRUH*8, DSSOLFDWLRQVZLOOEHEXLOWIURPVRPHVRUWRIPRGHOEXWZKLFK" $GYDQFHG *8, DSSOLFDWLRQV IHDWXUHV EHKDYLRU WKDW FDQQRW EH GHVFULEHG >@>@ XVLQJ WKH PRGHOOLQJ WHFKQLTXHV IRXQG LQ *8,7HVWLQJ>@7KHVHEHKDYLRUVLQFOXGHVRQHVEDVHGRQQRQ VWDQGDUG_{*8,ZLGJHWVHJGUDZLQJDUHDRIDGUDZLQJWRRO}

G\QDPLFLQVWDQWLDWLRQRI*8,ZLGJHWVRUDFWLRQVWLHGWRPXOWLSOH HYHQWV VRXUFHV HJ PXOWLSOH ILQJHUV GXULQJ PXOWLWRXFK LQWHUDFWLRQ 7KH IROORZLQJ VXEVHFWLRQ GLVFXVVHV WKH VWDWH RI WKH DUW LQ GHVFULELQJ *8, DSSOLFDWLRQV WR KHOS LQ LGHQWLI\LQJ PRGHOOLQJ WHFKQLTXHV WKDW VXSSRUW WKH GHVFULSWLRQ RI WKHVH EHKDYLRUV WR HQDEOH WHVW FDVHV JHQHUDWLRQ ZLWK IRFXV RQ G\QDPLFLQVWDQWLDWLRQRI*8,ZLGJHWV

B. Modelling of GUI Applications

0RGHO%DVHG 7HVWLQJ 0%7 RI VRIWZDUH UHOLHV RQ H[SOLFLW EHKDYLRU PRGHOV RI D V\VWHP WR GHULYH WHVW FDVHV >@ 7KH FRPSOH[LW\RIGHULYLQJFRPSUHKHQVLYHWHVWFDVHVLQFUHDVHVZLWK WKHFRPSOH[LW\RIWKH$87ZLWKFRPSOH[EHKDYLRUVFRPHWKH QHHGIRUPRGHOOLQJWHFKQLTXHVH[SUHVVLYHHQRXJKWRGHVFULEH WKHFRPSOH[LW\RIWKH$87EHKDYLRU>@$Q\EHKDYLRUWKDW FDQQRWEHGHVFULEHGFDQQRWKDYHWHVWFDVHVJHQHUDWHGIRUXVLQJ 0%7 7KH WHFKQLTXHV IRU GHVFULELQJ WKH EHKDYLRU RI *8, DSSOLFDWLRQV DUH FDOOHG 8VHU ,QWHUIDFH 'HVFULSWLRQ /DQJXDJHV 8,'/,Q>@+DPRQHWDOLGHQWLILHGVHYHUDOFKDUDFWHULVWLFV 8,'/VPXVWKDQGOHWRHQDEOHWKHFRPSUHKHQVLYHGHVFULSWLRQRI XVHU LQWHUIDFHV EHKDYLRUV 7KHVH FKDUDFWHULVWLFV LQFOXGH VRPH FRPPRQWRDOO*8,DSSOLFDWLRQVZKLOHRWKHUDSSOLHVVSHFLILFDOO\ WR DSSOLFDWLRQV IHDWXULQJ G\QDPLF LQVWDQWLDWLRQ RI ZLGJHWV PXOWLPRGDOLQWHUDFWLRQHWF$VWKHIRFXVRIWKLVSDSHULVRQWKH WHVWLQJRIDSSOLFDWLRQVIHDWXULQJG\QDPLFLQVWDQWLDWLRQRI*8, ZLGJHWV ZH LQYHVWLJDWHG WKH IROORZLQJ VXEVHW RI WKH FKDUDFWHULVWLFVIURP>@ x 'DWDGHVFULSWLRQ IRUGHVFULELQJWKHREMHFWVDQGYDOXHV VRLQIRUPDWLRQVXFKDVZLGJHWORFDWLRQDWDJLYHQWLPH LVNQRZ x 6WDWH(YHQWUHSUHVHQWDWLRQDVFRPPRQO\GHVFULEHG LV0RGHO%DVHGWHFKQLTXHVIRU*8,7HVWLQJ>@

x 4XDOLWDWLYH WLPH EHWZHHQ WZR FRQVHFXWLYH PRGHO

HOHPHQWVDLPVDWUHSUHVHQWLQJRUGHULQJRIDFWLRQVVXFK DVSUHFHGHQFHVXFFHVVLRQDQGVLPXOWDQHLW\

x 4XDQWLWDWLYH WLPH EHWZHHQ WZR FRQVHFXWLYH PRGHO

HOHPHQWV UHSUHVHQWV EHKDYLRUDO WHPSRUDO HYROXWLRQV UHODWHGWRDJLYHQDPRXQWRIWLPHXVXDOO\H[SUHVVHG LQPLOOLVHFRQGV

x &RQFXUUHQW EHKDYLRU UHSUHVHQWDWLRQ LV QHFHVVDU\

ZKHQ WKH LQWHUDFWLYH V\VWHPV IHDWXUH PXOWLPRGDO LQWHUDFWLRQV

x '\QDPLFLQVWDQWLDWLRQRIZLGJHWVLVDFKDUDFWHULVWLF

UHTXLUHGIRUWKHGHVFULSWLRQRILQWHUIDFHVZKHUHREMHFWV DUHQRWDYDLODEOHDWWKHFUHDWLRQRIWKHLQWHUIDFHDVIRU

_{:H FDOO QRQVWDQGDUG WKH *8, ZLGJHWV WKDW ZHUH EXLOW}

(4)

LQVWDQFHLQGHVNWRSOLNHLQWHUIDFHVZKHUHQHZLFRQV DUHFUHDWHGDFFRUGLQJWRXVHUDFWLRQV

7DEOH GHWDLOV WKH H[SUHVVLYHQHVV RI 8,'/V UHJDUGLQJ WKHVH FKDUDFWHULVWLFV)RUDOOFKDUDFWHULVWLFVWKHUHDUHWKUHHSRVVLEOH YDOXHV x <HVPHDQVWKDWWKDWFKDUDFWHULVWLFLVH[SOLFLWO\KDQGOHG E\WKH8,'/ x 1R PHDQV WKDW WKH FKDUDFWHULVWLF LV QRW H[SOLFLWO\ KDQGOHG x &RGHPHDQVWKDWWKHFKDUDFWHULVWLFLVPDGHH[SOLFLWEXW RQO\DWWKHFRGHOHYHODQGLVWKXVQRWDFRQVWUXFWRIWKH 8,'/

Table 1. UIDL expressiveness for the main characteristics of modern application (adapted from [17]).

'XHWRVSDFHFRQVWUDLQWZHZRQ¶WGHWDLORQWKHKDQGOLQJRIDOO WKHVHFKDUDFWHULVWLFVE\DOOWKH8,'/VZHSUHVHQW<HWWKDQNV WRDQDQDO\VLVEDVHGRQFDWHJRULHVRI8,'/VZHFDQREVHUYH WKDWVWDWHEDVHGDQGIORZEDVHGDSSURDFKDOVRXVHGLQ0%7RI *8, DSSOLFDWLRQV DV VKRZQ LQ >@ DUH QRW VXLWDEOH IRU DQ HIIHFWLYH GHVFULSWLRQ RI DGYDQFHG *8, DSSOLFDWLRQV $GGLWLRQDOO\ ZH REVHUYH WKDW 8,'/V EDVHG RQ 3HWUL 1HWV DXJPHQWHG ZLWK WHPSRUDO DVSHFWV WLPH ZDV LQWHQWLRQDOO\ DYRLGHGLQ3HWUL¶VRULJLQDOZRUN >@VXFKDV&31>@DQG ,&2>@DUHDPRQJVWWKH PRVWH[SUHVVLYH8,'/V<HWRQO\ ,&2 >@ DSSHDUV WR EH H[SUHVVLYH HQRXJK WR VXSSRUW WKH FKDUDFWHULVWLFV UHTXLUHG IRU FRPSUHKHQVLYHO\ GHVFULELQJ *8, DSSOLFDWLRQIHDWXULQJG\QDPLFLQVWDQWLDWLRQRI*8,ZLGJHWV,Q WKHIROORZLQJRIWKLVSDSHUZHZLOOLQWURGXFHWKH,&28,'/ DQG GHPRQVWUDWH WKDW LWV H[SUHVVLYHQHVV FRPELQHG ZLWK WRRO VXSSRUWHQDEOHVWHVWFDVHVJHQHUDWLRQIRUVXFKDSSOLFDWLRQVRQ DQH[DPSOHEDVHGRQDQ$LU7UDIILF&RQWURO$SSOLFDWLRQ ,,, (;$03/(2)$1$33/,&$7,21)($785,1*'<1$0,& ,167$1&,$7,212)*8,:,'*(767+($,575$)),& &21752/6<67(0 7KHFDVHVWXG\LQWKLVSDSHUIRFXVHVRQWKH*8,DVSHFWVRIDQ $LU 7UDIILF &RQWURO DSSOLFDWLRQ 1RZDGD\V WKH DLUVSDFH LV GLYLGHG LQ VHFWRUV HDFK RI WKHP EHLQJ FRQWUROOHG E\ WZR DLU WUDIILF FRQWUROOHUV PDQDJLQJ GLIIHUHQWWDVNV DQG ZRUNLQJ LQ D FRRSHUDWLYHZD\7KHDLUWUDIILFFRQWUROOHUKDVDWKLVGLVSRVDOD

ZRUNVWDWLRQ IRU KDQGOLQJ WKH WUDIILF RYHU D JLYHQ VHFWRU E\ FRPPXQLFDWLQJZLWKWKHSLORWVLQWKHSODQHVFXUUHQWO\IO\LQJLQ WKH VHFWRU 7KLV VHFWLRQ SUHVHQWV DQ RYHUYLHZ RI WKH $7& DSSOLFDWLRQDQGWKHUHTXLUHPHQWVLWPXVWIXOILOOV

A. Overview of the ATC Application

)URPKLVKHUSRVLWLRQDQDLUWUDIILFFRQWUROOHUFDQLQWHUDFWZLWK DLUFUDIWSLORWVXVLQJWZRPHDQVYRLFHDQGGDWDOLQNPHVVDJHV )RUWKH$LU7UDIILF&RQWUROOHUWKH$7&$SSOLFDWLRQVHH)LJ LVWKHHQWU\SRLQWIRUHYHU\H[FKDQJHVKHZDQWVWRLQLWLDWHZLWK WKH SLORW 7KH DLU WUDIILF FRQWUROOHUV PD\ UHTXHVW WKH SLORW WR FRQWDFWWKHPXVLQJWKHUDGLR92,&(FRPPDQGRUVHQGWKHP FRPPDQGWH[WPHVVDJHWKURXJKGDWDOLQNWRFKDQJHWKHFRXUVH RIWKHLUIOLJKWV7KHVHFRPPDQGVDUHWKHIROORZLQJDQGDUHDOO IROORZHGE\DSDUDPHWHUYDOXH x )5(4IRUDVNLQJWKHSLORWWRVZLWFKIURPRQHUDGLR IUHTXHQF\WRDQRWKHURQH x &)/IRUQRWLI\LQJWKHQHZ &OHDUHG)OLJKW/HYHOIRU WKHIOLJKWLHWKHDOWLWXGHLWLVDOORZHGWRIO\DW x 63(('IRUUHTXHVWLQJDVSHHGFKDQJH x +($'IRUUHTXHVWLQJDQKHDGLQJFKDQJH x %($&21IRUUHTXHVWLQJDFKDQJHRIWKHURXWHRIWKH SODFHE\GHVLJQDWLQJWKHQH[WEHDFRQLWKDVWRIO\RYHU

Fig. 1. The ATC Application with the pop-up menu for flight DLH5629 opened and SPEED sub-menu selected.

7KH*8,RIWKH$7&DSSOLFDWLRQ)LJFDQEHVSOLWLQWKUHH x WKH OLJKW JUH\ SDUW ZKLFK FRUUHVSRQG WR WKH VHFWRU

KDQGOHG E\ WKH FRQWUROOHU ,Q WKLV VHFWRU ZKLWH OLQH UHSUHVHQW SODQH URXWHV 2Q WKHVH URXWHV SODQHV DUH UHSUHVHQWHGE\DVXFFHVVLRQRIZKLWHGRWVWKHILUVWRQH EHLQJWKHDFWXDOSRVLWLRQRIWKHSODQHDQGWKHVPDOOHU RQHV LWV SUHYLRXV SRVLWLRQV $ORQJ ZLWK WKHVH GRWV LQIRUPDWLRQ FRQFHUQLQJ WKH IOLJKW DUH GLVSOD\HG WKH FDOOVLJQRIWKH IOLJKWLWV VSHHGLWVKHDGLQJDQGWKH QH[WEHDFRQLWLVVXSSRVHGWRIO\RYHU

x WKHGDUNJUH\SDUWZKLFKUHSUHVHQWDOOWKHRXWVLGHRI WKHFRQWUROOHGVHFWRU

(5)

o 2QWKHOHIWVLGHWKHGLUHFWFRPPDQGPHQXQR SDUDPHWHU LV QHHGHG ZKLFK RIIHUV WKUHH FRPPDQGV6(1'DQG$%257IRUVHQGLQJ

RUFDQFHOOLQJWKHFXUUHQWGDWDOLQNFRPPDQG DQG92,&(IRUDVNLQJWKHSLORWVWRFDOOWKH

FRQWUROOHU XVLQJ WKH UDGLR VHQGLQJ WKH UHTXHVW LV VWLOO UHTXLUHG RQFH 92,&( LV VHOHFWHG

o 2QWKHULJKW VLGHRI WKH PHQXWKHXVHUFDQ VHOHFW RQH RI ILYH FRPPDQGV ZKLFK QHHG SDUDPHWHU )5(4 &)/ 63((' +($'

DQG%($&21 (DFK WLPH WKH FRQWUROOHU

VHOHFWVRQHRIWKHVHFRPPDQGVDQRWKHUSRS XSPHQXDSSHDUVWKDWDOORZVIRUHQWHULQJWKH SDUDPHWHURIWKHFRPPDQG%\SUHVVLQJ&1& FDQFHO WKLV SRSXS PHQX LV FORVHG E\ VHOHFWLQJDYDOXHWKHSDUDPHWHULVVHW7KHDLU WUDIILF FRQWUROOHU FDQ FKRRVH DPRQJVW DGGLWLRQDO YDOXHV E\ VFUROOLQJ DPRQJVW WKH DYDLODEOHRQHVXVLQJWKHDUURZVDWWKHWRSDQG ERWWRPRIWKHYDOXHOLVW

B. Requirements for the ATC Application

7KHIROORZLQJVHWRIUHTXLUHPHQWVKDYHWREHIXOILOOHGE\WKH $7&DSSOLFDWLRQ

x $FRQWURORUGHULVRQO\UHFHLYHGE\RQHSODQH x $Q\UHTXHVWVHQWE\DFRQWUROOHU ZLOOEHUHFHLYHGDW

VRPHWLPHE\DSLORW

x $ FRQWURO RUGHU LV VHQW WR RQO\ RQH SODQH RQO\ RQH SODQHFDQEHVHOHFWHGDWDWLPH x )RUHDFKFRQWURORUGHURQO\RQHLQIRUPDWLRQLVVHQW x $OOFRQWURORUGHUVILQLVKE\HLWKHU$ERUWRU6HQG x ,WLVQRWSRVVLEOHWREXLOGVHYHUDORUGHUVDWDWLPH 7KHVHUHTXLUHPHQWVFDQEHFODVVLILHGLQWZRFDWHJRULHVKLJK OHYHOUHTXLUHPHQWVUHODWHGWRWKHYHU\ VHPDQWLFVRIDLUWUDIILF FRQWURODQGORZHUOHYHORQHVUHODWHGWRLQWHUDFWLRQWHFKQLTXHV ,QWKHIROORZLQJRIWKLVSDSHUZHZLOOGLVFXVVRQKRZWRPRGHO DQG WHVW VRPH RI WKHVH UHTXLUHPHQWV RQ *8, ZLGJHWV UHSUHVHQWLQJWKHDLUFUDIWVWKDWDUHWKHG\QDPLFDOO\LQVWDQWLDWHG ZLGJHWVLQWKLVDSSOLFDWLRQ

,9 7+(,&286(5,17(5)$&('(6&5,37,21/$1*8$*( %\DQDO\]LQJWKHH[SUHVVLYHQHVVRI8,'/VZHKDYHHVWDEOLVKHG WKDW ,&2 ,QWHUDFWLYH &RRSHUDWLYH 2EMHW LV RQH RI WKH PRVW UHOHYDQW IRU GHVFULELQJ WKH FKDUDFWHULVWLFV RI DGYDQFHG *8, DSSOLFDWLRQV,QWKLVVHFWLRQZHLQWURGXFH,&2DQGGHWDLOVKRZ LWHQDEOHVWKHGHVFULSWLRQRIWKHEHKDYLRURIWKH*8,

A. Informal Presentation of ICO

,&2V ,QWHUDFWLYH &RRSHUDWLYHV 2EMHFWV DUH D IRUPDO GHVFULSWLRQ WHFKQLTXH GHGLFDWHG WR WKH VSHFLILFDWLRQ RI LQWHUDFWLYH V\VWHPV ,&2 XVHV FRQFHSWV ERUURZHG IURP WKH REMHFWRULHQWHGDSSURDFKG\QDPLFLQVWDQWLDWLRQFODVVLILFDWLRQ HQFDSVXODWLRQ LQKHULWDQFH FOLHQWVHUYHU UHODWLRQVKLS WR GHVFULEH WKH VWUXFWXUDO RU VWDWLF DVSHFWV RI V\VWHPV DQG XVHV KLJKOHYHO3HWULQHWVWRGHVFULEHWKHLUG\QDPLFVRUEHKDYLRU7KH ,&2 QRWDWLRQ LV EDVHG RQ D EHKDYLRUDO GHVFULSWLRQ RI WKH

LQWHUDFWLYHV\VWHPXVLQJWKH&RRSHUDWLYHREMHFWVIRUPDOLVPWKDW GHVFULEHVKRZWKHREMHFWUHDFWVWRH[WHUQDOVWLPXOLDFFRUGLQJWR LWV LQQHU VWDWH 7KLV EHKDYLRU FDOOHG WKH 2EMHFW &RQWURO 6WUXFWXUH 2E&6 LV GHVFULEHG E\ PHDQV RI 2EMHFW3HWUL1HW 231$Q2E&6FDQKDYHPXOWLSOHSODFHVDQGWUDQVLWLRQVWKDW DUHOLQNHGZLWKDUFVDVZLWKVWDQGDUG3HWULQHWV$VDQH[WHQVLRQ WRWKHVHVWDQGDUGDUFV,&2DOORZVXVLQJWHVWDUFVDQGLQKLELWRU DUFV(DFKSODFHKDVDQLQLWLDOPDUNLQJUHSUHVHQWHGE\RQHRU VHYHUDOWRNHQVLQWKHSODFHGHVFULELQJWKHLQLWLDOVWDWHRIWKH V\VWHP7RFRQYH\WKHUHODWLRQVKLSEHWZHHQWKHHYROXWLRQRIWKH EHKDYLRUDO PRGHO DQG LWV LPSDFW RQ WKH *8, ,&2 XVHV UHQGHULQJDQGDFWLYDWLRQIXQFWLRQVW\LQJSODFHVDQGWUDQVLWLRQV WR SURSHUWLHVHYHQW KDQGOHUV RI WKH *8, ZLGJHWV 7KH ,&2 QRWDWLRQ LV IXOO\ VXSSRUWHG E\ D &$6( WRRO FDOOHG 3HW6KRS >@>@,&2KDVDOUHDG\EHHQDSSOLHGLQWKHILHOGRI$LU7UDIILF &RQWURO LQWHUDFWLYH DSSOLFDWLRQV >@ VSDFH FRPPDQG DQG FRQWURO JURXQG V\VWHPV >@ LQWHUDFWLYH PLOLWDU\ >@ RU FLYLO FRFNSLWV >@ ,Q SUHYLRXV ZRUN WKH QRWDWLRQ FDSDELOLW\ WR VXSSRUW WKH WHVWLQJ RI PXOWLWRXFK LQWHUDFWLRQ WHFKQLTXH KDV EHHQGHPRQVWUDWHG>@\HWDVPHQWLRQHGSUHYLRXVO\WHVWLQJ RILQWHUDFWLRQWHFKQLTXHVFRPHVZLWKLWVRZQVHWRIFKDOOHQJHV WKDWDUHRXWRIVFRSHRIWKLVSDSHU

B. Notation for the description of the behavioral part of the GUI

7KH,&2QRWDWLRQXVHVFRQFHSWVERUURZHGIURP3HWULQHWV)LJ SUHVHQWVDQH[DPSOHRIVXFK3HWULQHWZKLFKLVRQHRIWKH,&2 PRGHOZHXVHWRGHVFULEHWKH$7&DSSOLFDWLRQ,WLVPDGHRI

SODFHV RYDO VKDSHV HJ )OLJKWV WUDQVLWLRQV UHFWDQJXODU VKDSHVHJDGGLQJ)OLJKWDQGDUFV7KHPDUNLQJRIWKLV3HWUL

QHWLVWKHGLVWULEXWLRQRIWRNHQVQXPEHUHGFLUFOHVLQWKHSODFHV

LQ WKH GLIIHUHQW SODFHV DW D JLYHQ WLPH )LJ D DQG)LJ E SUHVHQWVWZRGLIIHUHQWPDUNLQJV$V,&2UHOLHVRQ2EMHFW3HWUL QHWVWRNHQVFDQFDUU\YDOXHVIURPJHQHULFW\SHVHJLQWIORDW WRFRPSOH[REMHFWVLQFOXGLQJLQVWDQFHVRIRWKHU2E&6HV)LJ GHWDLOVWKH PDUNLQJRIWKH SODFH³)OLJKW´ IRXQGLQ)LJD (DFKOLQHRIWKHWDEOHLQ)LJGHVFULEHVWRNHQV+HUHWKHUHDUH OLQHV ZLWK D PXOWLSOLFLW\ RI PHDQLQJ WKDWWKHUH DUH WKUHH WRNHQ LQ WKH SODFH HDFK FDUU\LQJ D GLIIHUHQW REMHFW ,Q WKDW SDUWLFXODU FDVH WKH REMHFWV FDUULHG DUH WKUHH LQVWDQFHV RI 2E&6HV FRUUHVSRQGLQJ WR WKH ,&2 GHVFULSWLRQ RI 6LPSOH)OLJKWV

Fig. 2. Marking of the place "Flights" of the model PlaneManager1.

7KHPDUNLQJRIDQ,&2FDQFKDQJHDIWHULQYRFDWLRQVRUILULQJ RI DUPHG WUDQVLWLRQV 7KHUH DUH WZR NLQGV RI WUDQVLWLRQV LQ

,&2VWDQGDUG WUDQVLWLRQV DQG HYHQW WUDQVLWLRQV 7KH\ DUH

(6)

LQFRPLQJDUFV_{)RULQVWDQFHLQ)LJWKH³WUDQVLWLRQ´FDQEH}

ILUHGLI³SODFH´FRQWDLQVDWRNHQFDUU\LQJWKHYDOXHLQLWLDOO\ LWFRQWDLQVRIWKHP1RWHWKDWRQO\RQHWRNHQLVFRQVXPHG ZKHQWKHWUDQVLWLRQLVILUHGVRVWHSDUHUHTXLUHGWRFRQVXPHU WKH WZR WRNHQV ZLWK YDOXH 1RWH WKDW LQ 3HWVKRS DUPHG WUDQVLWLRQ DUH UHSUHVHQWHG LQ SXUSOH QRQDUPHG RQHV LQ JUH\ VHH)LJ init) first firing) second firing)

Fig. 3. Example of a transition with a pre-condition. Initially, place0 contains 9 tokens, 2 of which carrying a value of 5.

(YHQWWUDQVLWLRQVGLIIHUIURPVWDQGDUGRQHVDVWKH\DUHILUHGLI

DQGRQO\LIWKH\DUHDUPHGDQGWKHHYHQWWKH\DUHOLVWHQLQJIRU KDVEHHQSURGXFHG(YHQWWUDQVLWLRQVDUHWKHRQHVUHOHYDQW IRU JHQHUDWLQJ HYHQW VHTXHQFHV IRU *8, 7HVWLQJ ,Q WKH

ERWWRPOHIWRI)LJEWKHWUDQVLWLRQ³VHQGLQJ&RPPDQG´LVDQ H[DPSOHRIHYHQWWUDQVLWLRQ1H[WWRLWVWLWOH³VHQG´LQGLFDWHV WKH QDPH RI WKH HYHQW WKH WUDQVLWLRQ LV OLVWHQLQJ IRU %HORZ GHWDLOVDUHSURYLGHGUHJDUGLQJWKLVHYHQWVXFKDVWKHH[SHFWHG VRXUFH³IURP´menuDQRWKHUPRGHOGHVFULELQJDPHQXWKH SDUDPHWHUV RI WKH HYHQW LI DQ\ LQ WKDW FDVH WKH command

_Note:_{$WUDQVLWLRQZLWKRXWLQFRPLQJDUFVDQGLWVSUHFRQGLWLRQ} VHWWR³WUXH´LVFRQVLGHUHGDUPHG VHOHFWHGLQWKHPHQXDQGWKHSUHFRQGLWLRQVRQWKHHYHQW$WWKH YHU\ERWWRPRIWKHWUDQVLWLRQVDUHLQYRFDWLRQVRQRWKHU2E&6HV ,QYRFDWLRQVHQDEOHVXQLFDVWDQGV\QFKURQRXVFRPPXQLFDWLRQ

UHSUHVHQWHG E\ PHWKRG FDOOV LQ ,&2 :KHQ DQ 2E&6 RIIHUV LQYRFDWLRQV WKH\ DUH HDFK PDSSHG LQWR D VHW RI WKUHH SODFHV UHSUHVHQWLQJWKUHHFRPPXQLFDWLRQSRUWVWKHLQYRFDWLRQLQSXW RXWSXWDQGH[FHSWLRQSRUWV)RULQVWDQFHLQ)LJDWKHSODFHV 6,3BDGG)OLJKW 623BDGG)OLJKW DQG 6(3BDGG)OLJKW DUH WKH LQSXW RXWSXW DQG H[FHSWLRQ SRUWV RI WKH PHWKRG DGG)OLJKW :KHQ WKLV PHWKRG LV FDOOHG D WRNHQ LV FUHDWHG KROGLQJ WKH SDUDPHWHUVRIWKHLQYRFDWLRQDQGLVSXWLQSODFH6,3BDGG)OLJKW 2E&6HV FDQ LQYRNH HDFK RWKHU IRU LQVWDQFH LQ WKH HYHQW WUDQVLWLRQ ³VHQGLQJ&RPPDQG´ WKH ERWWRP SDUW FRQWDLQV LQYRFDWLRQV RQ ERWK WKH 2E&6 GHVFULELQJ WKH PHQX

PHQXFORVH DQG WKH IOLJKW

IOLJKWVHQG&RPPDQGFRPPDQG

C. Notation for description of the presentation part of the GUI (rendering and activation)

,QDGGLWLRQWRWKH3HWULQHWGHVFULELQJWKHEHKDYLRURIWKH*8, ,&2GHVFULSWLRQGHILQHVWKHUHODWLRQVKLSEHWZHHQWKHEHKDYLRUDO PRGHO DQG WKH *8, HOHPHQWV 7R GR VR LW PDNHV XVH RI

UHQGHULQJ DQG DFWLYDWLRQ IXQFWLRQV 5HQGHULQJ KRZ WKH

FKDQJHVLQWKHEHKDYLRUDOPRGHODIIHFWVWKH*8,DQGDFWLYDWLRQ KRZ WKH DFWLRQV RQ WKH *8, ZLGJHWV DIIHFWV WKH EHKDYLRUDO PRGHO IXQFWLRQV DUH SURYLGHG LQ WKH IRUP RI WDEOHV VXFK DV 7DEOHDQG7DEOHUHVSHFWLYHO\

7KHUHQGHULQJIXQFWLRQVHH7DEOHDVVRFLDWHVWKHFKDQJHV

RI WKH SODFHV PDUNLQJV ZLWK UHQGHULQJ PHWKRGV RQ *8, HOHPHQWV)RUH[DPSOHLQ7DEOHUHQGHULQJPHWKRGVIRUWKH SODFH³)OLJKWV´RI)LJDUHDVVRFLDWHGWRWKHIROORZLQJHYHQW WRNHQHQWHULQJWRNHQBHQWHUDQGOHDYLQJWRNHQBUHPRYHGWKH SODFHDVZHOODVUHVHWWLQJWKHPDUNLQJPDUNLQJBUHVHWRIWKH SODFHLHJRLQJEDFNWRLQLWLDOPDUNLQJ7DEOHVKRZVWKDW D E

(7)

ZKHQHYHU D WRNHQ HQWHUV WKH SODFH ³)OLJKWV´ D FRPSRQHQW ZLGJHW LV DGGHG WR WKH GUDZLQJ DUHD RI WKH UDGDU VFUHHQ 6LPLODUO\ D ZLGJHW LV UHPRYHG ZKHQHYHU D WRNHQ OHDYHV WKH SODFH

Table 2. Excerpt of the rendering function associated with Fig. 4.

3ODFH 2E&6(YHQW 5HQGHULQJPHWKRG

)OLJKWV WRNHQBHQWHU DGGFRPSRQHQW RI WKH UDGDU GUDZLQJDUHD

)OLJKWV WRNHQBUHPRYHG UHPRYHFRPSRQHQW RI WKH UDGDUGUDZLQJDUHD

)OLJKW PDUNLQJBUHVHW UHPRYH$OO RI WKH UDGDU GUDZLQJDUHD

7KHDFWLYDWLRQ IXQFWLRQ VHH 7DEOH DVVRFLDWHV WKH *8,

HOHPHQWV ZLWK WKHHYHQWWUDQVLWLRQVELGLUHFWLRQDOO\2QRQH

KDQG WKH XVHU HYHQWV HJ PRXVH&OLFNHG RQ D IOLJKW DUH DVVRFLDWHGZLWKWKHHYHQWRIDWUDQVLWLRQHJRSHQ)OLJKW2Q WKH RWKHU WKH VWDWHV RI HYHQW WUDQVLWLRQV DUPHG RU QRW DUH DVVRFLDWHG ZLWK DFWLYDWLRQ PHWKRGV HJ VHW(QDEOHGERRO 7DEOH VKRZV WKDW WKH ³RSHQ)OLJKW´ HYHQW LQ WKH ³RSHQLQJ)OLJKW´ WUDQVLWLRQ IURP)LJ LV DVVRFLDWHG ZLWK D PRXVH&OLFNHG HYHQW RQ D IOLJKW 7DEOH DOVR LQGLFDWHV WKDW ZKHQHYHU WKH DVVRFLDWHG WUDQVLWLRQ LV DUPHG IOLJKWV PXVW EH HQDEOHG FOLFNDEOH GLVDEOHG QRQFOLFNDEOH RWKHUZLVH 1RWH WKDWHYHQWKRXJKWKH\DUHQRWYLVLEOHZKHQQRIOLJKWVDUHRSHQ 6(1'DQG$%257PHQXLWHPDUHHQDEOHGGLVDEOHGDFFRUGLQJ WRWKHVWDWHRIWKHLUDVVRFLDWHGHYHQWWUDQVLWLRQVWKXVSUHYHQWLQJ WKHXVHRIDFFHOHUDWRUNH\VLIDQ\

Table 3. Excerpt of the activation function associated with Fig. 4.

8VHU(YHQW 2E&6(YHQW $FWLYDWLRQPHWKRG PRXVH&OLFNHGRQ

DIOLJKW RSHQ)OLJKW VHW(QDEOHGERRO RIIOLJKWV

DFWLRQ3HUIRUPHG

RQ6(1' VHQG VHW(QDEOHGERRO RI6(1'PHQXLWHP

DFWLRQ3HUIRUPHG

RQ$%257 FDQFHO VHW(QDEOHGERRO RI$%257PHQXLWHP 9 02'(//,1*2)7+($7&$33/,&$7,21*8,86,1*,&2 7KHPRGHOOLQJRIWKH$7&DSSOLFDWLRQXVLQJ,&2UHTXLUHVWKH GHVFULSWLRQXVLQJ2E&6DQGUHQGHULQJDFWLYDWLRQIXQFWLRQVRI WKUHHHOHPHQWVWKHUDGDULPDJHIOLJKWDQGPHQX7KLV VHFWLRQLQWURGXFHVWKHGHVFULSWLRQRIWKHVHWKUHHHOHPHQWVDQG KLJKOLJKWVWKHLUPHDQVRIVXSSRUWLQJG\QDPLFLQVWDQWLDWLRQDV ZHOO DV WKH FKDUDFWHULVWLFV WKDW HQDEOHV WKHP WR VXSSRUW WKH UHTXLUHPHQWSUHYLRXVO\OLVWHG

A. Description of the Radar Image (Main Screen)

7KH³PDLQVFUHHQ´RIWKH$7&DSSOLFDWLRQFRQVLVWVLQDQLPDJH RIWKHVHFWRUFRQWUROOHGE\WKHDLUWUDIILFFRQWUROOHURQZKLFK DLUFUDIWDUHGUDZQDIWHUEHLQJG\QDPLFDOO\LQVWDQWLDWHG)LJ SUHVHQWVWZRPDUNLQJVRIWKHPRGHOGHVFULELQJWKHUDGDULPDJH EHKDYLRU 7KHG\QDPLFLQVWDQWLDWLRQRIIOLJKWVLVVXSSRUWHGWKURXJKWKH DGG)OLJKWLQYRFDWLRQ7KH³DGGLQJ)OLJKW´WUDQVLWLRQFRQVXPHV WKHWRNHQSODFHGLQWKH³6,3BDGG)OLJKW´DQGDGGWKHPWRWKH )OLJKWVSODFH ,QWHUDFWLRQZLWKWKHG\QDPLFDOO\LQVWDQWLDWHGDLUFUDIWPXVWEH FRQWUROOHGDVLWLVUHTXLUHGWKDW

x $ FRQWURO RUGHU LV VHQW WR RQO\ RQH SODQH RQO\ RQH SODQHFDQEHVHOHFWHGDWDWLPH x $OOFRQWURORUGHUVILQLVKE\HLWKHU$ERUWRU6HQG :LWKUHJDUGVWR*8,ZLGJHWVWKLVPHDQVWKDW x UHT$IOLJKWVKRXOGEHFOLFNDEOHLIDQGRQO\LIWKHUH LVQRIOLJKWDOUHDG\RSHQHG x UHT&ORVLQJRIWKHSRSXSPHQXLVDOORZHGRQO\DV DUHVXOWRIDFOLFNRQ6HQGRU$ERUWDVRSSRVHGWRWKH EHKDYLRURIFRQWH[WXDOPHQXVLQVWDQGDUGDSSOLFDWLRQV LHRQHVWKDWDUHFORVLQJDVVRRQDVDQRWKHULVRSHQHG ,Q)LJDWKUHHIOLJKWVDUHRQWKHVHFWRUSODFH³)OLJKWV´EXW QR IOLJKWV DUH RSHQHG SODFH ³2SHQHG)OLJKWV´ 7KXV LW LV SRVVLEOHWR³RSHQ´DIOLJKWWKHHYHQWWUDQVLWLRQ³2SHQLQJ)OLJKW LVDUPHG:KHQWKH³RSHQ)OLJKW´HYHQWLVUDLVHGE\DIOLJKWWKH PDUNLQJRIWKH,&2FKDQJHVWRWKHRQHSUHVHQWHGLQ)LJE7KH ILULQJRIWKHWUDQVLWLRQOHGWRWKHFRQVXPSWLRQRIWKHWRNHQLQ WKHSODFH³0D[2SHQ)OLJKW´WKHIOLJKWWRNHQLVFRSLHGDVDWHVW DUFLVXVHG7KLVGLVDUPHGWKHWUDQVLWLRQ³RSHQLQJ)OLJKW´$W WKLVSRLQWDQGDFFRUGLQJWRWKHDFWLYDWLRQIXQFWLRQ7DEOHDOO WKHIOLJKWVEHFRPHQRQFOLFNDEOHIXOILOOLQJUHT ,Q)LJEDIOLJKWLVRSHQHGZKLFKOHGWRWKHRSHQLQJRIWKH PHQX DFWLRQ ³PHQXRSHQIOLJKW´ LQ WKH ³RSHQLQJ)OLJKW´ WUDQVLWLRQ7KHIXOILOOPHQWRIUHTLVHQVXUHGE\WKHIDFWWKDW WKH ³PHQXFORVH´ LQYRFDWLRQ LV SHUIRUPHG RQO\ RQ WKH ³VHQGLQJ&RPPDQG´ DQG ³FDQFHOOLQJ&RPPDQG´ WUDQVLWLRQV 7KHVH WUDQVLWLRQV UHVSHFWLYHO\ KDQGOH ³VHQG´ DQG ³FDQFHO´ HYHQWVIURPWKHPHQX:KHQRQHRIWKHVHWUDQVLWLRQVLVILUHG WKHWRNHQLQ³2SHQHG)OLJKW´LVFRQVXPHGDQGDWRNHQLVDGGHG WR0D[2SHQ)OLJKWVWKHPRGHOUHWXUQVWRWKHPDUNLQJRI)LJD B. Description of a Flight

7KH PRGHOOLQJ RI WKH EHKDYLRU RI D IOLJKW HQDEOHV WKH YHULILFDWLRQ RI SURSHUWLHV WKDW JRHV EH\RQG WKH UHTXLUHPHQWV OLVWHG LQ WKH SUHVHQWDWLRQ LQ WKH FDVH VWXG\ )RU WKH $7& $SSOLFDWLRQWREHXVDEOHWKHGDWDLWSURYLGHVPXVWEHDFFXUDWH 0RGHOOLQJWKHEHKDYLRURIDLUFUDIWVHQDEOHVWKHYHULILFDWLRQRI ZLGJHWV SURSHUWLHV VXFK DV ORFDWLRQ ODEHO IRU IOLJKW QXPEHU VSHHGHWF,&2DOORZVWKHVHSDUDPHWHUVWREHSURYLGHGDWHYHU\ LQVWDQWLDWLRQRIDPRGHODQGRQHPD\DOORZWRXSGDWHWKHPDW DQ\WLPHYLDHYHQWWUDQVLWLRQVRULQYRFDWLRQRQFHWKHPRGHOLV LQVWDQWLDWHG 'XH WR VSDFH FRQVWUDLQW ZH GR QRW SUHVHQW WKH PRGHORID6LPSOH)OLJKWZKLFKLVPDLQO\FRPSRVHGRIJHWWHUV DQGVHWWHUVLQWKH IRUPRILQYRFDWLRQIRUSURSHUWLHVVXFKDV ODWLWXGHORQJLWXGHRUVSHHGDQGRIJHWWHUVIRUSURSHUWLHVVXFKDV SRVLWLRQGHULYHGIURPODWLWXGHDQGORQJLWXGH

C. Description of the Menu

(8)

³FRPPDQG:LWKRXW3DUDP´ RU D FRPPDQG WKDW LV QRW WKH ³92,&(´ RQH LV VHOHFWHG ZLWK D SDUDPHWHU VHW SODFH ³3$5$0(7(5´

1RWHWKDWWRSUHYHQWVHQGLQJDSDUDPHWHULQFRPSDWLEOHZLWKWKH VHOHFWHGFRPPDQGWKLVSDWWHUQLVGHVLJQHGWRUHPRYHDQ\WRNHQ IURP WKH SODFH ³3$5$0(7(5´ ZKHQ D FOLFN RQ D QHZ FRPPDQG LV GRQH D WRNHQ LV DGGHG LQ ³&/($5 3$5$0(7(5´LIDSDUDPHWHUZDVVHWWKHWRNHQLVFRQVXPHG E\ WKH WUDQVLWLRQ ³FOHDULQJ3DUDPHWHU´ 7KH DUF EHWZHHQ WKH SODFH ³3$5$0(7(5´ DQG WKH ³QR3DUDPHWHU7R&OHDU´ WUDQVLWLRQ LV DQ LQKLELWRU RQH LW LPSOLHV WKDW WKH WUDQVLWLRQ LV DUPHGRQO\LI³3$5$0(7(5´LVHPSW\

Fig. 5. Example of a pattern preventing clicking on the "send" button with an incorrect command.

9, $352&(66)2502'(/%$6('7(67,1*2)$33/,&$7,216 )($785,1*'<1$0,&,167$1&,$7,212)*8,:,'*(76 6R IDU ZH VKRZHG WKDW ,&2 HQDEOHV WKH GHVFULSWLRQ RI *8, DSSOLFDWLRQVIHDWXULQJG\QDPLFLQVWDQWLDWLRQRI*8,DQGFDUULHV VRPHSURSHUWLHVUHJDUGLQJ*8,ZLGJHWVWKDQNVWRWKHUHQGHULQJ DQGDFWLYDWLRQVIXQFWLRQV,QWKLVVHFWLRQZHSUHVHQWDSURFHVV LQVSLUHGE\>@WRWDNHDGYDQWDJHRIWKH,&2H[SUHVVLYHQHVVWR HQDEOH

x 7KH LPSOHPHQWDWLRQ RI WKH $87 XVLQJ ,&2 DV VSHFLILFDWLRQ x 7KHJHQHUDWLRQRIWHVWFDVHVIRUWKH$87IURPWKH,&2 PRGHOV 7R FRSH ZLWK WKH FKDOOHQJHV DVVRFLDWHG ZLWK WHVW FDVHV JHQHUDWLRQWKLVSURFHVVLQYROYHVDUDQGRPH[SORUDWLRQRIWKH DSSOLFDWLRQ PRGHOFRQVWUDLQHGE\WHVWVHOHFWLRQFULWHULD7KLV DOORZVHOHFWLQJUHOHYDQWVDPSOHRIWKHLQILQLWHH[HFXWLRQVSDFH RIDSSOLFDWLRQVGXULQJJHQHUDWLRQ)LJSUHVHQWVRXUSURFHVV WKDWXVHVWKHUHTXLUHPHQWVIRUWKH$87DVLQSXWVWRSRI)LJ DQG UHOLHV RQ WKH H[DFW VDPH PRGHOV IRU GULYLQJ WKH LPSOHPHQWDWLRQ DQG WKH WHVWLQJ DFWLYLWLHV 7KH IROORZLQJ VHFWLRQVGHWDLOWKHDFWLYLWLHVRIWKLVSURFHVVDQGGLVFXVVHVWKH FKDOOHQJHVWKH\UDLVHVHVSHFLDOO\UHJDUGLQJWKHJHQHUDWLRQ RI UHOHYDQWLQYRFDWLRQVDQGHYHQWSDUDPHWHUV hd /KĞƐĐƌŝƉƚŝŽŶŽĨƚŚĞhd dĞƐƚKďũĞĐƚŝǀĞƐ dĞƐƚ^ĞůĞĐƚŝŽŶƌŝƚĞƌŝĂ ;Ğ͘Ő͘ŵĂŶĚĂƚŽƌǇ ƚƌĂŶƐŝƚŝŽŶͿ ZĞƋƵŝƌĞŵĞŶƚƐ DŽĚĞůůŝŶŐŽĨƚŚĞhdĂŶĚ ĞĨŝŶŝƚŝŽŶŽĨZĞŶĚĞƌŝŶŐ ĂŶĚĐƚŝǀĂƚŝŽŶ&ƵŶĐƚŝŽŶƐ ĞĨŝŶŝƚŝŽŶŽĨdĞƐƚ KďũĞĐƚŝǀĞƐ ^ŝŵƵůĂƚŝŽŶĂŶĚ>ŽŐŐŝŶŐŽĨ dĞƐƚĂƐĞƐ;ŝŶWĞƚƐŚŽƉ͕ ĐŽŶƐƚƌĂŝŶĞĚͲƌĂŶĚŽŵŶĞƐƐͿ ĞǀĞůŽƉŵĞŶƚŽĨƚŚĞhd dĞƐƚĂƐĞƐĂƐĞĚŽŶ ZĞƐƵůƚŽĨ/K ^ŝŵƵůĂƚŝŽŶ dĞƐƚ^ĐƌŝƉƚƐ/ŶƐƚĂŶĐŝĂƚŝŽŶ /ŶƐƚĂŶĐŝĂƚĞĚ dĞƐƚ^ĐƌŝƉƚƐ dĞƐƚǆĞĐƵƚŝŽŶ sĞƌĚŝĐƚ ĞĨŝŶŝƚŝŽŶŽĨdĞƐƚ ^ĞůĞĐƚŝŽŶƌŝƚĞƌŝĂ;tŝƚŚ dĞƐƚKďũĞĐƚŝǀĞƐŝŶDŝŶĚͿ ,ŝŐŚͲ>ĞǀĞůWĞƚƌŝ ŶĞƚDŽĚĞů ZĞŶĚĞƌŝŶŐĂŶĚ ĐƚŝǀĂƚŝŽŶ &ƵŶĐƚŝŽŶ ǆƚƌĂĐƚŝŽŶŽĨh/ŽŶƚĞŶƚ ĨƌŽŵƚŚĞhd h/ŽŶƚĞŶƚ ĞƐĐƌŝƉƚŝŽŶ

Fig. 6. The Process of Random-Model Based Testing of applications featuring dynamic instanciation of GUI widgets.

A. Preliminary activities

1) Definition of Test Objectives

(9)

specifications are correctly implemented, which is variously referred to in the literature as conformance testing, correctness testing, or functional testing. However, several other non-functional properties may be tested as well—including performance, reliability, and usability, among many others 'HILQLQJVXFKWHVWREMHFWLYHVLVVWDQGDUGSUDFWLFHLQVRIWZDUH HQJLQHHULQJ %\ EXLOGLQJ PRGHOV SULRU WR GHYHORSPHQW RXU DSSURDFK LV PDLQO\ JHDUHG WRZDUGV $FFHSWDQFH4XDOLILFDWLRQ 7HVWLQJ<HWRQHFDQHOHFWRXUDSSURDFKIRUWKHLGHQWLILFDWLRQ RIIDXOWVLQ5HOLDELOLW\HYDOXDWLRQ

2) Modelling of the AUT and Definition of Rendering and Activation Functions

7KHSURGXFWLRQRIDGHVFULSWLRQRIWKH$87LVDNH\HOHPHQWLQ RXUSURFHVVDVWKLVGHVFULSWLRQLVXVHGLQERWKWHVWJHQHUDWLRQ DQGGHYHORSPHQWDFWLYLWLHV)RUWKHModelling of the AUT and Definition of Rendering and Activation FunctionsDFWLYLW\WRS OHIWRI)LJDIRUPDOPRGHOHQJLQHHULVDSSRLQWHGZLWKWKH WDVN RI SURGXFLQJ DQ ,&2 GHVFULSWLRQ RI WKH DSSOLFDWLRQ EHKDYLRU LQ WKH &$6( WRRO VXSSRUWLQJ WKH QRWDWLRQ 3HWVKRS 7KLV PRGHO PXVW EH DFFRPSDQLHG E\ WKH GHVFULSWLRQ RI WKH UHQGHULQJ DQG DFWLYDWLRQ IXQFWLRQV LH WKH PDSSLQJ EHWZHHQ *8, HOHPHQWV SURSHUWLHV DQG LQIRUPDWLRQ DVVRFLDWHG ZLWK SODFHV WRNHQ HQWHUHG WRNHQ UHPRYHG DQG WUDQVLWLRQV WUDQVLWLRQDYDLODEOHRUQRWRIWKH3HWULQHWPRGHO

B. Test generation activities

7KHWHVWJHQHUDWLRQDFWLYLWLHVDLPVDWSURGXFLQJIURPWKH,&2 GHVFULSWLRQRIWKH$87WHVWFDVHVDQGWHVWVFULSWVIRUWKH WDUJHWHGLPSOHPHQWDWLRQ

1) Definition of Test Selection Criteria

7KH 'HILQLWLRQ RI 7HVW 6HOHFWLRQ &ULWHULD FRQVLVWV LQWR WUDQVODWLQJWKH7HVW2EMHFWLYHVLQWRFRQVWUDLQWVRQWKHPRGHOV IRUWKHWHVWFDVHVJHQHUDWLRQ)RULQVWDQFHLQDVFHQDULRZKHUH WKHREMHFWLYHLVAcceptance/Qualification TestingWKHJRDOLVWR LGHQWLI\UHOHYDQWSODFHVWUDQVLWLRQVfor verifying that customer’s requirements are met)RULQVWDQFHZLWKWKHUHTXLUHPHQW³DOO FRQWURORUGHUVILQLVKE\HLWKHU$ERUWRU6HQG´ZHZDQWWKHWHVW FDVHJHQHUDWRUWRSURGXFHWHVWFDVHVIURPWKHPRGHOE\VHOHFWLQJ SDWKVRQZKLFKWKHHYHQWVUHODWHGWRWKH³$ERUW´DQGRU³6HQG´ FRPPDQGDUHUDLVHG

2) Simulation and Logging of Test Cases

7KH 6LPXODWLRQ DQG /RJJLQJ RI 7HVW &DVHV KDSSHQV LQ WKH &$6(WRROVXSSRUWLQJWKH,&2QRWDWLRQ3HWVKRS7KHDLPRI WKLV VWHS LV WR SURGXFH WUDFHV RI DQ DXWRPDWHG VLPXODWLRQ FRQVWUDLQHGE\WKHWHVWVHOHFWLRQFULWHULDRIWKH3HWULQHWWKDW FRQWDLQV

x WKHLQYRFDWLRQVDQGHYHQWVWREHSOD\HGRQWKH687 x WKHH[SHFWHGVWDWHRIWKH*8,HOHPHQWVEHIRUHDIWHUWKH

LQYRFDWLRQHYHQWLVGRQHUDLVHG

7KH NH\ DW WKLV VWHS LV WKH SURGXFWLRQ RI UHOHYDQW LQYRFDWLRQHYHQWV ZKLFK FDQ EH PDGH GLIILFXOW E\ WKH SDUDPHWUL]HGQDWXUHRIVRPHRIWKHP,QGHHGZKLOHDFOLFNRQ DEXWWRQLVQRWSDUDPHWHUL]HGSURGXFLQJDQLQSXWRQDWH[WER[ LVDVLWUHTXLUHVWKHDFWXDOVWULQJLQSXW,QDVLPLODUZD\WKH VLPXODWRU PXVW NQRZ SDUDPHWHUV RI DQ LQYRFDWLRQ 2XU VWUDWHJLHVIRUGHDOLQJZLWKWKHVHLVVXHVDUHWKHIROORZLQJ

x (YHQW SDUDPHWHUV DUH REWDLQHG E\ VROYLQJ WKH SUH FRQGLWLRQDVVRFLDWHGZLWKWKHWUDQVLWLRQUHFHLYLQJWKH HYHQW XVLQJ WKH = 607 6DWLVILDELOLW\ 0RGXOR 7KHRU\VROYHU>@LQWHUIDFHGZLWK3HWVKRS

x ,QVWDQWLDWLRQDQGLQYRFDWLRQSDUDPHWHUVDUHSURYLGHG PDQXDOO\E\WKHXVHUDQGPD\HLWKHUEHGHVFULEHGDV XQLTXH YDOXH UDQJHV HJ SDUDPHWHU altitude

א

[0,25000]RUDUUD\VHJSDUDPHWHUflightNumber

א

(“U2 9876”, “NK 1234”, “D7 6543”)

$WWKHHQGRIWKHVLPXODWLRQWKHORJVDUHVDYHGLQDQ;0/ILOH )LJSURYLGHVDQRYHUYLHZRIWKH;6'DVVRFLDWHGZLWKWKLV ;0/ ILOH ,W VKRZV WKDW D WHVW GHVFULSWLRQ WHVWGHVF ILOHV FRQWDLQV

x $GHILQLWLRQRIWKH8VHU,QWHUIDFHXLGHIZLWKWKHOLVW RI 3HWUL QHWV LQYROYHG LQ WKH VLPXODWLRQ REFVHV DV ZHOODVWKHHYHQWVHYHQWWUDQVLWLRQLQYRFDWLRQVDQG UHQGHULQJVSODFHVWKH\FRQWDLQ

x $ OLVW RI DOO WKH XVHU LQWHUIDFH VWDWHV XLVWDWHV HQFRXQWHUHG GXULQJ WKH VLPXODWLRQ DQ XLVWDWH FRQWDLQLQJ WKH SDUDPHWHUV IRU DFWLYDWLRQV ZDV D WUDQVLWLRQDUPHGRUQRW"DQGUHQGHULQJVZKDW ZHUH WKHYDOXHFDUULHGE\WKHWRNHQVLQDSODFH"IXQFWLRQV x $OLVWRIWHVWFDVHVFRPSRVHGRIWHVWVWHSV(DFKWHVW

VWHS UHIHUV WR D XLVWDWH WKDW PXVW EH YHULILHG EHIRUH SHUIRUPLQJ DQ\ DFWLRQV 7KH DFWLRQV LQ D VWHS FDQ HLWKHU EH LQYRFDWLRQV RU SURGXFWLRQ RI HYHQWV $IWHU SHUIRUPLQJWKHVHDFWLRQVQRQGHWHUPLQLVWLFEHKDYLRU RI VRPH *8, DSSOLFDWLRQV PD\ OHDG WR GLIIHUHQW XLVWDWHVGHSHQGLQJRIWKHH[HFXWLRQ7RFRSHZLWKWKDW 3HWVKRSORJVDOOWKHSRVVLEOHQH[WVWHSVFRQWDLQLQJDQ 8,VWDWHWKH*8,DSSOLFDWLRQPD\UHDFK

(10)

3) Test Scripts Instanciation

7KH Test Scripts Instantiation FRQVLVW LV D VHPLDXWRPDWHG SURFHVVWKDWEXLOGVWKHVWUXFWXUHRIDWHVWVFULSW-XQLW06WHVW ;&7HVWHWFIURPWKUHHHOHPHQWV

x 7KHWHVWFDVHVEDVHGRQWKHUHVXOWVRIWKHVLPXODWLRQ x 7KH 8, &RQWHQW GHVFULSWLRQ QDPH RI ZLGJHWV

FRQWDLQHUVHWF

x 7KHOLVWRIUHQGHULQJDQGDFWLYDWLRQIXQFWLRQVIURPWKH ,&2GHVFULSWLRQ

)RUWKHWHVWVFULSWJHQHUDWLRQWRRFFXUWKHXVHUPXVWPDQXDOO\ VSHFLI\ WKH DFWXDO FRUUHVSRQGHQFH EHWZHHQ WKH UHQGHULQJVDFWLYDWLRQIXQFWLRQVDQGWKHDSSOLFDWLRQVRXUFHFRGH 1RWHWKDWRQHPD\KDYHGHFLGHGWRPRGHOWKHDSSOLFDWLRQZLWK D SDUWLFXODU WDUJHW SODWIRUP LQ PLQG HJ -DYD 6ZLQJ WKXV GLUHFWO\PHQWLRQLQJWKHPLQWKH,&2UHQGHULQJDQGDFWLYDWLRQ IXQFWLRQ <HW ZH FODLP WKDW WKHVH IXQFWLRQV VKRXOG EH JHQHULFDOO\GHVFULEHGHJ³'HDFWLYDWLRQRIWKHVHQGEXWWRQ´ VRWHVWVFULSWFDQEHLQVWDQWLDWHGIRUGLIIHUHQWWDUJHWSODWIRUP C. Development activities

1) Development of the AUT

7KHDevelopment of the AUTFRQVLVWVLQWKHSURGXFWLRQE\D WHDP RI VRIWZDUH HQJLQHHUV DQG GHYHORSHUV RI DQ LPSOHPHQWDWLRQRIWKH$87IRUWKHWDUJHWHGSODWIRUP'HWDLOV RQ LPSOHPHQWLQJ IURP DQ ,&2 VSHFLILFDWLRQ DUH SURYLGHG LQ >@DQG>@7KLVSURFHVVLQYROYHVREWDLQLQJWKHUHDFKDELOLW\ JUDSKRIWKH,&23HWULQHW$UHDFKDELOLW\JUDSKRID3HWULQHW LVDGLUHFWHGJUDSK* 9(ZKHUHYę9UHSUHVHQWVDFODVVRI UHDFKDEOHPDUNLQJVHę(UHSUHVHQWVDGLUHFWHGDUFIURPDFODVV RI PDUNLQJV WR DQRWKHU FODVV RI PDUNLQJV >@ :H WDNH DGYDQWDJH RI WKH $37 $QDO\VLV RI 3HWUL QHWV DQG ODEHOOHG WUDQVLWLRQV\VWHPVSURMHFWWRJHQHUDWHWKLVJUDSK>@

D. Test Execution

7KHTest ExecutionLVGXHWRWKHQDWXUHRIWKHVFULSWZHJHQHUDWH -XQLW06WHVW;&7HVWHWFWRWDOO\GHSHQGHQWRIWKH,'(WKH WHVWHUVDUH ZRUNLQJRQ)RUWKLVUHDVRQ ZH ZLOOQRWFRPPHQW IXUWKHURQWHVWH[HFXWLRQ

9,,&21&/86,21$1')8785(:25.

0RGHO%DVHG 7HVWLQJ UHTXLUHV PRGHOOLQJ WHFKQLTXHV WKDW DUH H[SUHVVLYHHQRXJKWRGHVFULEHWKHEHKDYLRURIWKH$SSOLFDWLRQ 8QGHU7HVW>@>@6LQFHORQJNQRZQIRUWKHLUFRPSOH[LW\LQ WKHILHOGRI*8,7HVWLQJ>@>@*8,DSSOLFDWLRQVDUHJHWWLQJ PRUH DQG PRUH FRPSOH[ IHDWXULQJ G\QDPLF LQVWDQWLDWLRQ RI ZLGJHWV DQG FRPSOH[ LQWHUDFWLRQ HJ PXOWLWRXFK RQHV >@>@ 7KLV LQFUHDVLQJ FRPSOH[LW\ DIIHFWV WKH DELOLW\ RI G\QDPLFDOO\ EXLOGLQJ PRGHO RI WKH $87 E\ H[SORULQJ LWV H[HFXWLRQVSDFHDVGRQHLQWKHILHOGRI*8,7HVWLQJ>@7R FRSH ZLWK WKH LQFUHDVLQJ FRPSOH[LW\ WKLV SDSHU SURSRVHG WR EXLOGWKHDSSOLFDWLRQPRGHOVIURPUHTXLUHPHQWVDVGRQHLQWKH VWDQGDUG0RGHO%DVHG7HVWLQJSURFHVV>@&RPSDULQJRYHU 8,'/VZHKDYHHVWDEOLVKHGWKDW,&2>@LVRQHRIWKHPRVW VXLWDEOHQRWDWLRQIRUGHVFULELQJWKHODUJHVHWRIEHKDYLRUIRXQG LQPRGHUQ*8,DSSOLFDWLRQV:HGHPRQVWUDWHGVRPHRIWKH,&2 DELOLWLHV E\ GHVFULELQJ DQ $LU 7UDIILF &RQWURO DSSOLFDWLRQ

IHDWXULQJ G\QDPLF LQVWDQWLDWLRQ RI ZLGJHWV DV ZHOO DV QRQ VWDQGDUGEHKDYLRUHJRQSRSXSPHQXDQGPHQXLWHPV 7KHH[SUHVVLYHQHVVRI,&2HQDEOHVWKHPRGHOVWREHXVHGIRU WZR GLIIHUHQW SXUSRVHV 2Q RQH KDQG WKH\ DUH VXLWDEOH DV VSHFLILFDWLRQIRUWKHGHYHORSPHQWRIWKHDSSOLFDWLRQ>@>@RQ WKHRWKHUWKH\HQDEOHWHVWFDVHVJHQHUDWLRQ,QWKLVSDSHUZH SURSRVHGDGHYHORSPHQWWHVWLQJSURFHVVWKDWWDNHVDGYDQWDJHRI ERWKRIWKHVHFKDUDFWHULVWLFV:KLOHRQHPD\GHFLGHWRPRGHO WKHDSSOLFDWLRQZLWKDSDUWLFXODUGHYHORSPHQWHQYLURQPHQWLQ PLQGZHSURSRVHWRUHO\RQWKHJHQHUDWLRQRIDEVWUDFWWHVWFDVHV WKHQLQVWDQWLDWHGIRUWKHWDUJHWHGSODWIRUPE\UHIHUULQJWRWKH ,&2DFWLYDWLRQDQGUHQGHULQJIXQFWLRQV 7KHDSSURDFKSURSRVHGLQWKLVSDSHUHQGVXSZLWKHQDEOLQJWKH WHVWLQJRIWKH*8,DSSOLFDWLRQVXVLQJWHVWVFULSWV7KXVZHDUH FRQGXFWLQJ ³RIIOLQH´ WHVWLQJ >@ :KLOH WKLV DOORZ IRU HDV\ UHSOD\LQJRIWHVWFDVHVWKLVDOVRLPSOLHVWKDWZHFRQVWUDLQWKH WHVW FDVH JHQHUDWLRQ VR WKH\ DUH PDQDJHDEOH LQ VL]H DQG JHQHUDWLRQLVQRWWRRORQJ:HDUHFXUUHQWO\LQYHVWLJDWLQJWKH RSSRUWXQLW\ RI FRQGXFWLQJ ³RQOLQH´ WHVWLQJ >@ XVLQJ ,&2 :LWKRQOLQHWHVWLQJWKHWHVWJHQHUDWLRQDOJRULWKPVFDQUHDFWWR WKHDFWXDORXWSXWVRIWKH6877KLVKDVWKHSRWHQWLDOWRHQDEOH ORQJHUWHVWUXQZKLOHEHQHILWLQJIURPWKHFRUUHFWQHVVRIPRGHOV EXLOWIURPWKHDSSOLFDWLRQUHTXLUHPHQWV 5()(5(1&(6 >@ &$SSHUWDQG0%HDXGRXLQ/DIRQ³6ZLQJ6WDWHVDGGLQJVWDWHPDFKLQHV WRWKHVZLQJWRRONLW´LQ3URFHHGLQJVRIWKHWKDQQXDO$&0V\PSRVLXP RQ8VHULQWHUIDFHVRIWZDUHDQGWHFKQRORJ\0RQWUHX[6ZLW]HUODQG SS±GRL >@ ,%DQHUMHH%1JX\HQ9*DURXVLDQG$00HPRQ³*UDSKLFDOXVHU LQWHUIDFH*8,WHVWLQJ6\VWHPDWLFPDSSLQJDQGUHSRVLWRU\´,QIRUPDWLRQ DQG6RIWZDUH7HFKQRORJ\YROQRSS±2FWGRL MLQIVRI

>@ ( %DUERQL 6 &RQYHUV\ ' 1DYDUUH DQG 3 3DODQTXH ³0RGHO%DVHG (QJLQHHULQJRI:LGJHWV8VHU$SSOLFDWLRQVDQG6HUYHUV&RPSOLDQWZLWK $5,1&6SHFLILFDWLRQ´LQ,QWHUDFWLYH6\VWHPV'HVLJQ6SHFLILFDWLRQ DQG9HULILFDWLRQ%HUOLQ+HLGHOEHUJSS±GRL B

>@ 5 %DVWLGH ' 1DYDUUH DQG 3 3DODQTXH ³$ 0RGHOEDVHG 7RRO IRU ,QWHUDFWLYH3URWRW\SLQJRI+LJKO\,QWHUDFWLYH$SSOLFDWLRQV´LQ&+,¶ ([WHQGHG$EVWUDFWVRQ+XPDQ)DFWRUVLQ&RPSXWLQJ6\VWHPV1HZ<RUN 1<86$SS±GRL

>@ 5%DVWLGH'1DYDUUH33DODQTXH$6FK\QDQG3'UDJLFHYLF³$ PRGHOEDVHG DSSURDFK IRU UHDOWLPH HPEHGGHG PXOWLPRGDO V\VWHPV LQ PLOLWDU\DLUFUDIWV´LQ3URFHHGLQJVRIWKHWKLQWHUQDWLRQDOFRQIHUHQFHRQ 0XOWLPRGDOLQWHUIDFHV6WDWH&ROOHJH3$86$SS±GRL

>@ ( %HVW DQG 8 6FKODFKWHU ³$QDO\VLV RI 3HWUL 1HWV DQG 7UDQVLWLRQ 6\VWHPV´(OHFWURQ3URF7KHRU&RPSXW6FLYROSS±$XJ GRL(37&6 >@ 5$%ROW³µ3XWWKDWWKHUH¶9RLFHDQGJHVWXUHDWWKHJUDSKLFVLQWHUIDFH´ LQ3URFHHGLQJVRIWKHWKDQQXDOFRQIHUHQFHRQ&RPSXWHUJUDSKLFVDQG LQWHUDFWLYH WHFKQLTXHV 6HDWWOH :DVKLQJWRQ 86$ SS ± GRL >@ -&&DPSRVHWDO³$0RUH,QWHOOLJHQW7HVW&DVH*HQHUDWLRQ$SSURDFK 7KURXJK 7DVN 0RGHOV 0DQLSXODWLRQ´ 3URF $&0 +XP&RPSXW ,QWHUDFWYROQR(,&6SS±-XQGRL >@ $&DQQ\(%RX]HNUL&0DUWLQLHDQG33DODQTXH³5DWLRQDOL]LQJWKH 1HHGRI$UFKLWHFWXUH'ULYHQ7HVWLQJRI,QWHUDFWLYH6\VWHPV´LQ+XPDQ &HQWHUHGDQG(UURU5HVLOLHQW6\VWHPV'HYHORSPHQW

(11)

>@ / &DUGHOOL DQG 5 3LNH ³6TXHDN DODQJXDJH IRU FRPPXQLFDWLQJZLWK PLFH´ 6,**5$3+ &RPSXW *UDSK YRO QRSS± -XO GRL

>@ /&KHQJ-&KDQJ=<DQJDQG&:DQJ³*8,&DW*8,WHVWLQJDVD VHUYLFH´ LQ VW ,((($&0 ,QWHUQDWLRQDO &RQIHUHQFH RQ $XWRPDWHG6RIWZDUH(QJLQHHULQJ$6(SS±

>@ 6 5 &KRXGKDU\ $ *RUOD DQG $ 2UVR ³$XWRPDWHG 7HVW ,QSXW *HQHUDWLRQIRU$QGURLG$UH:H7KHUH<HW"´LQWK,((($&0 ,QWHUQDWLRQDO &RQIHUHQFH RQ $XWRPDWHG 6RIWZDUH (QJLQHHULQJ $6( SS±GRL$6(

>@ 3 'UDJLFHYLF DQG -' )HNHWH ³6XSSRUW IRU LQSXW DGDSWDELOLW\ LQ WKH ,&21 WRRONLW´ LQ 3URFHHGLQJV RI WKH WK LQWHUQDWLRQDO FRQIHUHQFH RQ 0XOWLPRGDOLQWHUIDFHV6WDWH&ROOHJH3$86$SS±GRL

>@ %'XPDV'/DODQQHDQG5,QJROG³+HSKDLV7.DWRRONLWIRUUDSLG SURWRW\SLQJ RI PXOWLPRGDO LQWHUIDFHV´ LQ 3URFHHGLQJV RI WKH LQWHUQDWLRQDO FRQIHUHQFH RQ 0XOWLPRGDO LQWHUIDFHV &DPEULGJH 0DVVDFKXVHWWV 86$ SS ± GRL

>@ -*UHJRU\/%HUWKRXG77U\IRQDV$5RVVLJQRODQG/)DXUH³7KH ORQJ DQG ZLQGLQJ URDG 0%6( DGRSWLRQ IRU IXQFWLRQDO DYLRQLFV RI VSDFHFUDIW´-RXUQDORI6\VWHPVDQG6RIWZDUHYROS)HE GRLMMVV

>@ $+DPRQ33DODQTXH-/6LOYD<'HOHULVDQG(%DUERQL³)RUPDO 'HVFULSWLRQRI0XOWLWRXFK,QWHUDFWLRQV´LQ3URFHHGLQJVRIWKHWK$&0 6,*&+, 6\PSRVLXP RQ (QJLQHHULQJ ,QWHUDFWLYH &RPSXWLQJ 6\VWHPV 1HZ <RUN 1< 86$ SS ± GRL

>@ $+DPRQ33DODQTXH0&URQHO5$QGUp(%DUERQLDQG'1DYDUUH ³)RUPDO 0RGHOOLQJ RI '\QDPLF ,QVWDQWLDWLRQ RI ,QSXW 'HYLFHV DQG ,QWHUDFWLRQ 7HFKQLTXHV $SSOLFDWLRQ WR 0XOWLWRXFK ,QWHUDFWLRQV´ LQ 3URFHHGLQJV RI WKH $&0 6,*&+, 6\PSRVLXP RQ (QJLQHHULQJ ,QWHUDFWLYH&RPSXWLQJ6\VWHPV1HZ<RUN1<86$SS± GRL

>@ .+LQFNOH\0&]HUZLQVNLDQG06LQFODLU³,QWHUDFWLRQDQGPRGHOLQJ WHFKQLTXHV IRU GHVNWRS WZRKDQGHG LQSXW´ LQ 3URFHHGLQJV RI WKH WK DQQXDO$&0V\PSRVLXPRQ8VHULQWHUIDFHVRIWZDUHDQGWHFKQRORJ\6DQ )UDQFLVFR &DOLIRUQLD 86$ SS ± GRL

>@ ,((( &RPSXWHU 6RFLHW\ 3 %RXUTXH DQG 5 ( )DLUOH\ *XLGH WR WKH 6RIWZDUH(QJLQHHULQJ%RG\RI.QRZOHGJH6:(%2.59HUVLRQ UGHG/RV$ODPLWRV&$86$,(((&RPSXWHU6RFLHW\3UHVV >@ .-HQVHQ/0.ULVWHQVHQDQG/:HOOV³&RORXUHG3HWUL1HWVDQG&31

7RROVIRUPRGHOOLQJDQGYDOLGDWLRQRIFRQFXUUHQWV\VWHPV´,QW-6RIWZ 7RROV 7HFKQRO 7UDQVIHU YRO QR SS ± -XQ GRL V[ >@ '.DPPHU-:RMG]LDN0.HFN5*URKDQG67DUDQNR³7RZDUGVD IRUPDOL]DWLRQRIPXOWLWRXFKJHVWXUHV´LQ$&0,QWHUQDWLRQDO&RQIHUHQFH RQ,QWHUDFWLYH7DEOHWRSVDQG6XUIDFHV6DDUEUFNHQ*HUPDQ\SS ±GRL >@ ..DWVXUDGD<1DNDPXUD+<DPDGDDQG71LWWD³;,6/DODQJXDJH IRUGHVFULELQJPXOWLPRGDOLQWHUDFWLRQVFHQDULRV´LQ3URFHHGLQJVRIWKH WK LQWHUQDWLRQDO FRQIHUHQFH RQ 0XOWLPRGDO LQWHUIDFHV 9DQFRXYHU %ULWLVK &ROXPELD &DQDGD SS ± GRL

>@ 9/HOOL$%ORXLQ%%DXGU\DQG)&RXORQ³2QPRGHOEDVHGWHVWLQJ DGYDQFHG *8,V´ LQ ,((( (LJKWK ,QWHUQDWLRQDO &RQIHUHQFH RQ 6RIWZDUH 7HVWLQJ 9HULILFDWLRQ DQG 9DOLGDWLRQ :RUNVKRSV ,&67: SS±GRL,&67:

>@ / 0DULDQL 0 3H]]H 2 5LJDQHOOL DQG 0 6DQWRUR ³$XWR%ODFN7HVW $XWRPDWLF %ODFN%R[ 7HVWLQJ RI ,QWHUDFWLYH $SSOLFDWLRQV´ LQ 9HULILFDWLRQDQG9DOLGDWLRQ,((()LIWK,QWHUQDWLRQDO&RQIHUHQFHRQ 6RIWZDUH7HVWLQJSS±GRL,&67 >@ $00HPRQDQG%11JX\HQ³$GYDQFHVLQ$XWRPDWHG0RGHO%DVHG

6\VWHP 7HVWLQJ RI 6RIWZDUH $SSOLFDWLRQV ZLWK D *8, )URQW(QG´ LQ $GYDQFHVLQ&RPSXWHUVYRO09=HONRZLW](G(OVHYLHUSS ±

>@ '1DYDUUH33DODQTXH-)/DGU\DQG(%DUERQL³,&2V$0RGHO EDVHG 8VHU ,QWHUIDFH 'HVFULSWLRQ 7HFKQLTXH 'HGLFDWHG WR ,QWHUDFWLYH 6\VWHPV$GGUHVVLQJ8VDELOLW\5HOLDELOLW\DQG6FDODELOLW\´$&07UDQV &RPSXW+XP,QWHUDFWYROQRSS±1RYGRL

>@ /GH0RXUDDQG1%M¡UQHU³=$Q(IILFLHQW6076ROYHU´LQ7RROV DQG $OJRULWKPV IRU WKH &RQVWUXFWLRQ DQG $QDO\VLV RI 6\VWHPV %HUOLQ +HLGHOEHUJSS±GRLB >@ %11JX\HQ%5REELQV,%DQHUMHHDQG$0HPRQ³*8,7$5DQ LQQRYDWLYHWRROIRUDXWRPDWHGWHVWLQJRI *8,GULYHQVRIWZDUH´$XWRP 6RIWZ(QJYROQRSS±0DUGRLV >@ 62QH\%0\HUVDQG-%UDQGW³&RQVWUDLQW-6SURJUDPPLQJLQWHUDFWLYH EHKDYLRUV IRU WKH ZHE E\ LQWHJUDWLQJ FRQVWUDLQWV DQG VWDWHV´ LQ 3URFHHGLQJV RI WKH WK DQQXDO $&0 V\PSRVLXP RQ 8VHU LQWHUIDFH VRIWZDUH DQG WHFKQRORJ\ &DPEULGJH 0DVVDFKXVHWWV 86$ SS ±GRL

>@ 3$3DODQTXH5%DVWLGH/'RXUWHDQG&6LEHUWLQ%ODQF³'HVLJQRI XVHUGULYHQ LQWHUIDFHV XVLQJ 3HWUL QHWV DQG REMHFWV´ LQ $GYDQFHG ,QIRUPDWLRQ 6\VWHPV (QJLQHHULQJ %HUOLQ +HLGHOEHUJ SS ± GRLB

>@ 33DODQTXH5%DVWLGHDQG)3DWHUQz³)RUPDO6SHFLILFDWLRQDVD7RRO IRU 2EMHFWLYH $VVHVVPHQW RI 6DIHW\&ULWLFDO ,QWHUDFWLYH 6\VWHPV´ LQ +XPDQ&RPSXWHU,QWHUDFWLRQ,17(5$&7¶,),37&,QWHUQDWLRQDO &RQIHUHQFH RQ +XPDQ&RPSXWHU ,QWHUDFWLRQ WK±WK -XO\ 6\GQH\ $XVWUDOLD 6 +RZDUG - +DPPRQG DQG * /LQGJDDUG (GV %RVWRQ0$6SULQJHU86SS±

>@ 3 3DODQTXH 5 %HUQKDXSW ' 1DYDUUH 0 2XOG DQG 0 :LQFNOHU ³6XSSRUWLQJ8VDELOLW\(YDOXDWLRQRI0XOWLPRGDO0DQ0DFKLQH,QWHUIDFHV IRU6SDFH*URXQG6HJPHQW$SSOLFDWLRQV8VLQJ3HWULQHWV%DVHG)RUPDO 6SHFLILFDWLRQ´LQ6SDFH2SV&RQIHUHQFHYROV$PHULFDQ,QVWLWXWH RI$HURQDXWLFVDQG$VWURQDXWLFV

>@ 3 3DODQTXH -) /DGU\ ' 1DYDUUH DQG ( %DUERQL ³+LJK)LGHOLW\ 3URWRW\SLQJ RI ,QWHUDFWLYH 6\VWHPV &DQ %H )RUPDO 7RR´ LQ +XPDQ &RPSXWHU,QWHUDFWLRQ1HZ7UHQGV%HUOLQ+HLGHOEHUJSS± GRLB

>@ & $ 3HWUL ³&RPPXQLFDWLRQ ZLWK DXWRPDWD´ KWWSHGRFVXEXQL KDPEXUJGHLQIRUPDWLNYROOWH[WHSGIGLVVBSHWULBHQJOSGI

>@ 0 3H]]q 3 5RQGHQD DQG ' =XGGDV ³$XWRPDWLF *8, 7HVWLQJ RI 'HVNWRS$SSOLFDWLRQV$Q(PSLULFDO$VVHVVPHQWRIWKH6WDWHRIWKH$UW´ LQ &RPSDQLRQ 3URFHHGLQJV IRU WKH ,667$(&223 :RUNVKRSV 1HZ<RUN1<86$SS±GRL >@ 08WWLQJ$3UHWVFKQHUDQG%/HJHDUG³$WD[RQRP\RIPRGHOEDVHG WHVWLQJDSSURDFKHV´6RIWZ7HVW9HULI5HOLDEYROQRSS± $XJGRLVWYU

>@ . 9RURE\RY DQG 3 .ULVKQDQ ³&RPSDULQJ PRGHO FKHFNLQJ DQG VWDWLF SURJUDP DQDO\VLV $ FDVH VWXG\ LQ HUURU GHWHFWLRQ DSSURDFKHV´ 3URFHHGLQJVRI669

>@ 7(-9RV30.UXVH1&RQGRUL)HUQiQGH]6%DXHUVIHOGDQG- :HJHQHU ³7(67$5 7RRO 6XSSRUW IRU 7HVW $XWRPDWLRQ DW WKH 8VHU ,QWHUIDFH /HYHO´ ,-,60' YRO QR SS ± -XO GRL ,-,60'