HAL Id: hal-01022139
https://hal-enac.archives-ouvertes.fr/hal-01022139
Submitted on 23 Jul 2014
HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Supporting multidisciplinary software composition for interactive applications
Stéphane Chatty
To cite this version:
Stéphane Chatty. Supporting multidisciplinary software composition for interactive applications. SC
2008, 7th International Symposium on Software Composition, Mar 2008, Budapest, Hungary. pp
173-189, �10.1007/978-3-540-78789-1_14�. �hal-01022139�
Supporting multidisiplinary software
omposition for interative appliations
StéphaneChatty
ENAC IntuiLab
Laboratoired'InformatiqueInterative LesTriadesA,rueGalilée
31055Toulouse,Frane 31672Labège,Frane
hattyena.fr hattyena.fr
Abstrat. Produinginterativeappliationsisamultidisiplinarysoft-
ware ompositionativity.This,and the nature ofuserinterfae ode,
putspartiularrequirementsonomponentompositionframeworks.We
desribeaomponentmodelthatreliesonahierarhial treeofhetero-
geneous elements ommuniatingthrough events and data ows. This
modelallows to assemble, reuse and apply late binding tehniques to
omponentsasdiverseasdatamanagement,algorithms,interationwid-
gets,graphial objets,or speehreognitionrulesat alllevels ofgran-
ularity. We desribe implementations of the model and example uses.
Finally,weoutline researhdiretions formakingthemodelmoreom-
pleteandompatiblewithmainstreamsoftwaremodels.
1 Introdution
Graphidesigners andusability experts areinreasingly involvedin thedesign
ofappliations, espeially whentheuserinterfaegoesbeyondtraditional wid-
gets. Until reently, they did it by produing speiations that programmers
triedtofollow.This proesswasnotoptimal:workwasdupliated,mistakesor
tehnialonstraintsalteredtheoriginaldesign,anditforedasequentialwork-
owbetween ators.It also impeded theredesign of appliations. If amedial
imagingompanyaquired asolutionforanalysing images,wantedto mergeit
with their image apture solution, and had onsisteny problems between the
twouserinterfaes,theyhadto reprogrammajorpartsofthesoftware.
Anemerging alternativeproess isthemultidisiplinaryprodution ofsoft-
ware[1℄.Graphidesignersproduethevisual parts,interation designerspro-
dueinterativebehaviours,andprogrammersonlyproduethefuntionalore
(data managementandalgorithms) and theoverallappliation struture. This
reduestheglobalamountofwork,eliminatesprogrammer-induedmistakesas
well asinompatibilities, and allowsfor onurrent engineering: all ators an
workinparallelandassembletheirworkjustbeforedelivering.
Inthis artilewe propose aomponent model to support thisnew proess.
Themainontributionsare:
ananalysisofhowthisproessandthenatureofinterativesoftwareallfor
asoftwareomposition model,appliableto alltypesofuserinterfaesand
Published in the Proceedings of the 7th International Syposium on Software Composition, Lecture Notes in Computer
Science. Copyright (c) 2008 by Springer-Verlag. Available online at http://www.springer.de/comp/lncs/index.html
owsforommuniationsamong omponents, aimedataddressingtheor-
respondingrequirements.
Contrasting with mostmodels that desribe graphialinterative omponents,
ourmodelisaimedatdesribingallpartsofanappliation,inludingnon-visual
interation as well as the parts that do not belong to the user interfae. We
examplifytheuseofthismodelthroughseveraldevelopmentsenarios,involving
variousdegreesofinteration.Finally,weoutlinesomeresearhdiretions.
2 Motivation: assembling interative software
Interativesoftwareishardtodevelop[2℄.Thisisinpartbeausetheuserinter-
faeperse,whihaountsforhalfofthesizeofinterativeappliations,obeys
dierentpriniples thantheotherhalf.It hasexternalontrol,dealswithstate
rather than omputation, and heavily usesreferenes beauseits objetshave
multiple interdependenies.Withimperativeorfuntionallanguages,itsobjet
behaviourstendtobesplitarossmultiplefuntions.Thearhiteturepatterns
usedforinterativeomponentsevengivethemonurrentsemantis[3℄.
Butmostofall,thewayinterativesoftwareisdesignedandproduedposesa
softwareompositionproblemthatmustbeaddressed.Ifsoftwareompositionis
aboutassemblingomponentsthat havenotbeplanned anddesignedtogether,
then building an interative appliation is a ontinuous software omposition
ativity: from the beginning, unplanned reorganisationis the rule rather than
theexeption.Moreover,itdealswithelementsproduedbyatorswithvaried
bakgroundsandmethodsatanylevelofgranularityandanydegreeofloality.
2.1 Variedstakeholders
Interativesoftwarerequiresskillsfromusabilityexperts,domainexperts,users,
programmers,interation designersandgraphialdesigners.Theirontribution
anbeveryonrete in the atualprodutionof software, whether during the
developmentorlaterduring appliation'revamping' orustomisation:
domainexpertsdenesequenesofusertasks(gamelevels,forinstane);
graphialdesignersdene allthegraphisandthegeometriallayout;
natural language grammar designers, sound designers or other speialists
mayalsoproduepartsoftheappliation;
usability experts or interation designers may dene the ne behaviour of
interativeomponents:shouldbuttonshighlightwhenoneentersthemfrom
thesideoronlywhenpressingdiretly?
usersorsupport stamayredenethelayoutoralternate inputongura-
tionstosuit theirspeialneeds.
Allofthesearetheownersofonernsthatshouldlegitimatelyformindivid-
ualomponents.All havetheirownabstrations andtoolstomanipulatethem,
Theaboveatorsdonotfollowthetraditionalshedulesofsoftwaredevelopment.
Asalreadymentioned,theirtasksarebestahievedinparallel.More,thedesign
isiterative:beauseuserneedsarediulttoeliit,iterativeproessesbasedon
prototypingandevaluationhavebeendevised.Theiterationsmayontinueand
importantdesigndeisionsmaybedelayedwhiletherestofthesoftwareisbeing
developed:fromthebeginning,theappliationisinanunplannedustomisation
phase.Thisreatesdiultiesinlargeprojetssuhasairtraontrolsystems:
one bothhas to delay interfae design issues and hoose an arhiteture very
early, whih often leads to arhiteture onits later in the projet. Only a
omponentmodelsuitingalltypesofuserinterfaeswouldalleviatethisproblem.
Interativesoftwarealsohasthelassialissuesofappliationredesign:graph-
isinsteadoftextdialogue,post-WIMP 1
,interfaesinsteadofwidgets,orfusion
ofseveralappliationsunderauniedinterfae.Theadaptationtovaryingplat-
forms (sreen size, input devies) also requires a redesign or even a dynami
adaptationof the visual layout,thedialogue sequene, oreventhe interation
style (a buttondoesnotworkthe samewithamouse oratouhsreen). This
willulminatewiththeubiquitousomputingparadigm,whenappliationswill
disovertheirexeutionontextatruntime.Insummary,softwareomposition
oursat anytimefrominitialdevelopmenttorun-time.
2.3 Componentgranularity
Theomponentsthatareassembledorinterhangedanhaveverydierentsize
and omplexity. At thelargest saleis theintegration of twoappliations into
one:anemailappliationandaWebbrowser,forinstane.Atasmallersale,a
given widgetmustsometimesbereplaed:hangingalassiretangular menu
in favourofapie-shapedonethat worksbetteron tabletopuserinterfaes,for
instane. In some asesthe replaement is at an even smaller sale: swithing
from a mouse interfae to a touh sreen interfae just requires to hange a
partof theinternalsof someinterators 2
.Forinterationdesigners thisis best
seen asahange of sub-interatorsized omponentsfrom theirown library of
omponents: in buttonsfor instane, replaing the behaviour that reatsonly
toliksinitiatedonthegraphialobjet('mouse-oriented'behaviour)withone
that allowstostartbesidethenenter('touh-sreen-oriented'behaviour).
Furthermore, these various granularities annot be handled independently:
oneoftenhastoreplaeaomponentbyanotherofaverydierentsize.Consider
adesktopmetaphor inwhihappliationsareshownaspagesin abook.When
turningthepages,youseeanimageoneahpage;butassoonasapageisat,
theimageisreplaedwiththeatualrunningappliation.Asanotherexample,
when adding animation to user interfaes so that visual hanges are not too
sudden,onehastoreplaeassignmentsof numerialvaluesbywholeanimation
1
interfaesthatdonotrelyontheWindows-Ions-Mouse-Pointingparadigm
2
omponentsthatdealwithagiveninteration:awidget,adraganddropsequene,
will makea ontinuousmovement. And thetarget itself anbe aonstant, or
an be obtained by ativating a speeh grammar rule, or through a 'wizard'
appliationthat helpstheusermakethehoie.
2.4 Crossutting onerns
Notallhangesareasloalastheonesabove.When're-skinning'auserinterfae,
theyonlyhangethegraphisbutallthevisualomponentsintheappliationare
modied.Manyfaetsofuserinterfaesarerossuttingonerns inthesense
ofaspet-orientedprogramming:graphialstyle,geometriallayout,animation,
draganddropmanagement,loalisation,timeonstants,et.Spaghettiofode
is stillthe normtoday in mostreasonable-sizeinterativeappliations beause
of this. Forinstane, buildingdrag and dropbehaviourasaomponentrather
thanablakboxoraseriesofodefragmentsisstill ahallenge.
3 Requirements on omponent models
Thesituationdesribedabovegeneratesrequirementsontheomponentmodel
used for organising appliations. Some are not new and have been adressed
individuallyinthepast.Butthenewmultidisiplinaryproessesandpost-WIMP
interationexaerbatethemandmakeitimportanttoaddressthemolletively.
Uniedframework. Beauseoftheplanningissuesdesribedabove,itisdesirable
to haveauniquemodelforallomponentsinanappliation,soasnottolimit
howand whenomponentsanbeinterhangedandonneted.This appliesto
omponentsintheuserinterfaeaswellasthefuntionalore:forinstane,an
animatedobjetsuh asasrollbarindex anbeonnetedtothe mouse,toa
lok,or tothele-loadingomponentatdierenttimes. Post-WIMPinterfae
designers rely heavily on this, whereas most user interfae omponent models
applyonlytographialinterativeomponents,andtotheirsub-omponentsas
longastheyare themselvesgraphialinterativeomponents.Thisonlyovers
veryfewompositionsenarios,andforesprogrammerstouseseveralsoftware
omposition modelsinthesameappliation.
Heterogeneity support. An interativeappliationisaheterogeneoussystemby
itself. Not only do developers ome from dierent bakgrounds, not only do
theymanipulateverydierententities,buttheirpreferredomputation models
are also very dierent. Some interation styles are best dened through state
mahines; others are easier with data ows. Graphis are often seen as pure
delarativeobjets.Somedialoguesequenesarepurelylinear(levelsinagame,
stepsinawizard)butonurrenyisalwayspresent,ifonlytoprovideanimated
feedbak.Gesturereognitionandsimilaralgorithms,likeomputations,arewell
desribed withfuntional programming,while inputhandlingalls forreative
programming[5℄.Withintheproposedmodel,itmustbepossibletobuildom-
ponents asdierent asgraphialobjets,omputations, interative behaviours
nents, as identied in the previous setion. The model must therefore havea
omponent onept that is the same at all sales, from basi instrutions to
wholeappliations, likedoesfuntionalprogramming.
Modularity. Not alluserinterfaesareonlygraphial.Somehavesound,speeh
reognition,orvideoapture. Some even hange overtime: an appliationan
atasavoieserverwhenyouareawayfromtheoe andlaunhagraphial
interfae when you use your omputer. It is important that the parts of the
framework that manage these modalities are asmodular as dynami libraries
aretoday,andthatdevelopersanhoosetousethemornot.
Behaviour heking. The model mustprovidesupport for hekingomponent
omposition,beauseinterativesoftwarealsohastheissuesoftraditionalsoft-
ware.It is partiularlyimportant to hekthe ompatibility of omponent be-
haviours,andnotonlydatatypes[6℄.
Delarativeness. Some stakeholders in the development use purely graphial
tools.Iftheyaretoontributeeiently,theymustbeallowedtomodifyappli-
ationswithoutthe helpof programmers.Thereforethemodelmust supporta
delarativestyleofomposition,thatisastyleinwhihtheexisteneofagiven
omponentat agivenloationfullydeterminesitssemantis.
External ontrol ows. Inuserinterfaes,whattriggersanationis notaon-
trol owfromthemain programbutanexternalondition:user'sation,lok
signal,et.Funtionallsdonotproperlysupportthisbeausetheyrequirethat
the soure of the ontrol owhas information about the reipient and thus is
developedafterit.Itusuallyisdeveloppedbefore:deviedriversandinterative
omponentspredateappliations. Funtionreferenesand allbaks,ortheuse
of late binding for that purpose are workarounds that sometimes indue pro-
grammers into mistakes[3℄. Events are more useful than funtions, espeially
withpost-WIMPuserinterfaes.
Conurreny. Some interative omponents require onurrent semantis, for
instanewhentwousersmanipulatetwomenusonatabletopinterfae.Conur-
renyalsoshowsmoresubtlywhentwoprogrammerssubsribetwoomponents
to the same eventwithout knowing about the other, and a third programmer
ombinestheiromponentsandexpetsthemto respetsomesequeningprop-
erties. Not only does the model need to support onurreny,it also needs to
providewaysof reasoningaboutitandexpressingorderingonstraints.
4 The I* omponent model
Toaddresstheaboverequirementsweproposeahierarhialomponentmodel
implementedin theIntuiKitmodel-orientedprogrammingframeworkand used
in an industrial ontext sine2003. Its majorfeatures are itstree of elements,
itsevent-basedommuniationmodel,anditsmodularexeutionmodel.
4.1 The element tree
IntheI*model,anappliationisatreeofelements. Anelementismadeof:
asetofnamed properties,thatstoreitsstate;
asetofnamed hildrenelements;
aninterfaethatexportsthenamesofertainhildren,propertiesandevents,
andmanagesinternal operationsonhildrenelementandproperties.
Some elements, alled atomi, are built using the host language and their
internalsarenotaessiblewithin themodel.All others,alledomponents,are
builtbyassemblingelements,reatingpropertiesanddeninginterfaes.TheI*
model onsists of thedesriptionof elementsand operationson them, ofa set
ofatomielementsthat desribeontrol,andoftheirexeutionsemantis.
Appliationstruture. Thetreeofelementsnotonlyrepresentsthearhiteture
of the appliation, but also the logialstruture of its interfae. It providesa
refereneframework forall atorsof thedevelopment.Forinstane, alassial
imageeditingprogramismadeofapaletteomponent,amenubaromponent,
a drawing area omponent, and a few pop-up dialogue box omponents. The
paletteontainsbuttons,andsoonreursively.All interatorsareomponents,
andtheirhildrenareomponents(smallerinterators)oratomielements.
Atomi elements. Inmost userinterfae models, interators areatomi. Here,
atomi elementsare smaller and moreheterogeneous: omputations, graphial
objets,speehrules,statemahines,eventnotiations,propertyassignments.
Themodelallowsamappingfromobjet-orientedlassestoatomielements,so
asto failitateintegrationwith thehostlanguage.Tobuild anatomi element
onetakesalassandturnsaninstaneofitintoanelement,seletingsomelass
membersasexportedpropertiesandsomemethodsasexportedhildren.
Graphialobjetsandgraphialontextobjets(brushes,gradients,et)are
atomi elements. Forinstane, a retangle is anelement, and thus forms ale-
gitimateappliation;runningitatuallydisplaysaretangleonthesreen.The
odebelowshowshowthisisdonewiththeIntuiKitPerlprogramminginterfae:
my $r = new GUI::Rectangle ( -x => 0, -y => 0, -width => 100, -height => 100);
$r->run;
Thesameprinipleappliesto other interation media;for instane, theIn-
tuiKit environment also implements elements that represent 3D sounds and
speeh grammar rules. One of the lassial tehniques for desribing the be-
haviourofinteratorsistheuseofnitestatemahines.These,aswellasdata-
owonnetions for ontinuous behavioursand algorithms that reognise ges-
pliationdomainobjets.These tooareelements,andareurrentlymostoften
implementedasatomielementsbyappliationprogrammers.
Elementaggregation. Componentsarebuiltbyassemblingotherelements.Some-
times mere juxtaposition is enough, for instane when building a multimodal
dialogueboxbyassemblingaretangularframe,twobuttons(YesandNo),and
a speeh grammar rule that reognises yes and no. Moreusually, elements
needtobeinteronnetedsoastoexhangeeventsorvalues,forinstanewhen
ouplingawritingzone,agesturereognitionelement,and atextelementthat
showswhathasbeenreognised;wewilllaterseehoweventanddata-owprop-
agationaredesribedthroughspeialisedatomielements.
Butinsomeasesthehildrenelementsaretoonegraintohaveasigniant
semantisassuh,andmustbeombinedtightlytoprodueasignianteet:
the state of a state mahine has a meaning only if it is also the state of a
pereptibleelement.Byextendingthemodeltosub-interatorelements,wehave
lost the naturalsharingof data between thetwo parts of aninterator. Using
eventommuniationwould beasolution,but at theost ofapoorlyjustied
memory overhead.To avoid it, atightaggregation mehanism alled property
merging is proposed, and managed in the parent omponent's interfae. The
resultisthat amemoryslotfor onepropertyonlyisused, andthispropertyis
aessibleunderdierentnamesfromthehildrenelements.Controlpropagation
whenthepropertyhangesoursasaspeialaseofdata-ow.
For instane, here is how one would desribe a button made of arbitrary
graphisforeahofitstwostates,withanelementoftypeSwith thatusesits
branhpropertytohoosewhih ofitshildrenisativeatagiventime:
$btn = new Component;
$sw = new Switch (-parent => $btn);
$on = load Element (-parent => $sw, -name => ’on’, -file => ’on.svg’);
$off = load Element (-parent => $sw, -name => ’off’, -file => ’off.svg’);
$fsm = load Element (-parent => $btn, -file => ’behaviour.xml’);
$btn->merge (-names => [$fsm->state, $sw->branch]);
$b->run;
Beauseit allowstodelaytheassoiationof graphis(or any otherperep-
tivehannel)andbehaviour,mergingisuseful formanagingheterogeneityina
groupofdevelopers.Oneaonventionhasbeenestablishedaboutthenamesof
elementsandtheirproperties,aprogrammeranbuildaomponentinwhihhe
orshejustnamesthehildrenandspeiesthemergedproperties,aninteration
designerbuildsastatemahinethatdesribeshowtheuser'sinputismanaged,
andagraphi designerbuildsaset ofgraphialobjets;thenalomponentis
assembledinaompilationorlinkingphase,justpriortoexeutingtheprogram.
Information hiding The names given to hildren elements and properties are
visibleto allhildrenofaomponent,aswellastheeventsdenedbyhildren.
isalsoamanipulationofsymboltableswithin theomponent'sinterfae.
Namehidingis forlassialsoftwareengineering purposes.Renamingis for
interativesoftwarearhiteturepurposes.Interativeomponentsaredenedin
termsof interation onepts;names suh as'button', 'ion', 'lik', 'press', or
'drag'areused.Atsomepoint,theyneedtobeonnetedtothefuntionalore
thatusesnamessuhas'le','appliation','launh',or'ship','prole','math'.
This onnetion implies two operations. First, one needs to math onepts;
for instane, a ship is represented as an ion, a prole as a button, and the
'math'operationisassoiatedtothe'press'event.Then,beausethenamesare
dierent,oneneedstotranslatethem.Thisisalledfuntionaloreadaptation;
inobjet-orientedframeworks,itisimplementedwithlasseswhoseonlyroleis
toglueobjetsofinompatibletypestogether.Here,renamingmakesfuntional
oreadaptationaframework-levelfeature, optimisedoutat ompiletime.
Another peuliarity is that there are dierent publis to hide information
from.Appliationprogrammersdonotneedtoseetheimplementationdetailsof
abutton,butdesignerswhoustomiseanappliationdoneedit;symmetrially
they do not areaboutthe external interfae of the button. This is urrently
handled at theimplementation levelonly: namehidingapplies to allprogram-
ming interfaesto theI*treelanguagessuhasPerl,C++ orJava,andnotto
interfaesinlanguagesfordesignerssuhasCSS.
4.2 Communiation and ontrol
Theprevalentexeutionmodel in userinterfaes,and partiularlypost-WIMP
userinterfaes,is thereativemodel.Thismodelusuallyoexists withthepro-
edural model brought by the programming language. To satisfy the uniform
frameworkrequirement,I*solely relies oneventommuniationandavariant:
data-owommuniation.
Events. Someelementsinthetreeareabletoemiteventswhenertainonditions
aremet.Alokemitseventsatregularintervals,agraphialobjetemitsevents
when it is liked on with the mouse, a nite state mahine when it hanges
state, an animation when it ends, a button when its state mahine hanges
state.Funtionaloreelementsanalsobesoures:aplaneemitsaneventwhen
it hanges altitude or position, a le when it hanges size, and so on. Event
subsriptionsarerepresentedasBindings,thatisatomielementsthatassoiate
ationstoonditions.ABindingisdened with:
areferenetoasoure,thatisapropertyoranelementthatmayemitevents;
aneventspeiation,thatisasoure-speiexpressionthatdesribeswhat
eventsareseleted;
areferenetoanation,thatisanelementthatisexeutedwhenamathing
eventisemitted.
my $table = find Element (-uri => ’input:/intuiface’);
my $b = new Binding (-source => $table->pointers, -spec => ’add’,
-action => "GUI::Rectangle (-x => %X, -y => %Y)");
A nite state mahine is an atomi element made of a set of bindings that
are only ative when the mahine is in a given state. Atomi ations named
Notiationallowomponentbuildersto emittheir ownevents.Others named
Assignmentsetthevaluesofproperties.Callbakfuntionsinthehostlanguage
anbeenapsulatedasationsnamedNativeCode.
TheBindingelementsmakebehaviourdelarative:onereatesaontrolow
just byaddingtheappropriateBinding.It alsohelpsto reatestate-dependent
behaviours, bymaking bindings orstate mahines ativeor notdepending on
astate,withouthavingtointroduehierarhialstatemahinesorStateharts:
hierarhyisrepresentedbytheI*tree.
Data-ow. Data-owis a speial ase of eventommuniation. Properties are
eventsoures,andatomielementsalledConnetorsareBindingsdenedfrom
twoproperties:theytriggeranimpliitationthat opies thevalueoftherst
propertyintotheseondwhen ithanges.Forinstane withthefollowingode
aretanglefollowsthengeronatouhsreen.
my $t = find Element (-uri => ’input:/touchscreen’);
my $r = new GUI::Rectangle (-width => 10, -height => 10);
my $xc = new Connector (-in => $t->X, -out => $r->x);
my $yc = new Connector (-in => $t->Y, -out => $r->y);
Atomielementsnamed Wathers are used within elementsto bind ations to
hanges of their own properties. This allows to build data-ow briks suh as
thosedesribedin[4℄or[7℄,andproduestheontrolowsassoiatedtomerging.
Thisdenitionofdata-owdoesnotonlyprovideadelarativewayofbuild-
ing behaviours. It also allows to dene a onsistent sheduling for event and
data-owpropagation,sothat mixingthem leadsto preditableresults.Imple-
mentationsofI*inludeashedulingalgorithmbasedonproperties,omparable
tothose usedinsynhronousprogramming.
5 Implementing element semantis
We have built twoimplementations of the I* model named IntuiKit Perl and
IntuiKitC++.Wenowdesribewhatsemantis theygivetoelementsandhow
theirarhiteturehelpsfullltheinitialrequirements.
5.1 A model-basedimplementation
Foreah type ofelements,anXML formathasbeendened.Forinstane, the
SVGformatisusedforgraphis.IntuiKitinludes parsersforthese formats,in
XMLles,instantiatingelementsfrom ode,orboth.
UsingXML leshasallowedto useIntuiKit in aresearhprojetasthe -
nalexeutionengineinamodeltransformationhain. Italsohelpsmanagethe
heterogeneityofatorsandtheplanningissues:graphidesignersusetheirown
tools to build graphis and export them asSVG. Programmers or interation
designers an build the rest of theappliation in ode orXML. Then onean
hooseto loadtheXML lesat runtime,thus delayingintegration tothe last
minute,ortogenerateodefromthem.UsingXMLalsoallowstomigrateappli-
ationpartsfromoneIntuiKitimplementationtoanother.Thetypialintended
usefor thisis toarryoutiterativeprototyping withthePerlimplementation,
then export the graphis,behaviours, and struture of the appliation tree in
XMLandreuse theminthenalC++development.
The manipulation of part of the tree as data les introdues preliminary
phasesintheexeutionofappliations:theloadingorinstantiationofelements,
then theirlinking,priorto exeutingthetree. Soastomaketheprogramming
interfaesforinstantiatingelementsompatible withelementreationin graph-
ialeditors,instantiationhasbeendenedalongthelinesofprototype-oriented
languages:elementsanbeopiedfromothers,thenmodied.
5.2 Modulesand rendering engines
Followingtheonstrutionofthetree,IntuiKittakeshargeofexeuting('run-
ning')it.Theassoiatedsemantisisthateahelementrepresentsaninstrution
forapartof theexeutionenvironmentnamedamodule:graphialobjetsare
renderedbyagraphialengine,speehgrammarrulesaremanagedbyaspeeh
engine,bindings,ationsandotherbehaviour-orientedelementsareexeutedby
theoremodule.Thisaddressesthemodularityrequirement:eahmoduleisin
hargeofasetofelementtypes.
Eah module denes an XML namespae and implements the assoiated
parser,providesaprogramminginterfaeforinstantiatingtheelementsitdenes,
and inludes a renderingengine for them. Leavingaside user-dened modules
that ontain user-assembled omponents suh as WIMP interators (buttons,
menus, dialogue boxes) or dials for okpits, most modules introdue atomi
elements. The ore module provides the entral onepts of the model and a
fewtypesofontrolelements:bindings,onnetors,statemahines.Othermod-
ulesareusedonlywhenrequired:aGUImoduleforgraphialobjetsandbasi
WIMPobjetssuhaswindows,mouseandursors;aninputmoduleforatypial
inputdevies; ananimationmodule foranimationtrajetories;aspeeh reog-
nitionmoduleforgrammarrules.Suhmodulesareimplementedbyreusingan
existing renderingengine,either asalibrary oraserver,and enapsulatingits
primitivesintotheexeutionmethodsofatomielements.
Usingmodulesprovidessupportforthemanagementofrossuttingonerns
whilepreserving delarativeness:to enrihaomponentwith anewmedia,one
just needs to add a hild element from the orresponding module. All other
onedimensionisthesetofmodules,theotheristheappliationtreethatdrives
therenderingin allmodules. Inourview,this isthekeyforprovidinganlear
arhitetureformultimodalappliations.
Wehaveenounteredtwotypesofrenderingengineswiththatregard.Some,
suh as OpenGL, do not store the objets they render and need to be alled
periodially.Inthisase,theI*treeservesnotonlyastheappliationstruture
but also as thebasis forrendering: one the treeis run, thegraphial module
periodially traverses the tree, updates its rendering ontext or the engine's,
andhasgraphialobjetsrenderedbytheengineasitenountersthem.Inother
words,the restritionof thetree to ontainers and graphial elements hasthe
semantisofagraphialsenegraph.Other renderingenginesdomanagetheir
owninternalstruture.Inthatasethetreeisonlytraversedonetoreatethis
struture,andtheengineisthennotiedofhangesinthetreethatonernit;
theengineatsasaserver,andoneaninterpretthis asan extensionof event
ommuniationto therenderingitself.
6 Example appliations
IntuiLab andtheirpartnershaveusedIntuiKitduringveyearsfordeveloping
dozens of interativeappliations as diverse asar dashboard and multimedia
displays,airtraontroltools,geographialinformationsystemsontabletops,
multimodal information query systemsorlotto kiosks.We desribe here some
exampleusesthat demonstratetherobustnessof theI*model.
Fig.1.Ansetoftabsforaarmultimediasystem
6.1 Skinninga visualomponent
Figure1showsthe treestrutureofaomponentthatwasbuiltforaarmul-
timediasystem. Ithasastati bakground,fourtabsthat representfour parts
ofanappliation,aSwithelement,andanite statemahine.Thetransitions
of the state mahine are bound to events from aset of keysloated near the
steering wheel,and its stateis mergedwith that of the Swith. Depending on
theSVGleusedforthegraphialelements,theresultlooksasin Figure2aor
6.2 Building a multimodaldialogue box
ThefollowingodeshowshowonebuildsasimplemultimodalYes/Nodialogue
boxfromatomielements:aretangularframe;tworetanglesandbindingson
themthatemitY(resp.N)eventswhentheyarepressedon;aspeehgrammar;
two bindings on the reognition of words by the grammar. For onision the
parent omponent does not appear here, nor the arguments that reate the
elementswithin thisparent.
my $r = new GUI::Rectangle (-x => 0, -y => 0, -width => 200, -height => 100);
my $y = new GUI::Rectangle (-x => 20, -y => 30, -width => 60, -height => 40);
new Binding (-source => $y, -spec => ’ButtonPress’, -action => "notify(’Y’)");
my $n = new GUI::Rectangle (-x => 120, -y => 30, -width => 60, -height => 40);
new Binding (-source => $n, -spec => ’ButtonPress’, -action => "notify(’N’)");
my $g = new Speech::Grammar (-grammar => ’yes-no’);
new Binding (-source => $g, -spec => [command => ’yes’], -action => "notify(’Y’)");
new Binding (-source => $g, -spec => [command => ’no’], -action => "notify(’N’)");
Thesameeventsareemittedbythisdialogueboxwhetherthemouseorvoie
isused. Thespeeh grammar,sineitis ahildelementof thedialogue box,is
only ativewhen the box is ative; the same holds for the retangles and the
bindingsofourse.
6.3 Appliation design and development
Figure3illustratestheuseofIntuiKitinaphaseofthemultidisiplinaryproess
desribed in the introdution of this artile. The illustrated air tra ontrol
projet involved virtual paper: objets that felt like paper strips through a
ombination of visual eets, animationand gesturereognition. A rstphase
ofiterativedesignyieldedapaperprototypethatoutlinedthestrutureandthe
behaviour of the appliation. Designers and programmersused this prototype
to dene an I* tree and give names to elementsto beprodued by designers.
Theneahstartedtoprogram,draworotherwisebuildtheirelementsandgive
them the appropriate names.Fortest purposes, someone in the groupquikly
fromtheSVGle(left).Whenthenaldatamanagement,behaviour,animation
andgraphialelementswereready,theprogrammersjust hadtoput XMLles
deliveredbydesigners atthe rightplae, and testthe appliation(right).This
appliationlaterhadseveralsets ofgraphisfordierentustomersin Europe.
Measurementsarried outonthisasestudy (omparisonwithaprojetof
similar size and omplexity, by thesame team,using alinear proess)showed
a redutionof projetduration by about50%, expenses by about 30%,and a
dramatidereaseofoordinationosts(estimatednumberofphonealls)[1℄.
Fig.3.ATCappliationbeforeandafternalintegration
6.4 Transferringmore tasksto designers
Intheaboveexample,graphidesignersonlyproduedgraphis.However,some
arewillingtotakemoretasksfromprogrammers,andpartiularlyvisuallayout
anditsadaptationtosizehanges.Wehavedesignedartistiresizing[9℄,ateh-
niquewheregraphidesignersprovideexamplesofgraphialobjetsatdierent
sizes,andthesysteminterpolatestheirappearaneforanyhosensize.
ImplementingtheartistiresizingalgorithmwithIntuiKitwasasimpleappli-
ationoftheI*model:webuiltanewatomielementthathaspropertieswidth
and height, implements the artisti resizing algorithm, is dened by passing
it the examples as hildren elements, and then behaves as a single graphial
element. This new element an then be plaed in the tree wherever a resiz-
ablegraphi elementis desired,and its properties onnetedto thesize of the
available window.Fromthenon, thegraphialobjetadapts tothe size ofthe
window,respetingthedesigner'snon-lineartransformations.
6.5 Input management
One of the future hallenges for interative software is that when building an
appliation,developerswillnothaveapreiseideaofwhatinputdevieswillbe
availableatruntime.WehavebeenabletobuildanIntuiKitmoduletoaddress
are out of ontrol of developers.It makessense to deide that the appliation
tree is just an element of a largerI* tree that ontainsthe omputer devies.
Therefore,wejust hadtoreateanewelementsetandelementdisoveryfun-
tionstoallowprogrammerstotest anduseinputdevies[8℄.Usingatehnique
used forommuniatingdynamidata reationfrom thefuntionaloreto the
userinterfae,the hot-pluggingof deviesis reportedasaneventbytheset of
inputdevies, whihis anautomatially managedomponentthat ontainsall
input devie elements. In that ontext, multimodal fusion, that is the ombi-
nationof inputs from dierentsoures, beomes amatter of reatingelements
thatsubsribetodierentsouresandimplementoneombinationpoliyorthe
other:timewindows,forinstane.
7 Researh diretions
TheI*modelanditsimplementationshaveallowedustoturninnovationinuser
interfaesintoamoreindustrialativity.Butquestionsremaintobeaddressed,
to givethemodelmoresolid foundationsand to overissuesurrentlynotad-
dressed. Firstof all, the ontrol strutures desribedabove are insuient for
building allof thefuntional ore; thisfores developersto build it asaset of
atomi elements, and breaks the requirement for auniform model. Similarly,
one needs to devise a data-passing sheme that makesthe implementation of
data-owelementsaseasy asfuntions in afuntional framework,aswell asa
typing system for ontrolling bindings and onnetions. We may also need to
proposeaservieall ommuniationsystemontopofeventommuniation,
forthefewaseswheretheallerisdened aftertheallee.
Inanotherdiretion,deningaformalsemantisfortheI*treeanditsom-
muniationswouldprovidedeveloperswithanunmbiguousunderstandingofhow
their omponents behave,and help ompare with moregeneralframeworks.It
ouldalsoserveasthebasisforompilingomponentsratherthanjustinterpret-
ingthem:whereasduringexeutionIntuiKit,eveninPerl,omparesfavourably
withallrihgraphisframeworks,interpretationtimesarenotsatisfatory.
Finally, astrong similitude appears between elementsand proesses in re-
ativesystemsorotheronurrentmodels, but theonsequenesofhoosinga
given semantisneed to be explored. In partiular, we must understand what
level of ontrol programmers and designers need over the sequening of their
ations,andhowittsin theavailablemodelsofonurreny.
8 Related work
Many omposition senarios and requirements have been studied by user in-
terfaesoftware speialists. Theproposed solutionseither havebeenhigh level
guidelines or patterns foused on a given requirement: for instane MVC or
straintsfordesribinguserinput,layoutoranimation[4,11℄;hierarhiesofvisual
omponentsasinSelf[12℄;theJavasoure/listenerandQt signal/slotpatterns
foreventommuniation.Mostsuhpatternsimplementareativeomposition
modelontopofanexistingfuntion-orientedlanguage(usinginheritane,forin-
stane),thus notaddressingtheuniformframeworkrequirement.Noneofthese
haveexploredtheheterogeneityrequirement.
Reentprodutssupportthenewdevelopmentproesses.Flashallowsgraph-
ialdesignerstobuildompleteappliations;programmersanextendtheseus-
ing a dediated language oreven a mainstream language. Other solutions for
Webappliations,suhasSVG+JavasriptorMirosoftSilverlight,takeasim-
ilarhybridapproah.However,suhsolutionsareveryspeito graphis,and
donotproposeauniedframeworkforomplexappliations:Flashhaslimited
enapsulationfeaturesandtheothersfallin thehybridmodelategory.
Solutionsforprogramminguserinterfaeshavebeenproposedfornearlyev-
ery programming paradigm: objet-oriented programming of ourse, but also
reativeprogramming[13℄,funtionalprogramming[5℄, et.Manyofthese ap-
proahes,withthenotableexeptionofreativeprogrammingandtheSmalltalk
language[14℄,onsist in providing patterns that extend oralterthe semantis
oftheoriginalframeworkto supportinterativeomponents.
Withtheadventoflargeheterogeneoussystems[15℄,researhonsoftwarear-
hitetureandsoftwareompositionaddressesrequirementsthatareverysimilar
toours.TheI*treeanbeomparedtothehierarhyofomponentsintheFra-
talframework[16℄;omponentinterfaes,inludingtheexperimentalbehaviour
inspetion features, andsome aspetsof internal ontrol in I*omponentsnot
desribed hereanbeompared to Fratalmembranes.Themain dierene is
probablythat Fratalisservie-orientedwhile I*isevent-oriented.Aspetpro-
gramming[17℄alsosharesrequirementswithI*,partiularlymodularity(forhan-
dlingross-uttingonerns)andexternalontrol.Oneaninterpretpoint-uts
andadviesastheI*bindingofationstopartiularsoures,withapartiular
event speiation language. The main dierene is that this event ommuni-
ationis themainontrol onstrutioninI*whereasaspet programminguses
it only for partiular software engineering ases. I* an also be onsidered as
anarhiteturedesriptionlanguage,but onethatwouldaim atdesribingthe
internalarhitetureofomponentsaswell,downtothelevelof instrutions.
9 Conlusion
We haveanalysed in this artile how thenew multidisiplinaryproesses used
for interative software inuene software arhiteture and omposition.They
reate a need for aomponent model that unies the heterogeneousonepts
used by the various stakeholders, that ombines with the moretraditional re-
quirements of user interfae software. Wehave desribed the main features of
theI*omponentmodelthataddressestheseissues.Inpartiular,theabilityto
developmentproesses.Oneofthemainhallengesnowistoompareourmodel
withmoremainstreamresultsinsoftwareengineering.Understanding thelinks
between interative software and other heterogeneous systems may prove fer-
tile, aswellasomparing I*withformal models fordesribingonurreny.In
the longterm, ourobjetive isto reonile userinterfae designwith software
engineeringtheories,pratiesandtools.
Aknowledgements
This work was partly funded by the Frenh government through the ITEA
EmodeprojetandbyAgeneNationaledelaReherhethroughtheDigitable
andIstarprojets.L.Bass,R.KazmanandS. Conversyprovidedusefuladvie
onthisartile.Theanonymousreviewershelpedalottoimproveit.
Referenes
1. Chatty,S.etal: Revisiting visualinterfae programming:reating GUItoolsfor
designersandprogrammers In:Pro.oftheACMUIST,Addison-Wesley(2004)
2. Myers,B.A.: Whyare human-omputerinterfaes diultto designand imple-
ment?TehnialReportCMU-CS-93-183,CarnegieMellonUniversity(1993)
3. Chatty,S.: Programs=data+algorithms+arhiteture. In:Pro.ofthe2007
onfereneonEngineeringInterativeSystems.LNCS,Springer-Verlag(2008)
4. Chatty,S.: Deningthebehaviourofanimatedinterfaes. In:Proeedingsofthe
IFIPWG2.7workingonferene,North-Holland(1992)95109
5. Elliott,C.,Hudak,P.:Funtionalreativeanimation. In:InternationalConferene
onFuntionalProgramming.(1997)
6. Aot,J.etal.: Formaltransduers:modelsofdeviesandbuildingbriksforthe
designofhighlyinterativesystems.In:Pro.ofDSVIS'97,Springer-Verlag(1997)
7. Dragievi,P.,Fekete,J.D.: Supportforinputadaptabilityintheiontoolkit.In:
ProeedingsofICMI'04,ACMPress(2004)212219
8. Chatty,S.etal.: Multipleinputsupportinamodel-basedinterationframework.
In:ProeedingsofTabletop2007,IEEEomputersoiety(2007)
9. Dragievi, P.etal.: Artistiresizing:A tehniquefor rihsale-sensitive vetor
graphis. In:ProeedingsoftheACMUIST,Addison-Wesley(2005)
10. Coutaz,J.: PAC,animplementationmodelfor dialogdesign. In:Proeedings of
theInterat'87Conferene,NorthHolland(1987)431436
11. Myers,B.: Separatingappliationodefromtoolkits:Eliminatingthespaghettiof
allbaks. In:ProeedingsoftheACMUIST,Addison-Wesley(1991)
12. Smith, R.B.etal: The Self-4.0User Interfae. In:OOPSLA'95 onferene pro-
eedings,Addison-Wesley4760
13. Clement, D., Inerpi, J.: Programming the behavior of graphial objets using
esterel. In:ProeedingsofTAPSOFT'89,LNCS352,SpringerVerlag(1989)
14. Kay,A.C.: TheearlyhistoryofSmalltalk. ACMSIGPLAN(3)(1993)6975
15. Hardebolle,C.etal.: Ageneri exeutionframeworkfor modelsofomputation.
In:ProeedingsofMOMPES2007,IEEEComputerSoiety(2007)4554
16. Bruneton, E. et al.: An open omponent model and its support in Java. In:
ProeedingsofCBSE2004.LNCS3054, Springer-Verlag(2004)
17. Kizales,G.: Aspet-orientedprogramming. ACMComp.Surveys28(4es)(1996)
