A Model-Based Approach for Dynamically Distributing Graphical User Interfaces Based on their Properties, Graphs, and Scenarios

HAL Id: tel-01273879

https://hal.archives-ouvertes.fr/tel-01273879

Submitted on 18 Feb 2016

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Jérémie Melchior. A Model-Based Approach for Dynamically Distributing Graphical User Interfaces Based on their Properties, Graphs, and Scenarios. Ubiquitous Computing. Université catholique de Louvain, 2016. English. �tel-01273879�

Computing Science and Engineering Pole

A Model-Based Approach for Dynamically

Distributing Graphical User Interfaces Based on

their Properties, Graphs, and Scenarios

J´er´emie Melchior

Thesis submitted in partial fulfillment of the requirements for the Degree of Doctor in Computer Science

Examination committee:

Prof. J. Vanderdonckt, Co-supervisor (UCL-Louvain School of Management & ICTeam, Belgium)

Prof. P. Van Roy, Co-supervisor (UCL-Computing Science and Engineering Pole & ICTeam, Belgium)

Prof. K. Mens, (UCL-Computing Science and Engineering Pole & ICTeam, Belgium) Prof. K. Coninx, (Hasselt University, Hasselt, Belgium)

Prof. J. Coutaz, (Universit´e Joseph Fourier, Grenoble, France)

Prof. C. Pecheur, President of the jury (UCL-Computing Science and Engineering Pole & ICTeam, Belgium)

This thesis has been made possible thanks to my advisors Jean Vander-donckt and Peter Van Roy. I would like to thank them for their motivation, ideas and for all their contributions. I would also like to thank the other members of the jury: Professors Jo¨elle Coutaz, Karin Coninx and Kim Mens. Many thanks to all my colleagues at the Operations and Information Management (OI) Department of the Louvain School of Management (LSM) and at the Computing Science and Engineering Pole (INGI) for their help, comments and friendship, especially Boriss Mej´ıas and Ruma Paul for the great help and devotion around my usage of Beernet, Yves Jaradin who ported Mozart to Android and worked on the Mozart 2 DSS, S´ebastien Doeraene, Anthony G´ego, Guillaume Derval and Benoit Daloze for the help regarding Mozart2. I am also greatly thankful to Tanguy Lepoutre for his help with the demonstrations and for his support until the end of the thesis. Finally I would like to thank my friends and my family, especially my dear love ´Emilie and my daughter ´Elise.

The thesis has been supported by:

ˆ The UsiXML project standing for USer Interface eXtensible Mark-up Language. ITEA2 European Project ITEA2-2008-0026. January 2010 - March 2013.

ˆ The SELFMAN project standing for Self Management for Large-Scale Distributed Systems based on Structured Overlay Networks and Com-ponents. European FP6 (Sixth Framework Programme, Priority 2, In-formation Society Technologies) Project under convention No. 034080. September 2008 - September 2009.

ˆ The SERENOA project. European FP7 (Seventh Framework

Programme) Project under grant agreement No. 258030 (FP7-ICT-2009-5).

ˆ The Usidistrib FIRST SpinOff Project under convention No. 1217843. May 2013 - May 2016.

1 Introduction 17

1.1 Distributed Tasks . . . 18

1.2 Models, Approaches, Software supports . . . 25

1.2.1 Models . . . 26

1.2.2 Approach . . . 28

1.2.3 Software support . . . 28

1.3 Thesis Statement . . . 29

1.3.1 Single VS Multiple distributions . . . 29

1.3.2 Scope . . . 30

1.3.3 Contributions . . . 32

1.4 Support for mobile devices . . . 33

1.5 Organization of the Thesis . . . 33

2 Related Work 37 2.0.1 Meta-UI for Ambient Spaces[COU06] . . . 42

2.0.2 Mobile and Intelligent Environments[DEE10] . . . 44

2.0.3 The Fiaa Platform Model . . . 44

2.0.4 An AUI model to support DUIs . . . 46

2.0.5 FRESCO . . . 46

2.0.6 CESAM . . . 46

2.0.7 Windows snipping in MME[HUT07] . . . 48

2.0.8 ARIS . . . 48

2.0.9 IMPROMPTU . . . 49

2.0.10 GUMMY . . . 49

2.0.11 Light-weight Services . . . 51

2.0.12 MASP . . . 51

2.0.13 Web sites and applications . . . 53

2.1 The related work along the three dimensions . . . 54

2.1.1 Modeling for Distributed Systems . . . 54

2.1.2 Tools for creating DDGUIs . . . 59

2.2 Summary . . . 60

2.2.1 List of shortcomings . . . 60

2.2.2 Comparison of software support . . . 62

palette . . . 79

3.2 Model-based approach for DDGUIs . . . 80

3.2.1 Modeling in software engineering . . . 80

3.2.2 Core models . . . 81

3.2.3 User Model . . . 81

3.2.4 Device Model . . . 82

3.2.5 The selection mechanism . . . 82

3.2.6 EBNF grammar . . . 83

3.3 Distribution Graph . . . 84

3.3.1 Graph . . . 85

3.3.2 Attribute-Value Pair pattern . . . 85

3.3.3 Vertices . . . 86

3.3.4 Arcs . . . 87

3.3.5 Distribution Graph . . . 88

3.3.6 Environment . . . 89

3.4 Behavior of a Distributed System . . . 89

3.4.1 Events . . . 89

3.4.2 Execution trace of the distributed system . . . 90

3.4.3 Event-Condition-Action pattern . . . 90

3.5 Graphical representation . . . 91

3.6 Application Graph . . . 92

3.7 Distribution Primitives . . . 97

3.7.1 Local User Interface actions . . . 97

3.7.2 Global UI actions . . . 98

3.7.3 Distributed actions . . . 103

3.7.4 Behavior behind the User Interface . . . 108

3.7.5 Distribution mechanisms . . . 109

4 Specifications of the toolkit 111 4.1 Definition of a DDUI . . . 111

4.2 Specification of JayTk . . . 112

4.2.1 Distribution primitives . . . 114

4.2.2 Events . . . 114

4.3.1 P2P network . . . 115 4.3.2 Coherent storage . . . 116 4.3.3 Atomicity . . . 118 4.3.4 Failure Detector . . . 119 4.4 Applications . . . 121 4.4.1 Mozart applications . . . 121 4.4.2 JayTk applications . . . 122 4.5 Conclusion . . . 122 5 Implementation of JayTk 125 5.1 Structure of the toolkit . . . 126

5.2 Running as a daemon . . . 127

5.3 Mozart Environment . . . 127

5.3.1 Communication between devices . . . 128

5.3.2 Mozart for all . . . 130

5.4 Implementation of the actions . . . 131

5.4.1 Using actions through different ways . . . 131

5.5 Porting Mozart to Android . . . 133

5.6 Implementation of JayTK on other OS’s . . . 134

5.6.1 Mozart . . . 134 5.6.2 Android . . . 136 5.6.3 Windows Phone . . . 137 5.6.4 Windows . . . 137 5.7 Conclusion . . . 138 6 Case Studies 141 6.1 DistribuChat . . . 141 6.1.1 Specification . . . 141 6.1.2 Implementation . . . 141 6.2 DeTransDraw - DeTransDrawid . . . 142 6.2.1 Specification . . . 143 6.2.2 Implementation . . . 144 6.2.3 Evaluation . . . 145 6.3 Mobictionary . . . 147 6.3.1 Specification . . . 147 6.3.2 Implementation . . . 149 6.3.3 Evaluation . . . 158 6.4 CarReservation . . . 161 6.4.1 Specification . . . 161 6.4.2 Implementation . . . 161 6.4.3 Evaluation . . . 162 6.5 Conclusion . . . 163

8.3 Future Work . . . 173 8.4 Final Word . . . 174

1.1 Evolution in capability of computing devices . . . 17

1.2 Quarterly sales market share in combined tablet and PC cat-egory. . . 18

1.3 Graph of the situation in which a tablet, an eReader and a smartphone are used by US inhabitants.[NIE11] . . . 19

1.4 Natural world vs. digital world [GRO05] . . . 19

1.5 An example of application molding . . . 20

1.6 The Scoop-and-Spread technique of HyperPalette . . . 21

1.7 WinCuts allows users to only display the areas they want to see. . . 22

1.8 Children playing pictionary on a white board. . . 22

1.9 A picture of the board of Pictionary. . . 23

1.10 An example of board for the GotG[GotG]. . . 25

1.11 Schema of the dimensions. . . 26

1.12 JayTk based on Beernet and implementing the concepts of the thesis. . . 32

1.13 Outline of the thesis . . . 34

1.14 Thesis roadmap . . . 35

2.1 Comparison between toolkits supporting DUIs[ROU06] . . . . 43

2.2 The ergonomic aspect of the User Interface[DEE10] . . . 44

2.3 The Fiaa Platform Model resource model from [QIU09] . . . 45

2.4 Their AUI model with the DUIs perspective from [PENA11B] 46 2.5 A print screen of the CESAM prototype with two devices connected[ROU06B]. . . 47

2.6 An application cut into parts . . . 47

2.7 Example of simplicity gains from research on DUIs[HUT07] . 48 2.8 The iconic map in ARIS[BIE04] . . . 48

2.9 Screen shot of the IMPROMPTU’s UI[BIE08] . . . 49

2.10 IMPROMPTU’s extra-UI for windows sharing (a) and dis-playing (b) [BIE08]. . . 50

2.11 The three main dialogues of the Gummy tool[MESK08]. . . . 50

2.21 A workflow representing the life cycle of a distributed task. . 58

2.22 Comparison of software support . . . 62

2.23 Update of the comparison of Meta-UI[COU06]. . . 63

2.24 Comparative Analysis of additional UIs: Their Interaction Styles . . . 65

2.25 Example of metaphors for MOVE and MERGE operations . . 67

2.26 Platform experience of participants. . . 69

2.27 Preferred interaction styles . . . 69

2.28 Distribution of participants’ comments for all distribution primitives . . . 70

2.29 Distribution of participants’ comments for all interaction styles 71 2.30 Examples of some possible touch and multi-touch gestures. . 72

2.31 Links between shortcomings and requirements. . . 74

3.1 The structure of common distributed applications . . . 76

3.2 The structure of a distributed application . . . 76

3.3 An example of a non-distributed application with and without distribution support. . . 77

3.4 An example of a distribution-aware application . . . 78

3.5 The structure of an application using the JayTk . . . 79

3.6 Modeling in software engineering and in HCI . . . 80

3.7 Device model . . . 82

3.8 EBNF grammar for the main terms . . . 84

3.9 Example of distribution graph for the example of the Painter’s Palette . . . 85

3.10 Evolution of the distribution graph when a second device has appeared . . . 85

3.11 An example of distribution graph with 2 users and 3 devices . 88 3.12 The graphical representation of the Event-Condition-Action pattern[PAS07] . . . 90

3.13 Representation of a distribution graph with two users and two devices . . . 91

3.14 Representation of a distribution graph with two users and two devices . . . 91

3.15 Representation of the same DG after disappearance of u2 and

d2 . . . 92

3.16 Representation of the same DG after disconnection of user vertex u1 . . . 92

3.17 Representation of the same DG after disappearance of u1 . . 92

3.18 Simple example of Distribution Graph. . . 93

3.19 Simple Application Graph from the Distribution Graph of Figure 3.18 . . . 93

3.20 Concepts of the thesis . . . 110

4.1 Global view of Beernet’s architecture[MEJ10] . . . 116

4.2 Consensus atomic commit on a DHT. . . 118

4.3 How mozart applications usually create their GUI. . . 121

4.4 How mozart applications create their GUI with JayTk. . . 121

4.5 Comparison of an application created without and with JayTk 122 5.1 Structure of the toolkit . . . 126

5.2 An example of GUI created with QTk . . . 127

5.3 Example of a command-line user interface. . . 133

6.1 DG of the DistribuChat demonstration . . . 142

6.2 The GUI used for the DistribuChat’s example . . . 142

6.3 The GUI of DistribuChat has been split into pieces: 1) the status of the chat, 2) the send button, 3) the chat window, 4) the typing window . . . 143

6.4 DG of DTD and DTDid demonstration . . . 143

6.5 The basic UI of the DTD application . . . 144

6.6 The basic UI of the DTDid application . . . 145

6.7 On the left, the user is in Asking for locks state and the handles of the big square are red. On the right, the user is in Got locks state with the handles in black. . . 146

6.8 The state diagram of a user . . . 146

6.9 The structure of DeTransDraw and DeTransDrawid applica-tions . . . 147

6.10 Distribution Graph of the Mobictionary demonstration. . . . 148

6.11 User Interface of the observers. . . 150

6.12 User Interface of the drawer. . . 150

6.13 User Interface of the guessers. . . 151

6.14 Creation of a game when no game is already started. . . 151

6.15 Pseudo-code for creating initial UI. . . 152

6.16 Second state. Player 1 and 2 are connected. . . 153

6.17 Pseudo-code for updating UI after game creation. . . 153

6.18 Creation of a game when no game is already started. . . 153

Journal article

[MEL12-IJHCI] Melchior, J., Vanderdonckt, J., and Van Roy, P.

A Comparative Evaluation of User Preferences for Extra-User Interfaces, In the International Jour-nal of Human-Computer Interaction (IJHCI), 28:11, Taylor & Francis, pp. 760-767, 2012.

[MEL11dui2] Melchior, J., Vanderdonckt, J., and Van Roy, P. Dis-tribution Primitives for Distributed User Interfaces, In Distributed User Interfaces: Designing Interfaces for the Distributed Ecosystem, Human-Computer In-teraction Series, Springer-Verlag, London, pp. 23-31, 2011.

Conference papers

[MEL09] Melchior, J., Grolaux, D., Vanderdonckt, J., and Van Roy, P. A Toolkit for Peer-to-Peer Distributed User Interfaces: Concepts, Implementation, and Applications, In Proceedings of the 1st ACM SIGCHI Symposium on Engineering Inter-active Computing Systems (EICS 2009), ACM Press, New York, pp. 69-78, Pittsburgh, PA, USA, July 15-17, 2009.

[MEL11] Melchior, J., Vanderdonckt, J., Van Roy, P. A Model-Based Approach for Distributed User Interfaces, In Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interac-tive Computing System (EICS 2011), Pisa, Italy, June 13-16, 2011.

Primitives for Distributed User Interfaces, In Proceedings of the Distributed User Interfaces CHI 2011 Workshop (DUI 2011), ACM Press, New York, pp. 29-32, Vancou-ver, British Columbia, Canada, 2011.

[MEL12dui] Melchior, J., Mej´ıas, B., Jaradin, Y., Van Roy, P., and Vanderdonckt, J. Improving DUIs with a decentralized approach with transactions and feedbacks, In Proceedings of the Distributed User Interfaces CHI 2011 Workshop (DUI 2012), ACM Press, New York, pp. 65-68, Austin, Texas, USA, 2012.

Doctoral consortium

[MEL11dc] Melchior, J. Distributed User Interfaces in Space and Time, In Proceedings of the 3rd ACM SIGCHI Symposium on En-gineering Interactive Computing System (EICS 2011), Pisa, Italy, June 13-16, 2011.

Recommandation to W3C

Vanderdonckt, J., Tesoriero, R., Mezhoudi, N., Motti, V., Beuvens, F., Melchior, J.,

W3C, 08 April 2014,

http://www.w3.org/TR/abstract-ui/

SELFMAN Self Management for Large-Scale Distributed Systems based on Structured Overlay Networks and Components. D5.9: Distributed mobile application on gPhone.

USIXML USer Interface eXtensible Mark-up Language.

D1.3: UsiXML definitions.

USIXML USer Interface eXtensible Markup Language

(UsiXML), W3C Working Group Submission 1 February 2012, Vanderdonckt, J., Beuvens, F., Melchior, J., Tesoriero, R., W3C,

1 February 2012,

http://www.w3.org/wiki/images/5/5d/ UsiXML submission to W3C.pdf

Introduction

The days when the desktop computer was the only computing device used by one single user at a time to work and play are over. The more compact a computing device is, the less capable it will be. A few years ago, the difference in capability between a phone and a PC was very constraining as depicted in the left-hand image of Figure 1.1.

Figure 1.1: Evolution in the capability of computing devices compared to their size.[PIE04]

The use of a cell phone was restricted to very basic games and appli-cations. Nowadays, computing devices offer more capabilities than in the past, even for the most portable ones (Figure 1.1) which are more frequently purchased over time (Figure 1.2).

In the right-hand image of Figure 1.1 we can see that the evolution of computing devices reduces the difference in capability between the more portable computing devices.

In a few years this difference may completely disappear if the trends keep going. The term luggability is used in this figure to describe the ease of portability of a computing device. Although these evolutions are considered significant, applications are still designed for a single computing device to be used by only one person at a time.

Figure 1.2: Quarterly sales market share in combined tablet and PC category.[VMDE13]

This assumption is becoming less and less true: a single user shares the time across different computing devices (Figure 1.3) and the same computing device can be used by different users. Users also more frequently carry out distributed tasks in many domains of human activity (e.g., management, finance, accounting, learning, gaming, ...).

1.1

Distributed Tasks

Applications created for desktop computers allow a human called the user to use a computing device to accomplish a certain goal called a task. However certain tasks can be very complex and can require more than one computing device or more than one user. A task that requires or benefits from more than one computing device is called a distributed task. The computing de-vices can either be used at different times, sequentially, or at the same time, concurrently. The definition of a distributed task is now provided.

Definition 1. A distributed task [DT] is a task that is accomplished with the help of more than one computing device.

To illustrate distributed tasks, we will give three running examples which will help the reader to understand the benefits of using multiple computing devices to accomplish a task. The first use case is a drawing tool that can be distributed in space. The second use case is the Pictionary game, which is naturally distributed in space and among users. And the last use case is a game based on the concept of the Game of the Goose and the Snakes and Ladders because it can show how distribution can be added to a non-naturally distributed game.

Figure 1.3: Graph of the situation in which a tablet, an eReader and a smartphone are used by US inhabitants.[NIE11]

Use Case 1: the Painter’s Palette

Here is a very basic drawing tool such as Microsoft Paint. Let us call it: the Painter’s Palette. Such applications are almost always designed for a single user with only one computing device: a desktop computer or a laptop. The common interaction for such applications is with a mouse: to choose the type and color of the pen, and to choose a shape (e.g., rectangle, ellipse, square, ...). However in the natural world we need several objects in order to create a painting. Figure 1.4 compares how to paint in the natural world and on a computing device.

Figure 1.5: An example of application molding

Definition 2. Distributed User Interfaces [DUIs] enable end users to distribute any user interface element at design- and/or run-time across different users, across different computing platforms, and across different physical environments.[VDD10]

There are very few solutions allowing an application to use several com-puting devices simultaneously. Most of these solutions have not been re-leased for developers to create their own applications. The current situation is to create several applications (one per computing device) and allow them to communicate together. Even such inter-applications communication is not easy to implement for a developer. So today, we think that there is no tool that allows developers to create distributed graphical user interfaces (DGUI) in a straightforward manner. Here is a definition of a DGUI as a specific DUI.

Definition 3. A Distributed Graphical User Interface [DGUI] is a DUI in which all components are only graphical components.

DGUIs and other DDUIs are said to be dynamic if the distribution occurs at run-time, and static if the distribution occurs at design-time. We will refer to Dynamic Distributed Graphical User Interfaces (DDGUI) if it is

dynamically distributed as opposed to Static Distributed Graphical User Interfaces (SDGUI). The thesis focuses on DDGUIs.

Definition 4. A Dynamic Distributed Graphical User Interface [DDGUI] is a DGUI that can be dynamically distributed at run-time. The whole distribution is not established at design-time.

Definition 5. A Static Distributed Graphical User Interface

[SDGUI] is a DGUI where the whole distribution is established at design-time and cannot be changed at run-design-time.

To allow this kind of distribution we would like to create a software sup-port that will connect computing devices together, and supsup-port distribution across them, e.g., to allow the UI to move from one computing device to another.

There are indeed applications such as Paint.NET [Paint.NET], Microsoft Visual Studio [VS], TeXnicCenter [TeXnicCenter] making it possible to sep-arate toolbars from the main window and to move them within the desktop, which itself could be decomposed into several viewpoints, such as in Compiz Fusion for Linux[Compiz], nVidia nView for Windows [nView], AMD/ATI Hydravision which is now AMD Eyefinity[HydraVision, Eyefinity]

These applications force the distributed elements to stay on the same computing devices and do not allow these toolbars to move from one com-puting device to another.

HyperPalette [AYA00] connects a small computing device which is a pointing device acting like a gesture command related to a real world action such as copy. The Scoop-and-Spread technique (Figure 1.6) allows users to cut some elements from a virtual drawing board and paste them somewhere else.

Figure 1.7: WinCuts allows users to only display the areas they want to see.

Use Case 2: Pictionary

Pictionary[Pictionary] is a guessing word game invented in 1985 by Robert Angel. An illustration of people playing Pictionary is depicted in Figure 1.8.

The game is designed for at least 4 players and can be played in teams. A player receives a word which the other players have to guess. The drawer can use a pen to draw sketches on paper. The other players attempt to say the correct word. The first player to guess the word gets a point for the team and the turn is over.

The board game

Pictionary was released as a board game and is sold by Hasbro and Mattel. The board is constituted of sequence of squares. Each square has a letter identifying the type of picture to be drawn (Figure 1.9).

Figure 1.9: A picture of the board of Pictionary.

Each team or player gets a piece that is placed on the first square of the board.

The objective of the game is to reach the last square of the board. At each turn the players from one team must guess the word or phrase being drawn by their current player. The role rotates with each word.

The drawer gets a card out of a deck. and draw sketches that help other players to guess the word that is on the card. The drawings cannot contain any numbers or letters, and no verbal clues can be given to help the other players guess. Their teammates try to guess the word the pictures are intended to represent and shout their proposals out loud.

There are five types of squares: P, O, A, D and AP. ˆ P: draw a person, a place or an animal

ˆ O: draw an object ˆ A: draw an action

Mobictionary

We have decided to choose this case study because the game is naturally distributed across these players. Depending on the role, the player is given a task to accomplish. A player is either the drawer or the guess player.

Each player has a different UI depending on the role. There are three roles: game manager, drawer and observer.

The game manager gives the word to the drawer and starts the timer. The drawers see the word which they need to make other players guess. They can use a pen on a drawing area. The other players can see the current drawing without the ability to draw anything.

There can be observers to the game. They can also see the drawing but they cannot play.

A timer is available on any UI for players and observers to see the time remaining before the end of the turn.

Use Case 3: Game of the Goose

Finally we will introduce another use case based on the Game of the Goose.

Game of the Goose

The Game of the Goose[GotG] is a board game created during the 16th century as family entertainment. The board consists of a sequence of 63 consecutively numbered squares (Figure 1.10).

They are usually arranged in a spiral. There are one or two dice. The current player throws the dice to move forward on the board.

Some squares have a goose depicted on them. The player who lands on a square with a goose is allowed to move again by the same distance. Some other squares have a different action associated to them: a bridge or a penalty. A bridge moves the player forward to another square. A penalty brings the player back to a previous square, or makes the player skip one or more turns.

Figure 1.10: An example of board for the Game of the Goose[GotG].

The distributed game

For our case study we would like to get inspiration from the Game of the Goose.

In our distributed game, every square of the board is a different game. We have added one role: the manager of the board. The manager can change the game affected to a square during the game itself. We think that this will create more fun to keep the competition alive.

1.2

Models, Approaches, Software supports

There are initiatives in which a software application allows some particular distribution. However, there is no way, either synthetic or organized, to specify and design a DDGUI

In order to foster an approach that is not tied to a particular distributed task or to a particular distribution, let us introduce three dimensions to help us organize the research and analyze related work: The first dimension is the models that describe and define the concepts of interest. Then there is the approach which can be followed to use the models efficiently. And finally, comes the software support which can be either a toolkit or a software application.

The software support uses the approach which is itself based on the models (Figure 1.11).

There are several entities that are part of a computing system and can communicate with each other. In order to understand, use, and support the whole computing system, we need to describe these entities to reason about them.

The main entities of a computing system are the computing devices. The evolution goes from one to many computing devices, leading to new capabilities. We need to model the different kinds of computing devices according to their size, their weight, their format, their capabilities and other aspects. Or, in short, any physical property of interest.

Computing devices range from a digital alarm clock or household appli-ances to cell phones and computers. Any domestic element could virtually be controlled from a smartphone or a computer (e.g., a television, some light, an audio player, a fridge, a microwave oven or the curtains of a window). According to research there are now more than one computing device per person[DEA08]. Instead of a single personal computer, people use several computing devices including desktop computers, laptops, mobile phones, digital cameras and media players. Applications could be made aware of all these computing devices and be able to manage information and activities across them. The next evolution could be the interconnection of all these computing devices wherever they are, in private, public or work spaces.

A closer look at the evolution of the market shows that the world tends to a full-interconnection of any physical objects, which is promoted by the concept of Internet Of Things (IOT). This makes it possible to create smart homes where people can listen to music and change songs, print a document, and control lights and room temperatures, from any remote computing de-vice: physical (e.g., a switch) or digital (e.g., a software application) devices. We also need to model the communication mechanisms that can be used along with computing devices and to describe how they work. This will allow us to choose what kind of communication mechanism we are going to use and support, with their advantages and drawbacks.

There is a plethora of communication mechanisms that allow these com-puting devices to inter-operate. Most of them rely on two types of addresses: IP and MAC addresses.

IPv6’s onset brought more than 300 undecillion (300 ∗ 1036) addresses, which at the moment, seems to be enough to address the whole world of computing devices.

The primary communication mechanisms are Ethernet, WiFi and Blue-tooth. But there are lots of other communication protocols and they all have their specific properties. Ethernet and WiFi allow devices to access routers and internet with or without wires. Bluetooth allows the transfer of information (e.g., sound, files, ...) from one computing device to another. NFC [NFC] is a near-field communication mechanism that can exchange a small amount of information when two computing devices are put against or close to each other. The primary use of NFC is to establish a Bluetooth connection between two computing devices but it can also send commands (e.g., with NFC tags) from one computing device to another. Today cloud computing appears as a solution to exchange or synchronize data across all the computing devices that are connected to it [cloud computing]. How-ever, only data can be stored in the cloud and there is no direct interaction between computing devices.

We will use the term distributed system to refer to a system of intercon-nected computing devices that will appear to the users as a coherent whole to accomplish the tasks that users want to carry out. The main reasons for using this term are that it is largely used and commonly accepted in the field of Distributed computing [AND00, PEL00], it is sufficiently expressive, and it encompasses all other possible terms [EMM98, 1].

Definition 6. A distributed system [DS] consists of a collection of au-tonomous computers, connected through a network and distribution middle-ware, which enables computers to coordinate their activities and to share the resources of the system, so that users perceive the system as a single, integrated computing facility [EMM98].

In the Painter’s Palette example, the distributed system contains all the computing devices that will be used by the drawing application and the user that will interact with the application and with these computing devices.

Distribution mechanisms provided by the domain of Distributed Com-puting can be a solution for managing the complexity of a DS. There are several DS properties to consider [EMM98]: the physical distribution of the computing devices and users, the problem of tasks running in parallel, the failure of a computing device or the failure of the communication between two computing devices, the lack of global knowledge and the dynamic as-pect of computing devices joining and leaving the network. Such complex algorithms would allow developers to support the distribution of the UIs.

Because of all these aspects, it is not possible to exactly know what happens in the system. Some computing devices are in a certain state while giving information, but as soon as the information goes through the network, it is not sure how long this information will stay valid or persistent.

DS. It should also be possible to see and understand the relationships and communications between devices, and how to distribute a DDGUI across these devices.

1.2.2 Approach

We have been looking for any form of methodological guidance such as guides, methods or approaches that define the aspects to consider. However these have proven to be very rare.

What we have found was either unavailable, or not sufficiently docu-mented to enable us to use it for our approach.

1.2.3 Software support

The last step is to create any kinds of software support for the method and the models.

Apple, Google and Microsoft have recently released services to allow users to display photos or to play music and videos on any connected de-vice, using technologies like AirPlay[AirPlay], AirPrint for Apple and Xbox SmartGlass[Xbox SmartGlass] for Microsoft. Lately, Apple has also intro-duced Continuity[Continuity] which allows people to connect an iPhone o an iPad to a MacBook in order to execute a few basic operations. For instance, an Apple TV can be controlled from any device such as a remote control, a smartphone, a tablet, or any compatible laptop or desktop.

Recent technologies like Miracast [Miracast], Air Display [Air Display] and Project My Screen [Project My Screen] allow people to display the screen of a smartphone, a tablet or a computer wirelessly to a compatible device, e.g., a T.V., an external screen or a compatible computer.

There are also a few brands that work together to simplify connection between a smartphone and their devices for home automation. This is a short-term solution to this problem, but they will never support all possible devices. They also use closed protocols which prevent them from extending their solution with others.

All these solutions have been created by different groups of people and these efforts have not been integrated into a single development tool. There is therefore a need for a software support that would allow developers to ben-efit from these results without having to learn how to use all these solutions separately.

1.3

Thesis Statement

The problem to be addressed in this thesis is to enable designers and de-velopers to create applications that support dynamic distributed graphical user interfaces (DDGUIs) and that can be used on all the available devices. DDGUIs enable end users to distribute any user interface element, rang-ing from the largest to the smallest, across one or many devices at both design-time and run-time. Using this world of fully interconnected devices will allow people to arrange and mold the applications according to their needs. In brief to support Distributed User Interfaces.

To create the software support that will allow us to create such appli-cations we first need to define conceptual models and to create our own approach based on these models. With this approach we want to hide the complexity of a distributed system inside distribution mechanisms.

Here is the thesis that we want to address:

In order to provide designers and developers with a model, an ap-proach and a toolkit to support dynamic distributed graphical user interfaces of interactive applications, we introduce the concepts of distribution graphs, distribution scenarios, in a model-based ap-proach that supports the properties of distributed systems and is implemented by a toolkit.

A DDGUI is not just the ability to move the UI from one device to another (migration). It also allows the use of several devices, the exploitation of their different sizes and characteristics, and their integration. Several users can fully interact together thanks to the support of distribution.

Along with DDGUI we want to support the properties of a DS such as the observation of devices joining and leaving, delays, and failures.

Facing a lack of definitions and models to support the creation of a DDGUI, we have decided to introduce some concepts that allow us to model and manage the distribution of UIs in a DS. Upon these concepts we have created a toolkit that demonstrates the possibilities offered by DDGUIs.

1.3.1 Single VS Multiple distributions

There are two ways to distribute the user interface (UI) across devices: single and multiple distributions.

A single distribution of the UI means that the features offered by an application can be migrated to other devices. It does not matter if the

concurrency between them. An example of multiple distribution is an appli-cation that controls room temperatures in a house. If two devices attempt to change room temperatures at the same time, this leads to a conflicting situation. How can the system know which of the change should take place?

1.3.2 Scope

Although DUIs could be applied to many domains of human activity and various contexts of use, this thesis states a series of assumptions (Ai) to

focus on a specific scope and leave other problems for future work. Let us start by introducing the general assumptions (GAi):

ˆ GA1: DDGUI only: no other modality of interaction

The thesis only focuses on visual modality. Vocal, taptic, haptic, or any multimodal systems are not addressed directly in the thesis. ˆ GA2: No complex or safety critical system

The thesis will only focus on interactive applications which can be rep-resented as a simple system. These applications are not safety critical (e.g., neither an unmanned aerial system, a train control/management system, nor health critical applications, ...).

Here are the assumptions regarding the model dimension (M Ai):

ˆ MA1: No coverage of DDGUI usability ergonomic aspects

Since this thesis is intended to introduce a principle-based way to de-sign DDGUI, we do not assume that any GUI resulting from a distri-bution issued by the method is usable. Further references on DDGUI usability include [DEE10].

ˆ MA2: No coverage of DDGUI security

The security of the whole DS and its applications is left for future work. However it is possible to add this concept to the distribution method introduced. Anyway, the security for the DDGUI is less critical than for the functional core. An example of security issue is someone pretending to be a user who he is not , also known as identity theft.

To cover this issue, we need to guess who is using a device with some recognition (e.g., face, voice recognition, biometrics, ...).

And regarding the approach (AAi):

ˆ AA1: handling 2 reference use cases

The concepts have been evaluated on a small number of case studies. We wanted to validate our concepts and approach on real case studies. The same reasoning can be applied on other case studies. We want to select a few case studies that will help us understand the concepts and the way we can apply them to the case studies. Any kind of applications could have been used as a case study. The first case study we have selected is Pictionary because this game is naturally distributed across several players that have different roles and these roles change depending on who wins. The second case study selected is an adaptation of an existing application: Transdraw, which is a drawing tool using transactions. This will prove how easy it is to adapt an existing application in order to support DDGUI.

ˆ AA2: focus on the distribution of the UI part of an application

We have focused the research on user interface (UI). The logic part of an application is already widely covered in distributed computing and their solutions still work with our conceptual solutions.

ˆ AA3: the logic part is supposed to be always running, active

and reliable

This logic part is supposed to be always running, active and reliable. If an application needs some warranty about the reliability of the core (where the logic is), it can still distribute the logic part using methods of distributed computing. Thus, if the logic is always running, the UI of a device can always be recreated in case of a crash.

Finally we also have assumptions for our software support (SAi)):

ˆ SA1: handling a set of supported devices

We support a subset of operating systems. The reason is that with a small number of operating systems we can cover more than 95 percent of the common smartphones, tablets, laptops and desktop computers. For this we have selected the most commonly used operating system. The choice of the operating systems supported does not influence the solutions proposed in this thesis. The potential candidates are all the operating systems. For computers we will support: Linux, Mac OS X and Windows. We also target mobile devices through the main operat-ing systems on tablets and smartphones: Android, iOS and Windows Phone.

There are several contributions that are brought by the thesis. Let us intro-duce them according to each dimension.

Models

Regarding the models, we introduce the concept of Distribution Graph to model a DS. It is a graph where all the computing devices and users are rep-resented as vertices, and their connections as arcs. Then we use the EBNF grammar to define formally the language expressing distribution primitives.

Approach

We have also created a model-based approach based on the concepts of distribution graphs, distribution scenarios and distribution primitive.

Software support

One of the results of this thesis is a software support in the form of a toolkit that allows the creation of distributed user interfaces. We have called this toolkit JayTk. It implements the concepts introduced during the thesis and is built on top of Beernet as depicted in Figure 1.12.

Figure 1.12: JayTk based on Beernet and implementing the concepts of the thesis.

JayTk is the main contribution brought by the thesis. It allows develop-ers to create applications with DDGUIs and to support the main properties of a DS. There are two different kinds of applications: distribution unaware and distribution-aware applications.

ˆ Distribution unaware applications do not need any information about distribution. They can use the toolkit with no or little modification of the code. They get the power of distribution through the toolkit which manages the distribution automatically or manually through an additional interface provided.

ˆ Distribution-aware applications take full power of the toolkit and the distribution mechanisms. These applications can be notified if one of the devices crash and react to it. They can also manage how the GUI is distributed when a device joins the network.

1.4

Support for mobile devices

The way our toolkit will allow applications to be distributed across mobile devices is different from how they are distributed across computers. Indeed, mobile devices will be used as a destination of distribution but will not create applications. The main reason is because Mozart and Beernet are not yet available on the operating systems that run on these devices. To our knowledge, there is no standard or well known way of supporting distribution mechanisms on these devices either.

However these mobiles devices can be used as weak-node in the peer-to-peer network. This means that they are not responsible and part of the distribution but they can receive and interact with applications that are run-ning in the network. This is how the toolkit supports mobile devices without offering a full compatibility. In the future, this limitation will be removed because these devices will become as efficient and powerful as computers.

1.5

Organization of the Thesis

Figure 1.13 describes the structure of the research. It clarifies the relations between the concerns, the shortcomings and the requirements of the thesis. The concerns have been defined in this chapter. The shortcomings and the requirements will later be derived from the concerns.

This chapter introduced the thesis topic by explaining the motivations. It describes what a DS is and all the important concepts that come with it. Then the scope of the thesis is set. And a summary of the main contributions is given.

In Chapter 2, we establish the related work of research related to the thesis. One of the aspects in the comparison is how a DS is modeled. Then

Figure 1.13: Outline of the thesis

we explore the existing tools that support the distribution of the application UI. Finally, we discuss other methods and studies on specific aspects covered here.

The method is then described in detail in Chapter 3. We first formally define the concept of distribution graph. Then, we select some operations that provide a sufficiently representative set of the possibilities offered by the distribution. The last part is the description of the method to create, support and manage DDGUIs in distribution scenarios.

The implementation of the toolkit with all the questions and choices that have been raised by using the method are provided in Chapter 4.

In Chapter 5, we use the toolkit to create a solution for several case studies. We also demonstrate a solution for some case studies: a Pictionary with three devices, a distributed transactional drawing tool using more than three devices and running on Android devices, and other small examples.

Chapter 6 evaluates and compares the results of the method with some important results in the related work. It is also the validation of the toolkit built during this thesis.

The last chapter concludes the thesis with a list of all the contributions. A list of the ongoing work is also provided.

The structure of the whole system is summarized on top of JayTk’s architecture (Figure 1.14).

Related Work

In this chapter we compiled the related work by first listing them all and then we evaluate their impacts on each dimension.

In order to create a list of the related work we first started from a pa-per classifying several papa-pers on DUIs[ROU06]. We have been through its references and iterated in each paper’s references when we found them as interesting for the thesis. We also included papers that have been submit-ted at the DUI workshop of the CHI conferences. The main topics used as criteria for the selection of a paper are UI and DUI, distributed comput-ing, models, approach and software supports. We will use these criteria as categories to discuss about these paper’s contributions through this chapter. Here is the expanded list of the related work we have built. The most important contributions will be detailed after the list. Other references can be described by analogy to them.

Models

There have been lots of papers on models. Software engineering is an important domain[DAM05, DAM06]. Mandviwalla[MAND94] has worked on requirements for groupware systems. Letier[LET01] has written a the-sis about agents in goal-oriented requirements engineering. Puerta and Eisenstein[EIS01, PUE02] have worked on XIML, a representation for in-teraction data.

There have been several papers on task models, and migratability. Bar-boni and co. [BARB10] have introduced a new notation for bridging the gap between tasks and systems models. Dittmar[DIT11] has worked the support of task migratability. Penichet and co.[PENI07] have worked on task mod-eling for Collaborative Systems. Wurdel and co.[WUR09] have worked on task modeling for smart environments.

Some papers have concentrated on the context model. Brdiczka and co.[BRD07] have worked on models for context-aware services. Shackel [SHAC09] has worked on the definition of usability. Terosiero,

Approaches

Although we have been through lots of papers we have only found one ap-proach to help developers to create applications. Martinie and co.[MART10] have worked on a model-based development approach to embed requirements at design time. The papers about software supports for DUIs describe their solution without providing the methodology they used.

User Interface

In order to address the problem of designing and developing a DDGUI we have reviewed the literature in HCI. This review has proven that there exist lots of references about UI. We first list them and will describe them later in the section.

In model-driven UI development, Breiner and co.[BRE10] have realized an evaluation of the UI adaptation. Caffiau and Girard[CAF10] have intro-duced a process for using model-driven approach in UI design. Gonz´ ales-Calleros, Guerrero-Garc´ıa, Vanderdonckt, and co.[GON09, GUE06, GUE09, VDD01] have worked on conceptual modeling of UIs. Griffiths and co. [GRI01] have worked on Teallach, a model-based UI development environ-ment for object databases. Ladry and co. [LAD10] have worked on usability evaluation of interactive techniques. Pastor and Molina [PAS07] have worked on a software production environment based on conceptual modeling. Rich [RIC09] has worked on building task-based UIs with the ANSI/CEA-2018 standard. Sousa [SOU09] has worked on the model-driven approach for UI in business process modeling. Wolff and Forbrig [WOL10] have worked on the development with the Eclipse Modeling Project. Zhang [ZHA10] has worked on an aspect-oriented UI Modeling.

Howell and co.[HOW03] have evaluated the runtime performance of GUI creation frameworks. Miah and Alty [MIA99] have realized an empirical study of an adaptive window management. Rashid and co. [RASH12] has compared the cost of display switching with mobile, large display and hy-brid UI configurations. Vrazalic [VRA03] has realized an evaluation of dis-tributed usability in an activity systems. Xiaojun and Balakrishnan [XIA09]

have presented a study comparing usage of a large display to single/dual-monitor configurations.

Moscovitch [MOS09] have worked on the use of the contact area instead of the contact point with touch UIs. Vanacken and co. [VAN08] have worked on additional UI for Multi-touch Interaction. Aslan and co.[ASL10], Avra-hami and co.[AVR89], Beaudouin-Lafon and co.[BEAU00, BEAU01], and Barralon and co.[BARR04, BARR07] have also written about additional UI to control the UI itself, commonly called meta-UI or extra-UI.

Other researchers have focused their work on more specific UI. Ali and co.[ALI01, ALI02], Bishop[BISH06], Ding and Litz[DIN06] have written about multi-platform UI. Kortuem and Kray [KOR05] have studied the HCI issues in multi-display environments (MDEs). Bickmore and co.[BIC99] have worked on a web page filtering for mobile devices. Pierce and Mahaney [PIE04] have also worked on mobile devices. Kavaldjan and co. [KAVA10] have worked on an automated optimization of UIs for screens with lim-ited resolution. Hutchings and Pierce[HUT06] have worked on divisible UIs. Hill[HIL92] has studied the abstraction-link-view paradigm to connect UIs to applications. Lorenz [LOR10] has studied the application of MVC in ambient computing environments. Vernier and Nigay [VERN99, VERN00] have worked on multi-modal UIs. Schlegel [SCHL10] and Schwartze and co. [SCHW10] have worked on the adaptation for UI at runtime.

There are groups of research that have already spent time on the next generation and the future of UIs. Shaer and co .[SHAE08] have worked on the use of UIDL with them. Myers and co. [MYE00, MYE01] have discussed about the future of software tools.

The work on UI for multi-device systems has led to the domain of Dis-tributed User Interfaces (DUIs).

Distributed User Interfaces

There are plenty of papers about DUIs which aim reaching Mark Weiser’s dream of Ubiquity[WEIS99, WEIS03]. Aksenov and co. have worked on reasoning over spatial relations for DUIs[AKS08, AKS09]. Balme and co. have worked on a reference model for DUIsband[BAL04].

There are teams of researchers that spent several years on this topic. Bang, Berglund, Fr¨oberg, Sj¨olund and co.[BANG05, BER02, FRO11, SJO04] have been among the first to work on DUIs. They have realized a prototype with a smartphone used as a remote for a computer.

Bandelloni, Ghiani, Manca, Mori, Patern`o, Santoro and co. have written several papers on DUIs [GHI10, PAT02, PAT07]. Their research has led to the development of TERESA (Transformation Environment for inteRactivE Systems representAtions)[BAND04, MOR03, MOR04, PAT01, PAT08]. An authoring tool which provides designers and developers automatic support for transformations of UIs. It has later led to the development of MARIA

Smart Home Energy Assistant (SHEA)[FEU07, WEIN10].

Their research has led to the development of MASP (Multi-Access Service Platform) [BLU10, FEU07, ROS09, ROS09B, WEIN10] which is a model-based run-time system for the creation of DUIs.

de la Guia, Gallud, Garrido, Lozano, Marco, Pe˜nalver, Penichet, Se-bast´ıan, Tesoriero, Villanueva, and co.[DLG10, GAR11, MARC11, SEB11, VIL11] is currently active on formally defined DUIs with models. Their work has led to the definition of an AUI model[PENA11, PENA11B, PENA12].

Coninx, Luyten, Meskens, Vanderhulst, Vandervelpen and co. [LUY02,

LUY04, VDH07, VDV04] have also been working a lot on DUIs.

Van-derhulst has written a thesis about Dynamic Distributed User Interfaces (DDUI) [VDH05]. They have also worked on models[VDH08C, VDH09]. Their work has led to the development of Light-Weight Distributed Web Interfaces[LUY05, LUY06, VDV05, VDV05B]. And it has also led to the de-velopment of GUMMY[MESK08, MESK09], a multi-platform GUI builder. They have also worked on ReWiRe and pervasive environments[VDH08, VDH08B, VDH08C, VDH10, VDH10C], and on software support[VDH09B, VDH10B].

Finally Grolaux, Lepreux, Vanderdonckt and Van Roy [GRO04, GRO05, LEP06, LEP06B, LEP11, VDD10] have worked on migratory UIs. Their work has led to the development of the AttachMe, DetachMe demonstration which is based on EBL/Tk (Enhanced Binding Layer/Toolkit), a middleware that interfaces with one or more graphical toolkits. This tool has led to this thesis.

Lots of small teams have also worked on DUIs:

Bardram and co. [BARD11], Barth and co. [BART11], Bharat and co. [BHA95], Cagle [CAG05], Chang and Li [CHA11], Chen and co. [CHE11], Dadlani and co. [DAD11, DAD11B], Ens and co. [ENS11], Fardoun and co. [FAR11], Lambropoulos and Danis [LAM11], Larsson, Ingmarsson and co. [LAR06, LAR07], Linten and Price [LIN93], L¨ochtefeld, and co. [LOC11], Marquardt and Greenberg [MARQ07], Molina and co. [MOL06, MOL06B],

Qiu and Graham [QIU09], Rodden and co. [ROD04], Seifried and co.

[SEI11], Sendin and L´opez [SEN11, SEN11B], Shen and co .[SHE04], , Yanagida and Nonaka [YAN08], and Z¨ollner, and co. [ZOL11]. Bell [BEL05]

has written a doctoral thesis on DUIs. Recently, Elmqvist [ELM11] has listed the state of the art of DUIs.

There are also lots of papers talking about multi-display and multi-device systems.

Hutchings, Stasko and co. have worked a lot on multiple monitors [HUT02, HUT02B, HUT03, HUT04, HUT04B, HUT04C, HUT04D, HUT05, HUT05B, HUT05C, HUT05D, HUT07, HUT07B] and on QuickSpace which provides new operations for computers[HUT02].

Ashdown and Sato [ASH04], Grudin [GRU01], Inkpen and Mandryk [INK05], Kaviani and co. [KAVI11], Lee and co. [LEE08], Mansoux and Nigay [MANS05] have also worked on multiple monitors.

Beale and Edmondson [BEAL07], Cardinaels and co. [CAR06], Dearman and Pierce [DEA08] have worked on multi-device and multi-screen systems.

Applications

Along with all these more theoretical papers, there are some applications that have grown up from these topics. Air Display [Air Display] and AirPlay [AirPlay] are applications to turn a device into a monitor. Ayatsuka and co. have worked on HyperPalette[AYA00]. Benoˆıt and co. have worked on a multimodal driving simulator[BEN07]. Englebert and Heymans have worked on MetaCASE tools[ENG07]. Black, Edwards and co. have worked on the Speakeasy approach[BLA02, EDW02]. Eychaner has developed a UI frame-work for controlling DS[EYC03]. Etherpad is a website where a user can create a text document and share it with other users[Etherpad]. Han and co. have worked on WebSplitter[HAN00]. Drag&Share is a shared workspace for distributed synchronous collaboration [MARC11]. Rekimoto and co. have worked on Pick-and-Drop and Proximal interactions software applications [REK97, REK03]. Rey and Coutaz have worked on the Contextor, an appli-cation for dynamic distribution of contextual information[REY04, REY06]. Multi is a multi-user laser table interface[STU04]. Wincuts allows the ma-nipulation of window regions[TAN03, TAN04].

Distributed computing

When writing about DUIs it is hard to avoid writing about Distributed computing. There are already many papers on distributed computing to list them all but here are the main litterature we have been through to understand and benefits from this topic. Andrews [AND00], Attiya and Welch [ATT98], Collet [COL07], Coulouris and co. [COUL05], Elmqvist [ELM15], Emmerich [EMM98], Ghosh [1], Peleg [PEL00], Tel [TEL95] have written about distributed computing.

A subtopic of distributed computing is the Peer-to-Peer architecture which has been covered by the following papers. Loeser and co.[LOE03],

Others

Some teams of researchers have worked on many of these topics together and cannot be put in one of these.

Calvary, Coutaz, Demeure, Frey, Graham, Lachenal, Roudaut, Sottet, Thevenin, Vanderdonckt and co. have worked on models [CAL97, COU05, COU05B, DEM05, SOT07], on plastic UI [CAL01, CAL04, COU10, SOT07, THE99, THE02, VDD08B], on ambient space [COU06, COU07, COU07B], on multi-device [COU03, COU03B, GRA00, LAC03], on DUI [DEM05B, DEM08, FRE09, ROU06, ROU06B] and on gestures [ROU09].

Dewan and co. have worked on multi-user systems[DEW98, DEW98B]. Dey and Abowd have worked on context-aware systems[DEY00, DEY01]. Jourde and co. have worked on multimodal systems[JOU10]. Robertson

and co. have worked on a flexible task management[ROB04].

Hyper-space is a high resolution display[SCREEN]. Seifried and co. have worked on CRISTAL, a collaborative home controller[SEI09]. Tullis and Albert have worked on usability metrics[TUL08]. Vanderdonckt has worked on a knowledge-based system for interaction styles[VDD97].

2.0.1 Meta-UI for Ambient Spaces[COU06]

In their paper on Meta-UI for Ambient Spaces[COU06]. They compare the tools that exist in 2006 such as ARIS [BIE04], Jigsaw [ROD04], AttachMe [GRO05] and SpeakEasy [BLA02, EDW02]. The dimensions used for the comparison were the discoverability, the coupling, the ability to redistribute and to remold the UI. The main conclusion is that researchers should be careful about the development of automatic systems for scientific challenge instead of involving and considering the level of control left to end-users. You can find their classification of related work in Figure 2.1. This table shows the different capabilities offered by the toolkits according to several axis: discovery, coupling, re-distribution, re-molding, parametrization and extensibility.

Figure 2.2: The ergonomic aspect of the User Interface[DEE10]

2.0.3 The Fiaa Platform Model

Another paper on modeling is the Fiaa Platform Model[QIU09]. The Fiia Platform Model described is based on a publish and subscribe architecture. Each devices may subscribe to information it is able to use or display. Less-powerful devices may only get partial and more specific information from parts of the application while more-powerful ones will user the complete application with the information related to it. Each device which is repre-sented as a node of the Fiaa Platform Model which stores its own platform information. The resource model of this Fiaa Platform Model is depicted in Figure 2.3.

Figure 2.4: Their AUI model with the DUIs perspective from [PENA11B] Although the research is at an early stage, this model supporting DUIs is the base of an approach that they are currently building.

2.0.5 FRESCO

FRESCO[LIN93] is a set of programming interfaces that expand the X win-dow system in Linux. It models UI components as objects in order to allow users to run an application on a remote machine. It allows the partial man-agement a DS. While X provides network-transparent access to some user’s display remotely; Fresco allows the distribution of components from UIs across several devices.

2.0.6 CESAM

CESAM[ROU06B] is a prototype to show how a UI could be extended to support some operations to distribute the UI.

In CESAM prototype, it is possible to provide the following information about a device: a name and, the width and the height of the screen resolution as in Figure 2.5. There is also other information such as a place, and the interconnection between devices. The two devices that are in the example have a different form factor: there is a laptop and a PDA.

Figure 2.5: A print screen of the CESAM prototype with two devices connected[ROU06B].

Via a drag&drop gesture of an item from the upper-left part to the right part of the window you can assemble devices together. In the example the two devices are linked together which is visible thanks to the red border that encapsulates them. For devices to be assembled they have to be in the same logical space. In this case the PDA and the laptop are in the space called Maison. CESAM also allows users to cut the UI into several parts. An example is provided in Figure 2.6.

Figure 2.6: A print screen of an application that has been cut into parts[ROU06B].

Figure 2.7: Example of simplicity gains from research on DUIs[HUT07]

2.0.8 ARIS

The ARIS[BIE04] interface allows users to relocate applications across a fixed disposition of devices.

In their setup, they have 5 screens hold by walls, with a PDA and two tablets. The tool provides an iconic map as a visualization of the static DS as in Figure 2.8.

2.0.9 IMPROMPTU

ARIS has been extended to an interaction framework in 2008. They renamed it to IMPROMPTU[BIE08]. It is a simplification of the interactive space as in Figure 2.9.

Figure 2.9: Screen shot of the IMPROMPTU’s UI[BIE08]

Users have the ability to share a view (read-only or editable) of their windows. Applications can be used remotely and each device is a server for its own applications. IMPROMPTU brings a specific UI (Figure 2.10) to support collaboration between several users.

There is a view on the windows shared by any user (Figure 2.10a), and there is a view on the windows shared by a user from the device where it is displayed (Figure 2.10b). The windows sharing is possible by capturing application window’s pixel data and reproducing it on other devices.

2.0.10 GUMMY

Gummy[MESK08] is a tool for building generic multi-platform GUIs. It starts with an initial GUI on a device. Then it adapts and combines its features into a new GUI for another device. It allows people to target new devices and keep UI consistent without requiring designers to start from scratch. A screen shot of Gummy is depicted in Figure 2.11.

In order to generate a Final User Interface, Gummy starts with a UI description which is converted thanks to a UIML vocabulary into the Final User Interface (Figure 2.12).

Figure 2.10: IMPROMPTU’s extra-UI for windows sharing (a) and display-ing (b) [BIE08].

Figure 2.12: A UIML vocabulary relates generic terms to concrete represen-tations [MESK08].

2.0.11 Light-weight Services

Light-weight Services [VDV05] supports multi-user collaboration thanks to an HTTP-based daemon allowing the distribution of web applications. It offers distribution of web applications through services. In Figure 2.13, there is an example of a website that is distributed across three devices. The zoom services is currently distributed on the device on the left.

This toolkit allows the full control of the distribution through user-driven distribution and support automatic distribution through system-driven dis-tribution. They allows automatic redistribution in case of changes in the interaction space [VDH08C, VDH09]. The data and the UI of a discon-nected device will not be lost, thanks to the redistribution of this part. The figure 2.14 shows an additional UI to control the application’s UI, one of the UI and the list of tasks provided by the example.

2.0.12 MASP

The Multi-Access Service Platform (MASP)[ROS09, ROS09B, BLU10] is an additional UI to control Smart Environments. It allows the control of the UIs from their SHEA assistant (Smart Home Energy Assistant) through an additional UI (Figure 2.15) based on a graphical representation of the different tasks that a user can display on a screen. Supported distribution primitives are involved in four services: migration for transferring a ser-vice from one interaction resource to another, adaptation to an interaction resource, distribution of UI elements across interaction resources, and multi-modality. It allows the distribution of a task on a screen. They also support a dynamic environment and support multimodal interactions with devices.

Figure 2.13: An example of website distributed across 3 devices thanks to the Light-weight services[VDV05].

Figure 2.14: The additional UI and the UI associated with the task play music[VDH08C].

Figure 2.15: A screen shot of MASP’s UI[MASP].

application is the distribution of User Interface to allow the move of some services UI from a device to another.

The description of the context with the room, and each users, is depicted in Figure 2.16.

2.0.13 Web sites and applications

Web sites and applications have been particularly investigated through the angle of distribution primitives with the Migration project [MOS09], Cameleon-RT [BAL04]. Most of the time, the extra-UI is separated [ROU06] from the UI subject to remolding or distribution.

Figure 2.16: A screen shot of the description of the context[MASP].

2.1

The related work along the three dimensions

2.1.1 Modeling for Distributed Systems

The first dimension we have explored is the model dimension. There are sev-eral concepts that can be modeled (e.g., users, tasks, context, environment, devices, connections).

Distributed Systems

The concept of a DS has been widely used in the related work however there are several terms that refer to it.

It was first introduced as ubiquitous computing in this famous state-ment: ”Specialized elements of hardware and software, connected by wires, radio waves and infrared, will be so ubiquitous that no one will notice their presence” [WEIS99].

Many papers instead used the term Interactive Space. An Interactive Space is a DS where users can interact with all the devices. But today there are DS where users cannot directly interact with all the devices (e.g., inter-net, cloud computing). They can only remotely access them. An interactive space is the subset of devices in a DS that users can interact with.

Later, the concept of Interactive Space has been replaced by Multiple Display Environments (MDEs) or Multiple Monitor Environments (MMEs) as in IMPROMPTU[BIE08]. The concept of user disappears and the only remaining concepts are the displaysand applications’ GUIs. An MDE is also a subset of a DS where the users are not represented and devices are only represented as displays. A device that has two displays will be considered as two different entities in this concept.

There is also a concept called Multi-Surface Interaction. It has been introduced in [COU03B]. Public walls, blackboards, desks and tables, the back of an envelope are some of the surfaces that can be used. In

com-puting science, these words define the same areas. However they can be augmented with computational capabilities. Each surface has its own inter-action properties. To define the concept, they first introduce the concept of an information surface which represents a physical surface able to dis-play information such as digital information. They call it a multi-surface interaction, when the information surfaces can be manipulated through a UI.

These concepts are also very close to our definitions of a device and a DS. Information surfaces as well as multi-surface interactions are some kind of devices. A device is an information surface if it has the ability to show information (i.e., the device has a display). The multi-surface interaction is the set or a subset of all the devices that support interactions in the DS.

In Figure 2.17 the distributed system set is represented as a super-set of all the other terms.

Figure 2.17: The distributed system set and the other sets

Lack of support for the dynamic aspect

Most of the papers consider the DS as something static and already known. In ARIS[BIE04], they have a iconic map with one PDA and two tablets. What will happen with their iconic map if we want to add another device in the DS? They do not support the dynamic aspect of a natural environment: users and devices can appear and disappear at run-time.

Also in all the papers we have been through, the connection between devices is considered as perfect. This means that the connection is assumed as permanent but in a real world there are failures and delays. None of the existing studies provide a tool to deal with these failures and delays.

Figure 2.18: The four principal components in a human-machine system [SHAC09]

Later the term tool has been replaced by the term device which encap-sulates the applications that run on top of it. The task has been dropped since it is a goal to achieve and not an agent in the system. This has led to the definition of the context of use[CAL04, COU05B, DEY00, DEY01]. The context of use has been defined as a triplet C = (U, P, E) where there is one user, one platform and one environment. In this definition a platform can be a cluster of platforms (e.g., a desktop computer with a tower, a monitor, a keyboard and a mouse). A context of use is always attached to one and only one user.

To deal with today’s reality there is a need to extend this highly-used concept to take care of all the components of a DS. In a DS, there can be several users, several devices and they can dynamically appear, disappear, join and leave at any time. The term platforms and devices are equivalent. In the Painter’s Palette example there is only one user and one device at the beginning. The current definition of the context of use does not allow us to represent a second device, or another user. Multi-device and multi-user scenarios appear more often than in the past. The support of both kinds of scenarios could be added to the representation.

At first we wanted to extend the definition of the context of use to a distributed context of use. A distributed context of use would have been Cd = (Ud, Pd, Ed) where Ud is the set of Users, Pd a set of Platforms and

Ed is the environment that contains all the users and platforms. Each user

would have got his/her own context of use and environment. Configurations like in Figure 2.19 would have then be possible.