The design and implementation of a multi-user interactive public display

The Design and Implementation of a Multi-User

Interactive Public Display

by

Sonia Garg

Submitted to the Department of Electrical Engineering and Computer Science

in partial fulfillment of the requirements for the Degree of

Master of Engineering in Electrical Engineering and Computer Science

at the

MASSACHUSETTS INSTITUTE OF TECHNOLOGY

1ay 2003

© Sonia Garg, MMIII. All rights reserved.

The author hereby grants MIT permission to reproduce and

distribute publicly paper and electronic copies of this

thesis and to grant others the right to do so.

Author ...

Department of Electrical Engineering and Computer Science

May 21, 2003

C ertified b y

Larry Rudolph

Tjesis

Supervisor

Arthur C. Smith

Chairman, Department Committee on Graduate Theses

MASSACHUSETTS INSTITUTE OF TECHNOLOGY

The Design and Implementation of a Multi-User Interactive Public Display

by

Sonia Garg Submitted to the

Department of Electrical Engineering and Computer Science May 21, 2003

in partial fulfillment of the requirements for the Degree of Master of Engineering in Electrical Engineering and Computer Science

Abstract

This thesis explores a new approach to exchanging information using an interactive public display. The system was designed to support multiple users in dynamic environments, where people connect and disconnect to the displays at will. The key goals were to ensure a seamless end user experience and provide the anonymous exchange of information. Also, users have the option of customizing and

personalizing their interaction with the public displays. A prototype of the exchange system was built to demonstrate a connection scheme for Bluetooth enabled devices. The interactive display system can be applied and extended by various pervasive computing endeavors.

Thesis Supervisor: Larry Rudolph Title: Principle Research Scientist

Acknowledgments

I would like to thank:

Larry Rudolph for his inspiration and guidance throughout this project. He has been a motivational advisor and I am grateful to have had the opportunity of working with him.

John Ankcorn and Jorge Ortiz for their help in designing and developing the Content Exchange system.

Jonathan Brunsman, Glenn Eguchi, Hana Kim, Kan Liu, Arjun Narayanswamy for being fun and energetic officemates.

Members of the MEng '02 group for helping to initiate my involvement with the Oxygen Research Group.

My parents, Gobind and Kamlesh Garg as well as my brother and sister, Rohit and

Nitasha Garg for their endless support and encouragement which have carried me this far.

Contents

1 Introduction ...

6

2 M otivation and Background ...

9

2.1

Group Background ...

9

2.2 Context-Aware Pervasive Computing Efforts... 10

2.3 Efforts with Public Displays... 11

3 The Content Exchange ... 13

3.1 User Scenarios... 13

3.2 Design Goals ... 14

3.3 System Architecture ... 16

3.4 Functional Design ... 20

4 Functional Im plem entation ... 26

4.1 Bluetooth Stack & Interfaces ... 27

4.2 CEA Discovery...28

4.3 Establishing a Channel with a CEA via Bluetooth ... 30

4.4 Security...34

4.5 Partial State of the System ... 36

5 Analysis & Extensions ... 39

5.1 Tradeoffs with Bluetooth... 39

5.2 User-Experience ... 41

5.4 Extensions to the CEA N etw ork ... 45

5.5 Location Aw are Technology ... 47

5.6 M ulti-M odal Input... 48

Chapter 1

Introduction

The focus of this thesis was to change the process of sharing and exchanging data by

creating a system in which users could interact with information shown on public

displays. A system called the Content Exchange was designed to demonstrate a

multi-user system that can perform assorted functionality based on a dynamic set of

attributes provided by the users. A prototype was implemented to define a

connection and resource negotiation protocol for Bluetooth enabled devices.

In the past 30 years, technological revolutions have redefined the way people think

and operate. With the birth of the Internet, information has been made widely

available. Databases, servers, and personal computers are filled with an abundance

of information. Knowledge portals and data repositories are one way to organize

and store such large amounts of information. In order to make the information

repositories useful, effective mechanisms for saving and retrieving data need to be

established. The key challenges in managing data and information effectively are

one, to identify the proper way to display the information and two, to devise the

correct scheme to transfer the information.

First, displaying information effectively depends on the content of the information

and the display devices available. Many research institutes and industry

corporations have moved towards using public displays to advertise products and services, show press releases, and publicize events. These displays are primarily used for the purposes of viewing static information.

Second, methods of exchanging information have drastically changed with the emergence of the Internet. Network improvements and increased bandwidth have led to the adoption of peer-to-peer file transfer systems and chat programs.

Exchanging information is an important element of creative interactive and collaborative environments. A system needs to be established that will allow multiple users to have similar interaction but where the information is on a public or private display.

This thesis presents a system designed to change the paradigm for transferring data into and out of an information repository. The idea is to combine a public display with peer-to-peer file transfer. The result is a large-scale display that people can use to download and upload content. The system is designed to account for dynamic environments where users a free to move around. Additionally the system has multi-user support so that viewers can interact with the system simultaneously. Finally, the system allows users to personalize their experience and customize its behavior to fits their individual needs.

A system called the Content Exchange was designed to achieve a new degree of personalization and interaction when exchanging information on a public display. In order to provide a significant level of customization, avenues within the system

were constructed to use real-time input from users and their environment. A

prototype of the Content Exchange was implemented to illustrate how devices

connect and disconnect to the system using Bluetooth. Implementing a prototype for

Bluetooth communication exposed limitations of the communication protocol causing

the prototype to deviate from original design decisions. In theory it makes sense for

mobile clients to be considered Bluetooth masters because this gives users more

control over their devices. Further, users may have their mobile devices already

configured as Bluetooth masters, such as cell phones that would have Bluetooth

slave accessories such as hands-free headsets. Also, in order to make use of many of

the Bluetooth features, communicating directly over the Bluetooth stack was

required. Since Bluetooth stacks are proprietary it was uncertain that

communication between different stacks could be achieved. Therefore a TCP

interface to Bluetooth was used to manage and control communication. Although

this added complexity to the communication procedures, as discussed in this thesis,

it helped make communication between Bluetooth devices possible.

This thesis discusses the design and implementation of a multi-user interactive

public display. Section 2 outlines the motivation behind the project and describes

various related projects. Section 3 discusses the design of the Content Exchange,

goals, system architecture and functionality. Section 4 provides an overview of the

implementation of the pilot system. Finally, an analysis of the pros and cons of the

system and possible future extension is discussed in Section 5.

Chapter 2

Motivation and Background

There have been many paradigm shifts when devising architectures that store and circulate information. Given the new requirements about mobility and dynamic systems, present day information transfer models can no longer address important

user scenarios. Current advertisements and other large-scale information displays are static and stand-alone. There is minimal interaction between the general public and the devices that are displaying information. Other systems such as public kiosks that do have user interaction are limited in functionality and do not provide multi-user support.

Various systems and mechanisms have tried to address context-aware and information exchange issues in order to provide users with personalized and more relevant services. The research group that supported this thesis has developed a suite of systems that help incorporate technology into people's daily routines. Other proposed solutions include public kiosks and information exchange protocols. The aim of this section is to provide an overview of the Oxygen Research Group and other work related to public displays and content exchange.

2.1

Group Background

Oxygen is a wide-ranged pervasive computing project that combines the use of different technologies.

The key theme of Project Oxygen is to "help people do more by doing less." [1] The project strives to make technology fit into the everyday lives of people as opposed to making people adapt around the needs of technology. The vision of the project is to create a single portable device that will perform all the desired communication tasks [2]. The human-centered approach addresses various issues, which include providing location-aware support and enabling the customization of anonymous devices so that people can have constant access to information [3].

2.2 Context-Aware Pervasive Computing Efforts

Efforts to develop a pervasive environment are widespread. Many different approaches are being investigated to achieve a human-centric environment that is responsive and customizable. The basic idea across the board is making computers fit the needs of humans.

Georgia Institute of Technology runs a program called the Aware Home which aims at creating a system which can understand the needs and desires of the its residents. Considerable effort is also placed into understanding the emotions of its residents. Using sensors including video cameras, wearable monitors, and location sensors, the system analyzes it's residents gestures, facial expressions, and movement. Based on this information, the aware home attempts to better predict the desires of its residents. [4, 5]

The University of California Berkeley has developed a software agent called the Communication Information Agent (CIA). They have used a service infrastructure approach to context-aware computing where middle-ware technologies can be accessed through a network. The CIA uses

context and human-to-human interaction to gather and deliver relevant and useful information. The information retrieval methods are based on the relationship of the information with the user that relies on location and speech technologies. [6, 7]

2.3 Efforts with Public Displays

Various other approaches to the dynamic public displays have been designed and implemented. Some solutions, although create an interactive display, are only valid for scenarios where the users are part of a small, related group. These solutions would not be applicable in a public setting where users are constantly connecting and disconnecting.

The GVU Center at Georgia Institute of Technology has designed Semi-Public

Displays, which are shared displays targeted for small groups [8]. These Semi-Public Displays are used in environments were there is a group of people working closely together. The purposes of the displays are twofold, one to help people collaborate and brainstorm on ideas and two, to inform people of the group's activities and interests. Although Semi-Public Displays promote collaborative environments and advertise events, the content on the displays is only partially dynamic. The information shown on the display is parsed from a weekly status report. Also, Semi-Public Displays are geared towards small, private groups and cannot be extended to public use.

At the University of Calgary, researchers have developed an interaction system called the Notification Collage (NC) [9]. The NC is designed to serve as a public

bulletin board for small, co-located groups. The NC has a separate interface for the general public display and personalized views for each user. The content on the NC is dynamic so that it can show updated video captures, sticky notes that show the message being typed, and various other multi-media elements. The NC is successful in creating an interactive environment, but similar to Semi-Public Displays it

targets a small private group. Extending the NC to a public domain would be nontrivial.

Tbe Software Infrastructures Group at Stanford has developed an IRoom where "people can work together in a technology rich space" [10]. The IRoom is an

interactive workspace that allows users to collaborative on school work using their mobile devices. The IRoom software infrastructure lets users enter and leave the workspace without affecting their experience. IRoom similar to other solutions focuses on building a collaborative environment and does not tackle the issue of a 2-way exchange of information.

Other display solutions are similar to Appliance Studio's Web Signs [11]. The Web Signs project is a flat panel display appliance that can run external applications. Web Signs provide interactive support, host dynamic content, and communicate with each other. Although Web Signs achieve a great deal of user interaction, it is mostly one-on-one interaction. Web Signs have a touch screen for users to "dig deeper" and get more information. However, Web Signs do not provide a framework for multiple users or for users to interact with the display using mobile devices.

Chapter 3

The Content Exchange

The Content Exchange is a system that is designed to serve as a mechanism for

users to interact with information on public displays, called Content Exchange

Appliances (CEA). These appliances are designed to be an information portal that

people can interact and exchange information with. This section serves to outline

the design goals, architecture and functionality of the system.

3.1

User Scenarios

The basic scenario for the Content Exchange system is to display general

information to the public. For example, viewers of the information can interact with

the display appliance while waiting for an appointment or a meeting. The displayed

content could include general health tips if deployed in a doctor's office or product

advertisements in the first floor lobby of a software company's building.

A second situation for the CEA is in a meeting room where information is shared

between multiple parties. The system could be used to assist in teleconferencing

because not only would an image be transferred but content could be exchanged as

well. This video relaying could also be applied in a family setting where different

members of the family can communicate from the home, office or while they are

walking around.

Another scenario for the display appliance is to support a collaborative environment where users can take full advantage of the display by uploading and downloading content. Co-workers or group mates can use the display to work together on a

project by uploading and displaying content.

3.2 Design Goals

The CEA can be considered a next generation public kiosk that serves multiple users at one time. The key aspect for the project was to keep the CEA human-centric because that would help instill faith in the end-user. In order for the CEA to be a success users have to feel comfortable enough to interact with and trust the

intentions of the system. Various goals were kept in mind when designing the CEA.

Seamless End-User Experience

To maximize the impact and usability of a CEA, an important aim is to provide a seamless end-user experience. The idea is that from arrival to departure, a user can just interact with the display without experiencing delays, configuring settings or

establishing connection interfaces. Device discovery and communication is initiated when a user is within local proximity of the public display. Similarly, a session ends when the user walks away from the public display. In addition, users do not have to be concerned with the actions of other users. Users should not have to wait for another user to finish their session or interaction. The system ensures that the navigation and actions of each user are independent.

Customized Interaction and Exchange

Users have the option to configure their settings to achieve a customized and more personalized experience with a display. Each user can define the type of information they download and upload. Since users of the CEA have personal mobile devices, they can define how they want to save the information they receive from the display. An important goal of the system is to build flexibility into the application that would allow users to personalize their experiences. Providing users the ability to control their personal settings regarding information retrieval and privacy makes the CEA applicable in more situations and more a larger variety of people.

Privacy and Security

Ensuring the privacy of users is important since the system is open to the public. The public displays are used as information portals and require the user to establish a connection via one of the various communication standards that exist. A

connection using Bluetooth or Wireless LAN may indirectly expose specific information about a user to the display application. The goal of the Content Exchange is to make it possible for users to anonymously interact with displays.

Despite knowing the location of the different displays, a user's location can remain unknown because the Bluetooth information associated with a device can be

changed with each session. Therefore, users can interact with public displays with the assurance that their sessions or actions are not tracked or recorded.

The goal is to provide avenues for users to specify their preferred security level in order to achieve the desired level of privacy. Although security was not the focus of

this project, putting structures in place that would support various secure protocols

was important. The goal was to achieve a minimal level of security that did not

require a lot of computation by the mobile devices. For this reason, the use of the

Secure Socket Layer protocol was minimized by creating a negotiation scheme (see

Section 4.4). Since the information in the CEA repository is not necessarily

sensitive, the default aim for security was minimal, but can be extended and

improved upon.

3.3

System Architecture

The overall architecture of the Content Exchange involves a public display running

content exchange services, called a Content Exchange Appliance (CEA) and

handheld devices (see Figure 1). The CEA can communicate and transfer data via

different protocols, including Bluetooth, Internet Protocol, and IR.

Content

Echange INTERNET-A

802.11b

Content Exchange IPAQ

0I

Figure 1: CEA connects to mobile devices over Bluetooth, IR, IP, etc.

Each handheld device is running an application that recognizes and understands the

messages sent by a CEA. In a typical scenario, a user will approach a CEA and

establish a connection. Once the CEA has secured a connection the user can proceed

to view, download and upload information to and from the CEA accordingly. Several components make up the multi-user content exchange system.

Content Exchange Appliance

CEA's are public displays hosting information, which can be viewed, transferred, and edited externally by users. A CEA is considered a "super-computer" because its computational ability, power supply, and memory dramatically outweigh the

properties handheld devices. They support the communication mediums of Bluetooth, 802.11b, Ethernet, and IR. CEA's accept connections from devices that would like to interact with the information on display.

CEA's can be controlled by typical network administrators to ensure that the information displayed and transferred is accurate and secure. A CEA is similar to the server side of an application, where people can use mobile devices to issue requests and make transactions.

Mobile Device

A mobile device serves as the gateway between the user and the CEA. Users can

interact with a CEA with any type of mobile device that supports one of the listed

communication protocols (see Content Exchange Appliance). The mobile device runs a client application that lets the device recognize a CEA. This application is

responsible for initiating a connection to the CEA and managing the information exchange. The mobile device provides the user with a personal viewing platform

regarding the contents of the display as well as an input panel when uploading and downloading information.

Personal Area Network

A personal area network is an ad-hoc network that exists for each CEA and the devices to which the CEA is connected. This project focuses on creating a

communication scheme using Bluetooth. A Bluetooth network is called a piconet, where one device, called a Bluetooth master, is connected to a maximum of 7 other devices, called Bluetooth slaves. The master of a piconet is responsible for

maintaining the frequency hopping for the connected devices. Bluetooth devices operate within the 2.45MHz frequency range. Bluetooth devices can communicate without interfering with each other by performing what is called "spread spectrum frequency hopping." Bluetooth devices communicate by hopping 1600 times per second between 79 different frequencies each 1 MHz apart. Bluetooth piconets do

not interfere with neighboring Bluetooth networks because the masters of each piconet ensure that the connected devices hop to different frequencies in unison. [12]

In the original design, the mobile devices were considered the Bluetooth masters of the system. This was because as users would most likely have their devices pre-configured as masters, regardless of their interaction with a CEA. In order to achieve this design and time scheduling routine must be put in place to balance the requests of the mobile clients, since the CEA would be the slave. To avoid that complexity, in the prototype of the CEA, the CEA was made the master which runs a module devoted to listening for incoming connections from other Bluetooth devices.

The piconet isolates the communication associated for each display and allows users

to interact with display anonymously. Once a user disconnects or leaves the vicinity

of a piconet they no longer have to be associated with the temporary IP they had

when connected.

Connection Channel

A connection channel is a communication link established between a CEA and a

mobile device. A connection channel is created when a user walks in the vicinity of a

public display and chooses to start interacting with the information. The channel is

an understood agreement between the CEA and the device regarding the properties

and settings of the user for a given device, user, session, or request.

Attributes

An attribute is a name value pair associated with a user, handheld or public display.

They are a means of personalizing and customizing one's interaction with a CEA.

Depending on the type of attribute, they are exchanged between the display and

handheld at the time of connection or when transferring data. There are two types

of attributes, session-based and action-based.

Session-based attributes are properties of a user's handheld and are therefore device

specific. They are transferred to the CEA when a connection channel is established.

Session-based attributes also describe privacy and security settings of a user (see

Section 3.2).

An action-based attribute is a name-value pair transferred over a connection channel when a user motions to exchange information with the CEA. Action-based attributes describe the type of information that is to be downloaded or uploaded. These attributes act as parameters to the different function calls on a CEA.

3.4

Functional Design

Interaction with the multi-user content exchange system is a result of various methods and functions built into the system. The functionality of the CEA depends on its purpose, the environment in which it is deployed, the users that interact with it, and the computational power available. The CEA is designed to be versatile enough to mold to various user scenarios and requirements, defined either by the administrator or the users. This section defines how the system is designed to operate and adjust to various users.

Configuring Settings & Attributes

An important goal the CEA is to provide a framework where users can customize their mobile client application in order to personalize their experience. Users can configure a variety of settings or use the pre-configured default values. Minimal setup is required if users choose the default implementation, where as customizing the settings requires a nominal amount of configuration time.

There are various common attributes that a user is more likely to configure to fit individual needs. For example a user can change the default connection behavior (see Section 4). When a user wants to create a connection channel with a CEA, the

user can choose to automatically connect to the closest CEA or see a list of possible devices before connecting. The default value, automatically connecting to a CEA, can be changed by users if they wish to have more control over their exchange sessions. Other attributes may be specific to the mobile device, such as the amount of free memory or the format and size of the device's screen.

Similarly, a user can configure their privacy settings, depending on how private they want to keep their interaction with a CEA. When connecting using Bluetooth a user can change the Bluetooth device descriptor and address that is affiliated with the mobile device. If a user wishes to remain anonymous then they have to option of changing the settings of their Bluetooth configuration so that each time they use the

CEA they are connecting under a different address.

Navigation

In the most common scenario, a CEA will have a top level page for the display application. This page remains visible on the screen for the public, even when users are interacting with the information. Users can "navigate" through the information because mouse movements from the users' mobile devices are captured and sent to the CEA (see Section 5). This positioning information is used to determine the actions intended by a user.

Pages below the top level or home page of the CEA are viewed on a user's mobile device. Depending on the type of the device and the session-based attributes defined by the user, the CEA sends the appropriate viewing format. For example, if a user

is connected to a CEA via a cell phone, then the CEA will respond to a request by sending pages with a specific format, in this case, WAP. This processing can be done

by the CEA, since its computational power outweighs that of a mobile device.

Other possibilities for navigating and viewing information depend on the type of

CEA deployed. Some CEA's may have space set aside on their display for users to

view information. This space can be used when session-based attributes are set to a lower level of privacy. This would mean that a user does not mind having their actions or navigation paths followed, perhaps because the information they are viewing is public or accessible by others. The designated space would have to be shared by all the users connected to a CEA and may require the implementation of a

scheduling system. This functionality is mentioned as a possibility but was not explored in detail for this project.

Downloading Content

In addition to viewing information on a CEA, a user can download information pertaining to the displayed information. Some possible examples of data

downloaded are information about events, advertisements, news, etc. Depending on the use of a CEA, a variety of information can be made available.

HOME PAGE _EET

abstract calendar launch

text sync email

Phone PC

1 2 3

Figure 2: Multiple functions exist within a single a location on a CEA.

The unique behavior of a CEA is that the function behind a button, link, or pixel of

its display can have different results (see Figure 2). In past models, a position (x, y)

of a display has a particular value associated with it, such a denoting a color. The

current approach has a particular (x, y) coordinate associated with a value and a

function, {(x, y), val, function}. This function is what makes displays and web pages

dynamic. Touching a button or clicking a position on a page calls a function

associated with that position. Similar to current representation, on a CEA a

position (x, y) is associated with a value and a function. The difference is that

functions on a CEA also take in parameters. These parameters a passed in by the

user as attributes, defined in Section 4.2. This means that on a CEA a point can

appear to have multiple functions, {(x, y), val, ({attr}, function)). Since attributes

define the type of information downloaded, one person may synchronize their

calendar with the CEA events calendar, while another person might download email

information regarding a particular event.

Uploading Content

In addition to downloading information and user can upload content depending on the settings of a CEA. When a user uploads content they can specify action-based attributes in order to create or add functionality to a particular pixel or position (x,

y) associated with that content. The attribute associated with the information a

person uploaded must be unique and not currently in use by the CEA. Users can also assign a value to a particular point on the CEA, but that requires additional access privileges on the side of the user. Content that is meant to be displayed and edited by particular people would need to have to proper attributes and permissions associated with it.

Modes of Operation

There are various modes in which a CEA can operate. Each mode pertains to specific functionality that depends on the configuration and purpose of the CEA.

The basic level of activity is when the CEA is in the display mode. In this mode the

CEA displays static information for the public. The information can be updated or

changed by system administers, but appears static to the user during a session. When in the display mode, the user may have limited interaction with the information, since the primary purpose is to view what is being presented.

In the receive mode, the CEA performs a partial set of the capabilities. The purpose of a CEA in this mode would be to distribute information to users. Users would only be able to download available information. Since users cannot upload information in

this mode, the content that they download is virtually static unless changed by the

CEA administrator.

The exchange mode is when the CEA is at full activity. In this mode, users can freely download and upload information. Downloading information is similar to the

process when the CEA is in the receive mode. When uploading information, users can specify the format and type of information by sending the appropriate action-based attributes with the request. The CEA interprets the attributes and store the information accordingly. Users also have the option of uploading collaborative information to leave for other users or the CE administrators.

Chapter 4

Functional Implementation

The pilot system that was implemented is focused on building a connection

framework for the CEA. On top of this framework, applications and extensions can

be built (see Section 5). The backbone developed defined the protocol of establishing

a connection between a CEA and a mobile client and extending that to building a

personal area Bluetooth network. A connection establishment protocol is defined in

detail because of its important role in public display systems such as the Content

Exchange. Designing a way to create and manage connections between mobile

clients and a CEA is difficult because of the system's unique requirements.

The default communication medium used in the pilot is the Bluetooth protocol, along

with various software modules and interfaces to help establish connectivity among

the devices. Bluetooth was chosen because it has a built in a device discovery

protocol and supports efficient close range communication. Devices that are

Bluetooth enabled, such as Compaq Ipaq, Nokia Cell Phone, and the Acer Tablet PC,

fall into the appropriate category of devices that are well suited for the purposes of a

data exchange system.

4.1

Bluetooth Stack & Interfaces

There are two main Bluetooth stacks available for Linux, which is the default operating system used for this research. The stacks, OpenBT, contributed by Axis, and Bluez, originally contributed by Qualcomm, are both open source and have strengths and weaknesses of their own. Bluez was the stack chosen for the

prototype of the Content Exchange. Bluez, although new compared to OpenBT, has several benefits that are relevant to the requirements of a CEA. Bluez is included in the standard Linux kernel which allows for better integration. Second, the stacks have different interfaced for drivers and applications. OpenBT uses a serial

abstraction, and BlueZ uses a network abstraction [website 1]. The network abstraction in Bluez is a key benefit because establishing connectivity between various devices is at the center of making the Content Exchange system work.

Bluez achieves a solid network abstraction over Bluetooth by using various profile and interface tools. PAN is the Bluetooth profile that defines how to do networking on the Bluez stack. PAN achieves IP support over Bluetooth, by using an interface called BNEP. BNEP is an interface, similar to ETH, which encapsulates Ethernet frames in an L2CAP session. L2CAP, Logic Link Control and Adaptation Protocol, runs on the control data link layer, and is a control protocol used for Bluetooth and sometimes Ethernet [12]. Finally a driver, either hci_uart or hci_usb was needed, depending on how Bluetooth is connected.

The BNEP interface for the mobile devices and the CEA are pre-configured with the following commands:

%> modprobe bnep

%> ifconfig bnepO <PAN IP ADDR>

The first command starts a new Ethernet emulation and the second command

configures TCP/IP on the newly created bnep interface. The PAN IP ADDR is an

internal IP address unique to the personal area network of a specific CEA. In the

pilot system a default PAN IP of 10.0.0.1 was given to the CEA. Once the interfaces

are configured, the devices and CEA can run their respective exchange applications.

4.2

CEA Discovery

A user with a handheld device detects a CEA when it is in local proximity to the

display, via Bluetooth. The user can manually connect to a CEA or rely on the

device discovery mechanism in Bluetooth. In either case, the discovery is done by

the user's device and not the CEA. By giving users the option to connect to a CEA

they can be assured that their privacy is protected.

Device discovery of Bluetooth works because the CEA is configured as a Bluetooth

master. As a master, the display is responsible for maintaining the frequency

hopping of its piconet. The PAN profile is used to run a server or daemon process on

the CEA so that it is constantly listening for incoming Bluetooth connection

requests. This process remains running despite the activity of the clients connected

to the CEA because it must stay ready for incoming connection from other clients.

A sequence of commands is issued after the BNEP interface is setup, to allow mobile

%> pand -master

%> pand -listen

These commands start a process that remains running on the CEA as long as it is powered and functioning. It is important to have these processing running in order to accept new connection from mobile Bluetooth enabled devices.

On the mobile client side, the users' handhelds are most often configured as

Bluetooth slaves. As slaves the mobile devices can also use the PAN profile in order to connect to the CEA. The mobile client has a loop that is constantly scanning the

environment in order to keep a look out for Bluetooth devices, using the command,

%>hcitool scan Scanning ...

00:80:37:B5:A8:3A CEA1-LCS

As shown, this command lists the descriptors and addresses of the Bluetooth devices that are in a local proximity to the mobile client. This means that the result of a Bluetooth scan will be a list of CEA's as well as the devices of the viewers using these displays. When other Bluetooth devices are detected, the client application prompts the user with a list of available Bluetooth enabled devices, listed the CEA marked entries first.

If there is only one CEA in the area, then depending on the user's settings, the client application will connect to that display. The default setting on the client is to automatically connect to the display if there is only one option. Users can change their settings ahead of time.

If multiple CEA devices are found during the Bluetooth scanning loop, then the

client can use local proximity as criteria to choose a CEA. The signal strength of the Bluetooth signal can be used to the degree of closeness, which would suggest that a user is closer to one CEA over another. Similar to other settings, the user can change the default setting from automatically connecting to the closest display, and instead choose to see a list of the options before connecting.

Finally in the case where the user has a hard time distinguishing between CEA's and other mobile clients, the client will have to parse out the CEA devices from the scan results. Before presenting a list to the user, the client application can use the device descriptor field to determine whether the corresponding Bluetooth address is a CEA or another mobile client.

If the user is presented with a list of possible CEA and Bluetooth devices, the user has the option of connecting to a CEA or not. The user must decide which Bluetooth device to connect to if there are multiple options. If the user decides not to connect to the display at that time, they have the option to continue scanning for Bluetooth devices or to quit the client application altogether.

4.3

Establishing a Channel with a CEA via Bluetooth

Once a CEA has been discovered by the client application, it is up to the client

application on the handheld to initiate a Bluetooth link before establishing a communication channel. There are a series of steps in the communication protocol that must be followed in order to ensure a successful connection to a CEA. The

connection protocol moves the client to and from various transition states, see

Figure 3.

detect set name TCP Socket

CEnA to ST ooenec

r

so

Si

S23S

Scan for BT Send

nBeeti' T SET Nait for Wonct ACX to

ecan c t~tno r

BT disconnect

Figure 3: This figure represents the transition states of the CE client application.

When establishing a connection, the client starts in SO when scanning for devices.

The client ends in S4 when it has received an ACK from the CEA, confirming that a

connection channel has been established.

While the client application is scanning for Bluetooth devices the client remains in

the initial state, state 0. The first step in connecting to a CEA occurs after the client

application detects the presence of a CEA. After the discovery takes place the client

moves to state 1. In this state, if the user has decided to connect to a CEA, the client

application chooses a random IP address from 10.0.0.2 to 10.0.0.255 to use for

communicating in the CEA's personal area network. If the user decides not to

connect to a CEA then the client returns to state 0. The random number is the IP

used to configure bnep interface of the client as described in Section 4.1. Once

interface is configured, the client changes the name of the device to the PAN IP

address using the command,

%> hciconfi -a hciO name 10.0.0.*

The name of the device is changed in order to provide the CEA with the IP

changed the client transitions to state 2, where the client issues a Bluetooth connection request by calling,

%> pand -c <bdaddr>

The command is part of the PAN profile and the argument, bdaddr, is the Bluetooth device address of the CEA. The client application gets the Bluetooth device address of the CEA from the scan request it loops through when searching for a CEA. After issuing the Bluetooth connection request, the client moves to state 3, where it waits for a response from the CEA. When a Bluetooth connection is received by the CEA, the device address and name of the handheld are made available, where the name is the PAN IP address of the mobile client requesting a connection.

Before a communication channel can be established, the CEA must check the validity of the device that is requesting the new connection. The CEA needs to ensure that the IP address submitted by the client is not currently in use within its network of devices. Second, the CEA must also make sure that there isn't another active session affiliated with the Bluetooth device address of the new client

If there is an IP address conflict that signals the CEA that the new client has chosen an IP address that is already in use. The CEA simply kills the Bluetooth connection using the command,

At this time the client is in state 3, waiting for a response from the CEA. If the client application has waited for longer then 10 seconds then it checks the Bluetooth connection it made with the CEA. Since the CEA just ended the connection, the client is transitions back to state 1 where it will generate a new IP address and then try to progress again. On the other hand, if the Bluetooth connection has not been killed, then client may experience delays because of an error with the CEA or the TCP socket request. In this case the client kills the Bluetooth connection and transitions to state 2, where it requests the connection again. The client does not need to go back to state 1 in this case because the delay was not necessarily due to

an IP conflict.

If the display application finds a Bluetooth device address conflict then the CEA does not know whether the new client has changed it's Bluetooth address or if it is an old client that has recently crashed and is trying to reconnect. The CEA then pings the IP on record for that particular Bluetooth address. If the ping is

successful, then the CEA kills the Bluetooth connection request made by the new client. Disconnecting the mobile device sends the client application back to state 1. On the other hand, if there is no response to the ping request, the CEA assumes that the old client is no longer connected to the network and therefore the CEA can reassign the IP to the new client. The CEA now continues with the connection procedures as would occur if there was no conflict.

If there are no conflicts with the IP addresses and the Bluetooth addresses then the

from the CEA. Without a ping from the display, the client would not be able to reach the CEA via its IP address. If the ping request is successful the server opens a TCP socket connection to the mobile device using the PAN IP address, in this case,

10.0.0.*. The CEA informs the client that the connection request has been accepted by sending a "CONNETACCEPTED" message. The CEA temporarily stores

information about the client's Bluetooth and IP addresses (see Partial State of the System).

Now the client device can communicate with the CEA. When the client receives the

"CONNECTACCEPTED" message, it transitions to state 4. In this state, the client

sends an ACK, or acknowledgement, back to the CEA. Once this is complete a connection channel has been established between the CEA and the client device.

4.4

Security

In order to provide viewers a minimal level of security and protocol was designed to

help guard against adversaries. After a connection channel is established between a

client and a CEA, various factors can effect the status of the channel. For example

the user could walk away from the display, the client application could fail, the

mobile device could power down, or various other random events. In order to preserve the security of a connection channel, the user and the CEA negotiate a secret key and nonce (see Figure 4). The negotiation tells the CEA that the mobile device connected on the other end is in fact the expected device and not someone pretending to be that device.

CEA

Mobile

Device Step 1 Cient

(Shared Key) -, Step 2 generate notice Step 3 e- _(nonce), -Step 4 decrypt nonce ACK Data 4- (nonce -text)S K

Figure 4: This figure illustrates the negotiation between the CEA and a mobile

client that takes place to ensure a secure connection channel in the CE.

The negotiation takes place after a connection channel has been established, which

is when the client has reached state 4, as described in Section 4.3. When the

connection channel is first established the Secure Socket Layer protocol is used to

keep the connection secure. In step 1 of the negotiation is the CEA sends a 32-bit

shared key to the client device over the created TCP socket. The shared key is

encrypted with the client's SSL public key. Once the shared key is received, the

client decrypts uses its private key to decrypt the message. From this point forward

SSL, a computationally expensive protocol, is no longer necessary because the client

and the mobile device can use the shared key to encrypt their messages. In step 2,

the client generates a random 32-bit number as a nonce. In the next step, the client

sends the nonce encrypted with the shared key to the display application. Here,

encryption is simply the XOR of the shared key and the nonce. In step 4 the CEA

receives and decrypts the nonce message using the previously sent shared key. The

as authentication. In the final step, the CEA sends and ACK message to the client

which acknowledges the receipt of the nonce.

Now that a secure channel has been established, all data sent by a client for a CEA

will have the nonce at the beginning of the transmission. The nonce will signal to

the CEA that the data it is about to receive is valid because it is coming from the

correct client.

4.5

Partial State of the System

The partial state of the Content Exchange system is necessary because information

regarding the mobile clients and the public displays is temporarily stored by the

client and display applications. Information is not permanently stored because of

the dynamic environment in which a CEA is deployed.

The client application keeps track of the history of displays that a user has visited.

The table, PASTCONNTABLE, stores the name of the CEA, the Bluetooth device

address of the CEA, and the BNEF IP addresses the client used to connect with the

CEA, see Figure 5. The client also saves the nonce and shared key that was

exchanged in case it can still be used.

CEA

CEA

BNEPO

Nonce

Secret Key

Name

btaddr

IP

(32-bit)

(32-bit)

Figure 5: This figure shows the table, PASTCONNTABLE, which is stored on the

client which allows for speedy re-connection.

The table is preserved until the user quits the client application or the mobile device

dies.

Storing the last used IP for a previously visited CEA will save the client time when

it tries to reconnect to that display, further discussed in Section 5.

On the CEA, the application stores information about the client, the connection, and

the user's device. The nonce and IP information affiliated with a particular

Bluetooth device address is saved in the CEA's CONNECTIONTABLE, see Figure

6. Each record is saved with a timestamp in order to keep the records up to date.

The expiration of the timestamp is determined by the settings of the CEA which

depends on the environment in which the CEA is deployed. The nonce is stored in a

table and mapped to the PAN IP address of the client. The nonce is used to

determine the authenticity of messages coming from a particular PAN IP.

Client IP

Client

Nonce

Secret Key

Timestamp

address

btaddr

(32-bit)

(32-bit)

Figure 6: This figure shows the format of the CONNECTIONTABLE stored on the

CEA. This table helps determine the validity of incoming requests and the

expiration of previously used nonces and secret keys.

The CONNECTIONTABLE also maps clients' Bluetooth device addresses to their

PAN IP addresses. This mapping allows the CEA to recognize mobile clients that

have visited in the recent past.

Session-based attributes, defined in Section 3.3, are also stored temporarily by the

repeated information with the action requests when navigating through the

information on a CEA. A set of session-based attributes is only active during a

single session or interaction between a user and a display.

Chapter 5

Analysis & Extensions

The Content Exchange was designed to propose a new way for users to transfer data

into and out of a public kiosk type device. The public display component added

different user scenarios and usability requirements to the standard information

repository problem. Multiple users accessing and interacting with the information

simultaneously adds complexity to the system. The concept of a public display

required devising proper protocols to protect the viewers and the CEA.

Efforts for the project are focused on furthering the development of pervasive computing

technologies. The Content Exchange system can incorporate the different pervasive

computing technologies of the Oxygen Research Group, such as location-aware

support, speech recognition, and face and gesture detection. This section discusses

the tradeoffs of the design and prototype of the CEA, and then proceeds to address

possible extensions to the system.

5.1

Tradeoffs with Bluetooth

The Bluetooth prototype described in Section 4 provided the means necessary to

achieve many of the original design goals for the Content Exchange. Using

Bluetooth made device discovery in a local proximity possible. Although the device

scanning described in Section 4.2 successfully identifies CEA's in the area, all other

Bluetooth enabled devices are also found. The option presented in Section 4.2 solves

the problem by using the Bluetooth device descriptor to extract the CEA entries.

Despite its feasibility, relying on a descriptor suggests that a standard naming

system is defined for every CEA, where the name is set to the value of the Bluetooth

device descriptor. Another possible solution is to scan for Bluetooth devices that

have been declared as Bluetooth masters [footnote?]. Since each CEA is supposed to

be a master of its own piconet, then scanning for masters will return a list of display

devices. Using this mechanism, mobile clients would not be detected because they

are configured as Bluetooth slaves.

Although Bluetooth was the preferred communication protocol used in this project,

using Bluetooth did pose certain networking limitations. According to the Bluetooth

specifications a piconet can consist of one master and a maximum of 7 slaves

[Bluetooth Spec]. The limit on the number of Bluetooth connections for a given CEA

limits the number of viewers that connect to the display using Bluetooth. This

means that once the 7 slots are taken, users with Bluetooth enabled devices will

have to wait until someone leaves, try connecting to a different CEA, or connect to

the same CEA using a different communication protocol, such as 802.11 or IR

Another limitation of Bluetooth was that there are different proprietary stacks that

can communicate over Bluetooth. Further research into Bluetooth needed to be done

in order to devise a way to communicate directly over the Bluez stack. Instead the

PAN profile on the BlueZ was used to assist with the prototyping effort, since it

communication directly over the Bluetooth stack was not achieved, using the BlueZ stack was both necessary and beneficial since it provided the foundation for

establishing a personal area network for each CEA. The PAN and bnep modules gave the CE the appropriate level of user control. Using the PAN module made development work simpler because communication from software could be done through standard TCP sockets. The BlueZ stack was also easy to integrate into a

common mobile device used in the testing environment, Compaq's Ipaq.

On the other hand, using BlueZ also requires setup time when the CEA's are powered up. This is understandable since a CEA is designed to stay powered on most of the time. For mobile clients that are using BlueZ the initial setup is auto-configured with the exchange client application. Another drawback to developing a solution for BlueZ is that this stack is not used on every Bluetooth enabled device. The portability of the exchange client applications to other Bluetooth stacks is not certain. Separate client applications may have to be written for various Bluetooth stacks. Although the programming logic would be similar, the connection and scanning commands may differ among the different stacks. This is evidence that further supports the argument for developing a Bluetooth industry standard.

5.2

User-Experience

An important goal for the CEA was to establish and maintain a seamless end-user experience. Each user is supposed to remain oblivious to the variability of each CEA's personal area network. The connections of each CEA are dynamic since

users come and go at random times. In the current design, viewers are not affected

by the arrival or departure of other users.

The system is designed to help make the users visit to each CEA enjoyable. The partial state of the system helps users achieve a good degree of mobility without many repercussions. The table maintained by the client application that stores a history of the displays visited saves the user time when revisiting a display. The client saves time because instead of attempting to connect with random IP address that might conflict with other devices, the client can use the IP address in it has on record for the corresponding display. The history table on the client only remains refreshed while the client application stays running. This helps ensure that the

PAN IP addresses stored in the table are up to date and can be used with confidence.

In addition to saving time upon connection, the stored information also saves the client from having to negotiate a shared key and nonce. The partial state of the system has helped make the user experience efficient and friendly.

A user may experience delays when trying to connect to a CEA if the randomly

generated IP is already in use. Although the probability of two devices picking the same PAN IP is low, the IP must also be sure not to overlap with LAN and WLAN IP's that are in use. The issue of IP overlap or collisions is important and can effect the performance of the CE. In this situation, the client applications can be

configured to chose random IP's from unregulated space. The current

range of possible PAN IP addresses would further reduce the chances of IP

collisions.

Another point in the system where users may be required to input information or

experience delays is when they are trying to establish a connection channel with a

CEA. Connection to a CEA may not happen automatically because that depends on

the environment of the CEA. The client application is responsible for scanning the

area for Bluetooth devices and parsing the list for display devices, which may

require input from the user. In the other situation if the viewer is using the default

settings, then the display chosen by the client application may not be accepting

connections or may not be the CEA the user intended to connect to. These scenarios

occur when displays are in high traffic areas or when there are multiple displays in

close proximity to each other.

5.3

Pros and Cons of Customizations

Providing users with a high degree of customization and personalization was an

important goal for the Content Exchange system. The system was designed to give

users the opportunity to set their own desired level of privacy and connection

behavior. In addition, users could define any number of personal settings that

described their mobile devices, storage destination when downloading, data source

when uploading, and various other session-based attributes. With this level of

customizations come various tradeoffs and outcomes.

Processing the different user settings and customizations levels takes place on the

CEA. Several operations performed by the CEA's require a lot of computation,

processing and storage. For example, for a predetermined length of time CEA's must store information regarding every mobile client that has connected to it. This information is maintained whether the client is still connected or not. Second, CEA's must monitor their current connections to keep track of when a mobile client is active or not. In addition, CEA's also have to manage the session-based attributes for each connected device. These attributes must be accessed each attempts

download or upload data, because the client does not send all the attribute

information with every request. Finally, when a user navigates to a new page the

CEA must determine the format of the data sent to the mobile client so that it can be

correctly displayed on that particular device. By design the CEA has to do a lot of computation and processing, but this was done intentionally since CEA's are

supercomputers compared to mobile devices. This design benefits the users since mobile devices have less bandwidth, slower processors, and minimal power resources compared to a CEA.

In the design of the CE users can achieve a strict level of privacy, but that requires a higher degree of customizations on the user's part. The default is not insecure

communication, but if a user wants complete anonymous communication they need to take appropriate steps to make that happen. The decisions to make a high level

of privacy added work for the user was made because the common scenario for the CEA is to exchange public data in either a confined environment such as a company

open, users that care about their privacy would not mind making extra effort to

achieve that. Setting session-based attributes requires work done by the user as

well. This is an initialization step that can be completed just once if the user decides

to keep the settings the same. If the user wishes to achieve varying behavior at

different times, then they must spend the extra time to make that happen.

Although this requires effort by the user, it is argued that users desiring this level of

personalization would not mind spending time setting up their attributes. The goal

of the system was to provide a framework that allowed users to customize their

exchanges with a CEA.

5.4

Extensions to the CEA Network

The implementation of the CEA does allow for various networking extensions. The

current solution was implemented to handle separate piconets for each CEA. One

possible extension is to form ad-hoc networks composed of piconets that correspond

to different CEA's. In Bluetooth these ad-hoc networks are called scatternets. The

added benefit from forming scatternets is that information can be transferred

between two or more CEA's. A example scenario involves a user that has uploaded

data to one CEA but wants it to appear on a collection of other CEA's. Instead of

manually uploading information to each, if the CEA's formed a scatternet the data

could be forwarded to the respective CEA's. Synchronizing displays could also be

achieved using other communication protocols, such as 802.11 or LAN. Another

scenario that would benefit from connected CEA's is that data uploaded by the user

could follow the user around as they moved from CEA to CEA. Similarly a user can

set their mobile device to act as a slave-master bridge and start to communicate