Data sharing among switching devices

In addition to switching devices we look also at how to switch from a single-user device to a device which can be operated by multiple people. We can say that mobile phones are personal devices and hold various personal contents such as photos, videos, contacts, etc. Viewing this content on the small screen display of phones is acceptable for a single user and for assuring the privacy of browsing. However, when it comes to sharing the viewing with other users simultaneously, the limited phone UI and display fail to deliver acceptable user experiences as those in single-user scenarios. The small screen allows for two or three users to have a clear view of the displayed material without the hassle experienced when trying to locate and move more viewers around the phone to have direct line-of-sight to its screen. Several approaches and technologies have emerged to overcome thisover-the-shoulderform of interaction. The most noticeable is the use of projectors, whether built-in or externally attached, to allow the projection of content onto any surface [93], allowing thus a multi-user access. Another solution relies on

pairing the phone with a public display to present the content to be shared [94]. Both applications however encounter the limitation of having only one user in control of the interaction with the displayed content. A more conceivable approach is based on integrating the mobile phone with a multi-touch interactive table [95] which receives the content from the phone and lays it on the surface for multiple users to view and interact with; hence, removing the burden of the limited access. The research conducted here focuses on this integration and it utilises Bluetooth, near field communication (NFC), 3D content and 3D manipulation to facilitate it for a group learning interactive experience in the medical domain. The system presented demonstrates how mobile phones and interactive tables can cooperatively work together for more collaborative interaction experience for medical students with medical content on their phones. Unlike most existing systems, it manipulates 3D content rather than 2D ones. The student basically carries this content on her phone and shares it with others over the table to promote a group discussion.

She does so by touching an NFC tag on the table with her NFC-enabled phone to identify the Bluetooth address of the server to connect to. Once identified and connected the server receives and displays the transmitted 3D content on the surface of the table. Here a detailed anatomical model of the upper leg muscle and bone construction segmented from MRI scan data has been used. Students are capable then of manipulating the model and its components through different 2D gestures on the table. They can move, rotate, zoom in/out, enable/disable its parts, and change their transparency.

NFC

Bluetooth

Interactive Table

Remote Client Server Application

Figure 3.14:Overall work flow of the system.

The OSG graphics library has been used to implement the viewer that renders these 3D mod-els and facilitates its manipulations. This library handles windows-based single-input events through one mouse pointer. It cannot handle multiple input pointers and this is not useful when it comes to interacting with multi-touch tables. This limitation has been resolved here through the use of the TUIO⁶open-source framework as shall be detailed next. A screenshot of the actual application visuals is shown in figure 3.15.

6TUIO open-source frameworkhttp://www.tuio.org

Figure 3.15: The interactive table application showing leg muscles and bones and the corresponding ontol-ogy.

3.8 Service Distribution and Render Context Switching

Internet

Clients VTO Service

Figure 3.16:The VTO service accessible through the Internet.

In figure 3.16 the concept of the VTO service is given, where on the left side is the providing infrastructure which offers the VTO through the Internet to the clients on the right side. As shown the client devices may differ and therefore have different interface requirements upon

which the VTO service has to reply and provide an adapted interface for the specific device.

This is done through device profiles. The service offers two types of visualization types and several stages of adaptation in order to provide an optimized visualization based upon the device and network constraints. These constraints include limited computational capabilities, specific display sizes, network bandwidth and transfer speed. In continuation of the work in [96] this system extends on the ability of switching between rendering and streaming methods. The base adaptation provides static image transfer on request or as a pushed stream of images with variable quality, interval and different image formats are supported. The optimal streaming is based on the VP8 codec⁷ implementation and provides streaming of the 3D content utilizing a micro-buffer in order to reduce stutter yet retain a real-time interactive experience. The client

VTO Service

Figure 3.17:Top level services, providing VTO deployment in an on-request load balanced setup.

application is implemented in Java and is available as a standalone application and as an applet and provided by a HyperText Transfer Protocol (HTTP) server and runs within the browser. In both cases it is dependent on Java and Windows environment. The decoding module at the client application is provided as a C++ dynamic library and is loaded through a Java Native Interface (JNI) wrapper. Once the client application is loaded it connects to the VTO service manager and sends a message requesting for a VTO Service. The manager keeps a list of service launchers and forwards the message to one of the Service Launcher using a least-burdenedload-balancing scheme. The service-launcher than can assign an already active VTO Service back to the VTO-Service manager or execute a new VTO-VTO-Service process and return its connection address. For each server there is one Service Launcher and basically is a generic process starter, using inter process communication the Service Launcher keeps track of the spawned services and monitors their activity. The connection information given back to the VTO Service manager is repacked

7WebM & VP8 codec web-link:http://www.webmproject.org/

into a message to the client, upon which the client directly connects to the VTO Service. The VTO Service than informs the Service Launcher of being in use. The Service Launcher registers the VTO service of being in use, using a time-out and a periodic testing of the state of its spawned services in order to avoid double assignment of users and zombie processes. Optionally, the user can provide login credentials and load preferences from a former session directly. Whereas the Service Launcher resides on the same computer as its spawned processes, the VTO Service Manager and Session Manager can reside elsewhere in the network. In case of service failure, the VTO service process is killed (in case still active) and a new VTO service is spawned with the same access parameters. The client will automatically reconnect (periodic based) to the service and provide the current user state back to the Service upon connection. In figure 3.18 the VTO

Event Manager Communication

Figure 3.18:Server side internal modules.

Service internal components are depicted. At the base there is theCommunication Managerwhich is responsible for marshalling an unmarshalling messages coming from the Service Launcher or the active client connection. These are two separate connections, whereas the connection to the Service Launcher is an IPC based and the connection with a client Socket (using TCP) based.

The message is translated into an event object and given to theEvent Managerwhich inspects the event and decides how to process it. There are four types of messages, Adaptation, Simulation, Interface and Interaction. Adaptation events are messages that come from the client informing the adaptation manager to change certain parameters (these could be frame rate, resolution, compression type and quality). Simulation events are specific functions directly called by the client, often without any user input but rather the client informing the simulation to be set to certain parameters (initial setup, or recovery). The Interface event is a single event requesting the service to send the interface elements. The client provides what kind of interface it needs based on the requested display size. The interface elements are generated dynamically depending on the VTO Module capabilities (e.g. body sizing parameters are translated in to a set of sliders).

A basic set of 2D interface elements, such as sliders, buttons and selection boxes, are supported.

Interaction events are separated in two types’ 3D interaction and 2D interaction. The main difference is the 3D interaction tends to be a stream of events, e.g. handling the virtual camera and therefore has a different handling scheme. Whereas the 2D interaction is a translation of the client state’s 2D interface element to a set of parameters which is provided to the function bound the application logic.