T
HE
T
ANGIBLE
B
OX
:
TANGIBLE AND
TACTILE GRAPHICS FOR LEARNING
ACTIVITIES
C’est dans cet esprit que j’ai pris la décision d’entreprendre une campagne
d’information, de renseigner, faire voir, me faire comprendre. Ce fut une
résolution immense, qui n’a rien changé du tout, mais ce fut très important
pour la résolution, qui est une grande vertu.
Romain Gary (Emile Ajar). Gros-Câlin.
Chapter structure
1. Introduction2. Design rationale 3. Implementation
4. Designing applications for the Tangible Box 5. Examples of Tangible Box applications 6. Discussion and perspectives
Chapter 4 – The Tangible Box
172
1
INTRODUCTION
1.1
M
OTIVATIONS AND RELATED WORKTUIs have a strong potential to enhance learning, by making the participants more engaged, fostering collaboration or supporting learning-by-doing and hands-on activities. Despite a vast amount of research work on the benefits of TUIs to support learning for sighted students, there has been very little work on the benefits of TUIs for visually impaired students, mainly because of a lack of accessible TUIs–an issue that we discussed in Chapter 2, Part E. We think that the technical solutions as well as the interaction techniques and feedback that we proposed in the previous chapter, as part of the development of the Tangible Reels prototype, could serve as a starting point for the design of TUIs for visually impaired students. However, the Tangible Reels, like most TUIs, rely on a set-up that is somewhat bulky and takes up space. Besides, as stated in the discussion section, one limitation of this prototype was the impossibility to rely on visual feedback as we used a transparent glass. The size of the Tangible Reels also makes it necessary to use a relatively large surface, which limits the type of environments where the interface can be installed.
If TUIs are to be used in specialized educational centers, they must be low-cost and easy to install and calibrate (or, ideally, do not require any calibration at all). In addition, they should be as compact as possible so that several teachers may have a device at their disposal, instead of having a single device shared between different groups of students or teachers. They should also be adapted to a large range of users who might have different visual impairments (blind or low- vision) and ages (children, teenagers, etc.). Another important aspect is the range of learning activities that a TUI used within a specialized educational center should support. Although specifically designing a TUI for a particular activity and a particular subject such as geometry or geography can be interesting, notably concerning the affordance of the tangible objects, it also means that teachers will need to develop and own several TUIs whenever they want to investigate the use of tangible interaction for a particular project. Such an approach raises issues in terms of storage, but also in terms of cost, time and skills required to develop and implement each new interface.
A number of projects have been conducted to facilitate the deployment of affordable and compact tabletop TUIs, based on various approaches: using daily objects (e.g. [186]), projecting images on non-interactive surfaces (e.g. [271]), tracking fingers with a camera instead of a touch- enabled device, placing the camera above the surface for tracking objects and fingers instead of below to prevent the need of using dedicated surfaces, etc. Wilson [341] summarized the main advantages and disadvantages of existing technologies for tabletop TUIs. As we described in Chapter 2, Part D, 3, one common approach is to place a camera above the tabletop, usually by hanging it on a shelf or on the ceiling. According to Wilson [341], the main disadvantages of this approach are the following: the installation is difficult and requires special hardware; once the camera is installed, it is not possible to use the TUI in a different place; calibration must be done regularly to compensate for the camera’s potential movements; user’s heads, arms and body can occlude the camera’s field of view. Innovative solutions have been proposed to reduce issues concerning the installation and calibration of the set-up, such as integrating the camera into a
173
lamp that can be placed on any surface [245] or using a smartphone’s or tablet’s integrated camera (e.g. Osmo53), but occlusions cannot be avoided. Although “there has been no systematic analysis
of the true impact of such occlusions” [341], systems that rely on a camera placed above the tabletop may not be the most appropriate for visually impaired users, who, unlike sighted users, cannot see whenever their hands or head is occluding one tangible object and therefore cannot move their hands or head in consequence.
The second most common approach is to place a camera below the tabletop. Wilson [341] indicated that this approach results in set-ups that are difficult to construct, do not allow users to put their legs under the table, provide only a limited resolution because a diffuser surface is often used, require a dedicated surface, and, overall, “present manufacturing and distribution problems for a real product”. As we already discussed, such set-ups are particularly limited by the projector’s throw (if an image is to be projected) or by the camera’s focal length [274]. Other approaches include commercialized interactive tabletops, which are expensive, or touch-enabled devices, including tablets, which are limited in size. Multitouch screens are an interesting alternative but do not inherently support object tracking and so dedicated tangible objects must therefore be designed (e.g. [210]). Also, affordable technologies for tracking such objects usually rely on computer vision algorithms that identify “footprints”, which make it necessary for the objects to be relatively large. In addition, techniques and/or technologies to ensure the stability of this type of tangible object have not been investigated.
To compensate for these limitations, Wilson introduced PlayAnyWhere [341], a compact and self- contained tabletop system that can project images onto any surface and track fingers and objects. Instead of being placed above or below a tabletop, both the camera and the projector are embedded into a single device that is “sitting off to the side of the active surface”. Instead of tracking the fingers themselves, their shadows are analyzed, making it possible for the system to detect pointing gestures (one finger per hand). However, the system did not prevent occlusions and, like all systems that rely on computer vision algorithms, was dependent on lighting conditions.
On the basis that there is a shortage of self-contained and low-cost TUIs, we aimed to design and implement a TUI for visually impaired students that would fulfill, as much as possible, the criteria listed above. By developing such a TUI, we aim to provide a platform that could be used by teachers to diversify the range of educational activities proposed to their students, but also by researchers to specifically investigate the benefits of educational activities that are based on the manipulation of tangible objects in (collaborative) learning for visually impaired students.
1.2
R
ESEARCH QUESTIONSTaking into account these considerations, this project was driven by the following research questions:
- How to design a low-cost, self-contained and portable TUI for visually impaired students? To date, there is a lack of research concerning the development of TUIs that
could be used within specialized educational centers for visually impaired students. In
Chapter 4 – The Tangible Box
174
addition to practical constraints (e.g. installation and calibration), some constraints must be taken into account for the TUI to be adapted to visually impaired users, namely the stability of the tangible representation and the fact that the TUI cannot solely rely on visual feedback to enhance the tangible representation.
- How to support different learning activities for a variety of subjects and users? If
the TUI is to be used by different teachers, it must support various learning activities as well as various types of graphical representations. Therefore the TUI should not be overly specific but should instead provide a generic set of tangible objects that could be adapted to several activities, subjects and users.
- What is the design space of learning activities supported by the proposed TUI? So
far, we mainly investigated how a TUI could be used to enable users to reconstruct a map or a diagram, even though we also provided an initial investigation of interaction techniques for two other tasks (construction and annotation). With this project, we aim to further investigate the design space of tasks that can be supported by tabletop tangible maps and diagrams.
1.3
C
HAPTER STRUCTUREThis chapter is organized as follows. Firstly, we provide in section 2 a detailed description of the Tangible Box–the prototype that we developed–along with reasons why specific technical choices were made. In section 3, we describe the implementation of the prototype, in terms of hardware and software. In section 4, we report ideas suggested by five specialized teachers with whom we organized participatory design sessions. Based on these ideas, we provide a framework for the design of learning activities supported by the Tangible Box. In section 5, we describe in detail three applications. Finally, we discuss in section 6 the benefits and limitations of the Tangible Box, before describing a number of perspectives to improve and enhance this interface.
2
DESIGN RATIONALE
To fulfill the aforementioned criteria, a number of choices were made concerning the design of the Tangible Box, which is illustrated in Figure 4.1. In this section, we specify the design rationale of the different elements of the interface and, in each sub-section, discuss the benefits and drawbacks of the design choices that we made.
2.1
U
SING STABLE TANGIBLE OBJECTS THAT CANNOT BE OCCLUDEDIn the previous chapter, we reviewed various techniques that have been considered to make tangible objects stable. We notably relied on the work of Hennecke et al. [99] who investigated different approaches including magnets, glue, electro-adhesion and vacuum-based adhesion. When designing the Tangible Reels, we tested various solutions based on glue and electro- adhesion, and none of them proved satisfactory. Vacuum-based adhesion was successfully used for the design of the Sucker Pads, but requires sucker pads that must be used on glass surfaces only and that are too large to be used on a surface of moderate size, which is an important criterion if the TUI is to be portable.
175
Figure 4.1. Overview of the Tangible Box. a) speaker; b) keyboard; c) a tangible object, composed of two parts (one above the surface, in red, and one below the surface); d) one
element of the fastening system that holds the supports in place; e) support used for the activity (here, a raised-line clock); f) dedicated area where unused tangible objects can be
placed.
In this project, we decided to investigate the use of magnets as a way to make tangible objects stable. However, because one requirement was to track the tangible objects using a camera placed below the tabletop (to avoid occlusions), it was not possible to simply place magnets on top of a magnetic board, as it is not a transparent surface. We therefore decided to use two magnets for each tangible object: one is placed above the surface and can be manipulated by the user; the other is magnetically attached to the first one, on the other side of the interaction surface, i.e. below the surface (Figure 4.2). Whenever the user moves the upper magnet, the lower magnet moves with it. By affixing a tag under the lower magnet, a camera placed below the tabletop can track it and, indirectly, track the upper magnet.
The main advantage of these objects is that they are very stable and that the user cannot occlude them as the tag is fixed under the bottom magnet, which is under the surface and therefore unreachable. Also, they are very easy to assemble and are low-cost (the magnets we used cost 0.30 € each). The main disadvantage is that once an object is placed on the table, it cannot be lifted and needs to be slid over the surface to stay connected with its paired magnet. Although this obviously restricts the design space of interaction, it ensures the stability of the tangible representation while making it possible to easily move the objects. Another issue is that if the user lifts the upper magnet, the lower magnet becomes detached. To prevent this from happening, a magnetic sheet is glued under the surface: therefore, even if an upper magnet is removed, the corresponding lower magnet does not detach, as it is held by the magnetic sheet.
Chapter 4 – The Tangible Box
176
Figure 4.2. Each tangible object is composed of two parts, magnetically attached together. Left: the surface is between the two parts. Middle: 3D-printed objects with magnets inside. Right: view from below: a colored tag is affixed under the lower magnet for tracking and a
magnetic sheet is glued (in white) to prevent an object from becoming detached.
2.2
S
UPPORTING MULTIPLE GRAPHICAL REPRESENTATIONS,
TASKS AND DOMAINSIn Chapter 2, Part B, we described various materials that are currently used within specialized educational centers: these materials allow different types of activities, depending on whether they are static (raised-line, embossed or swell maps and diagrams) or updatable (German film and cork, magnet of self-adhesive boards). In addition, they are used for various subjects, including mathematics, geography, Orientation & Mobility, etc. Also, static tactile graphics can be used to represent any type of graphical representation and they remain the best way to make relatively complex graphical representations accessible to visually impaired users. Based on these observations, we investigated whether the tangible objects that we designed could be used in combination with different supports. By doing so, we not only aimed to adapt to current practices, but also to enhance traditional supports by making them interactive using tangible interactions, while at the same time building on their advantages.
Firstly, the tangible objects can be placed above and slide along static tactile graphics (e.g. raised-lines and swell graphics), without being blocked by the tactile elements. In addition, users can still feel when they are moving a tangible object across a tactile element. The main advantage of using static graphics is related to the principle of division of functionality [196], which specifies that “fixed information should be represented by immovable physical objects” and “directly manipulated data should be represented by [tangible objects]”. Using static tactile graphics with tangible objects, it is possible to represent fixed and complex information (such as grids or a country’s boundary) on the graphic while updatable and simpler information (such as data points or cities) is represented by tangible objects. By doing so, the complexity of the tangible representation can be greatly enhanced, and the static and traditional surfaces made interactive and updatable. Similarly, thin 3D-printed graphical representations such as the maps developed by Götzelmann [86] can be used.
The Tangible Box also supports the use of German film. The magnets are strong enough to stay attached even if an additional and thick surface is used, such as the drawing mat that is usually placed below German film. If the German film is used by the students themselves, the Tangible Box can support interactive drawing activities where the tangible objects serve as tools, for
177
example to measure distances between two points or to annotate a drawing. The tangible objects can also be used as tokens: similar to the example we gave for the static graphics, it is possible to use a sheet of German paper onto which a grid has been printed and to ask the students to place a set of data points to create and possibly edit a graph. The German film can also be used by the teachers themselves in order to quickly create an interactive worksheet.
Whether it is with static graphics, 3D-printed graphics or German film, low-vision students are provided with visual feedback. However, if the device is to be used by students with moderate low-vision only, tactile feedback may be unnecessary. In that case, it is possible to simply use the Tangible Box with regular paper, or, for more updatable surfaces, with whiteboard sheets. Finally, the Tangible Box can be used with “ad-hoc” surfaces, i.e. surfaces designed for a particular activity and that can be 3D-printed or made out of different materials. For example, a wooden piece can be placed above the tabletop and temporarily attached with Blu-Tack to represent a tangible timeline around which students have to place different tangible objects that represent a particular event. Such “ad-hoc” surfaces, or tactile guides, can also be used to delineate different working areas, restrict where the students can place or move the tangible objects, reduce the size of the surface, etc.
To adapt to these various supports, which can be of different sizes and thicknesses, a fastening system has been designed, which we describe in detail in section 3.1.2. It allows teachers and students to easily place a support on top of the Tangible Box.
2.3
I
NPUT DEVICES AND AUDIO OUTPUTUsers can interact with the system by moving the tangible objects. In particular, for each application, it is possible to use one object as a selection tool: whenever this object is placed next to another object, the piece of information associated with it is read (e.g. its name, its description, its corresponding value, etc.). In addition, a numeric keypad is placed on top of the Tangible Box and enables students to easily interact with the system (to switch from one mode to another, to trigger a particular command, to select the application to be launched, etc.). Concerning output, audio feedback is provided by a speaker embedded into the Tangible Box, and the volume can be adjusted by the users themselves.
2.4
S
UPPORTINGP
ORTABILITY AND EASY CALIBRATIONThe camera used for tracking the tangible objects must capture the whole surface from below, which may result in the interface being too high to be portable and practical, as the camera must be placed at a sufficient distance. To tackle this issue, two solutions were considered: using a system of mirrors or using a wide-angle lens. Although using a mirror is efficient to reduce the height of the tangible box, it requires the camera to be placed sufficiently far from the mirror, and therefore makes the interface larger than necessary. We therefore opted for a wide-angle lens (170°), which reduced the required distance to less than 15 cm. With this solution, the final height of the Tangible box is 21 cm, which is significantly less than common tabletop TUIs that use a camera placed below the tabletop.
Chapter 4 – The Tangible Box
178
In addition, to make the interface relatively low-cost, self-contained and portable, the Tangible Box does not require a laptop: it relies on a Raspberry Pi, which is a small, lightweight and affordable single-board computer. This makes it possible to embed it together with the other pieces of hardware into a single box, hence the name of the prototype. In addition, all the pieces of hardware, and notably the camera, are firmly attached so that the Tangible Box can be moved without damaging the components. Concerning the objects, those that are not used can be placed on a dedicated part of the tabletop and do not need to be removed and stored elsewhere. For similar reasons, the fastening system that holds the different surfaces in place does not include