TELEPOINTER: WEARABLE HANDS-FREE COMPLETELY SELF-CONTAINED VISUAL AUGMENTED REALITY

Not surprisingly, Mann was generally upbeat at least about the technical possibilities of distributed body-worn computing, showing that he had already developed a combination wristwatch and imaging device that can send and receive video over short distances.

Meanwhile, in the debate from the floor that followed the panel discussion, ideas were thrown up, such as shoes as a mobile phone — powered by the mechanical energy of walking, and using the Dick Tracy watch as the user interface — and a more distributed model where spectacles are used to provide the visual interface;

an ear piece to provide audio; and even clothing to provide a key-pad or display.

and finally appeared on the cover ofLinux Journal, July 2000, issue 75, together with a feature article.

Although it was a useful invention, the idea of a wristwatch videoconferencing computer is fundamentally flawed, not so much because of the difficulty in inventing, designing, and building it but rather because it is difficult to operate without conscious thought and effort. In many ways the wristwatch computer was a failure not because of technology limitations but because it was not a very good idea to start with, when the goal is constant online connectivity that drops below the conscious level of awareness. The failure arose because of the need to lift the hand and shift focus of attention to the wrist.

2.5 TELEPOINTER: WEARABLE HANDS-FREE COMPLETELY SELF-CONTAINED VISUAL AUGMENTED REALITY

The obvious alternative to the flawed notion of a wristwatch computer is an eyeglass-based system because it would provide a constancy of interaction and

allow the apparatus to provide operational modes that drop below the conscious level of awareness. However, before we consider eyeglass-based systems, let us consider some other possibilities, especially in situations where reality only needs to be augmented (e.g., where nothing needs to be mediated, filtered, or blocked from view).

The telepointer is one such other possibility. The telepointer is a wearable hands-free, headwear-free device that allows the wearer to experience a visual collaborative telepresence, with text, graphics, and a shared cursor, displayed directly on real-world objects. A mobile person wears the device clipped onto his tie, which sends motion pictures to a video projector at a base (home) where another person can see everything the wearer sees. When the person at the base points a laser pointer at the projected image of the wearer’s site, the wearer’s aremac’s¹ servo’s points a laser at the same thing the wearer is looking at. It is completely portable and can be used almost anywhere, since it does not rely on infrastructure. It is operated through a reality user interface (RUI) that allows the person at the base to have direct interaction with the real world of the wearer, establishing a kind of computing that is completely free of metaphors, in the sense that a laser at the base controls the wearable laser aremac.

2.5.1 No Need for Headwear or Eyewear If Only Augmenting

Using a reality mediator (to be described in the next section) to do only augmented reality (which is a special case of mediated reality) is overkill. Therefore, if all that is desired is augmented reality (e.g., if no diminished reality or altered/mediated reality is needed), the telepointer is proposed as a direct user interface.

The wearable portion of the apparatus, denoted WEAR STATION in Figure 2.4, contains a camera, denotedWEAR CAM, that can send pictures thousands of miles away to the other portion of the apparatus, denotedBASE STATION, where the motion picture is stabilized by VideoOrbits (running on a base station computer denoted

BASE COMP) and then shown by a projector, denotedPROJ., at the base station. Rays of light denotedPROJ.LIGHTreach a beamsplitter, denotedB.B.S., in the apparatus of the base station, and are partially reflected; some projected rays are considered wasted light and denoted PROJ. WASTE.Some of the light from the projector will also pass through beamsplitterB.B.S., and emerge as light rays denotedBASE LIGHT. The projected image thus appears upon a wall or other projection surface denoted asSCREEN. A person at the base station can point to projected images of any of theSUBJECT MATTERby simply pointing a laser pointer at theSCREENwhere images of the SUBJECT MATTER appear. A camera at the base station, denoted as BASE CAMprovides an image of the screen to the base station computer (denotedBASE COMP), by way of beamsplitter B.B.S. The BASE CAM is usually equipped with a filter, denoted FILT., which is a narrowband bandpass filter having a passband to pass light from the laser pointer being used. Thus theBASE CAM will capture an image primarily of the laser dot on the screen, and especially since a laser

1An aremac is to a projector as a camera is to a scanner. The aremac directs light at 3-D objects.

SUBJECT

Figure 2.4 Telepointer system for collaborative visual telepresence without the need for eyewear or headwear or infrastructural support: The wearable apparatus is depicted on the left; the remote site is depicted on the right. The author wears the WEAR STATION, while his wife remotely watches on a video projector, at BASE STATION. She does not need to use a mouse, keyboard, or other computerlike device to interact with the author. She simply points a laser pointer at objects displayed on theSCREEN. For example, while the author is shopping, she can remotely see what’s in front of him projected on the livingroom wall. When he’s shopping, she sees pictures of the grocery store shelves transmitted from the grocery store to the livingroom wall. She points her laser pointer at these images of objects, and this pointing action teleoperates a servo-mounted laser pointer in the apparatus worn by the author. When she points her laser pointer at the picture of the 1% milk, the author sees a red dot appear on the actual carton of 1% milk in the store. The user interface metaphor is very simple, because there is none. This is an example of a reality user interface: when she points her laser at an image of the milk carton, the author’s laser points at the milk carton itself. Both parties see their respective red dots in the same place. If she scribbles a circle around the milk carton, the author will see the same circle scribbled around the milk carton.

pointer is typically quite bright compared to a projector, the image captured by

BASE CAMcan be very easily made, by an appropriate exposure setting of theBASE CAM, to be black everywhere except for a small point of light from which it can be determined where the laser pointer is pointing.

The BASE CAM transmits a signal back to the WEAR COMP, which controls a device called an AREMAC, after destabilizing the coordinates (to match the more jerky coordinate system of the WEAR CAM). SUBJECT MATTER within the field of illumination of the AREMAC scatters light from the AREMAC so that the output of

AREMACis visible to the person wearing theWEAR STATION. A beamsplitter, denoted

W.B.S., of theWEAR STATION, diverts some light fromSUBJECT MATTERto the wearable camera, WEAR CAM, while allowing SUBJECT MATTER to also be illuminated by the

AREMAC.

This shared telepresence facilitates collaboration, which is especially effective when combined with the voice communications capability afforded by the use of a wearable hands-free voice communications link together with the telepointer apparatus. (Typically theWEAR STATIONprovides a common data communications link having voice, video, and data communications routed through the WEAR COMP.)

VIS.

Figure 2.5 Details of the telepointer (TM) aremac and its operation. For simplicity the livingroom or manager’s office is depicted on the left, where the manager can point at the screen with a laser pointer. The photo studio, or grocery store, as the case may be, is depicted on the right, where a body-worn laser aremac is used to direct the beam at objects in the scene.

Figure 2.5 illustrates how the telepointer works to use a laser pointer (e.g., in the livingroom) to control an aremac (wearable computer controlled laser in the grocery store). For simplicity, Figure 2.5 corresponds to only the portion of the signal flow path shown in bold lines of Figure 2.4.

SUBJECT MATTER in front of the wearer of theWEAR STATION is transmitted and displayed as PICTURED SUBJECT MATTER on the projection screen. The screen is updated, typically, as a live video image in a graphical browser such as glynx, while theWEAR STATIONtransmits live video of theSUBJECT MATTER.

One or more persons at the base station are sitting at a desk, or on a sofa, watching the large projection screen, and pointing at this large projection screen using a laser pointer. The laser pointer makes, upon the screen, a bright red dot, designated in the figure asBASE POINT.

The BASE CAM, denoted in this figure as SCREEN CAMERA, is connected to a vision processor (denotedVIS. PROC.) of the BASE COMP, which simply determines the coordinates of the brightest point in the image seen by theSCREEN CAMERA. The

SCREEN CAMERAdoes not need to be a high-quality camera, since it will only be used to see where the laser pointer is pointing. A cheap black- and white-camera will suffice for this purpose.

Selection of the brightest pixel will tell us the coordinates, but a better estimate can be made by using the vision processor to determine the coordinates of a bright red blob,BASE POINT, to subpixel accuracy. This helps reduce the resolution needed, so that smaller images can be used, and therefore cheaper processing hardware and a lower-resolution camera can be used for theSCREEN CAMERA.

These coordinates are sent as signals denoted EL. SIG. and AZ. SIG. and are received at the WEAR STATION. They are fed to a galvo drive mechanism (servo)

that controls two galvos. Coordinate signal AZ. SIG. drives azimuthal galvo AZ.

Coordinate signal EL. SIG.drives elevational galvoEL.These galvos are calibrated by the unit denoted asGALVO DRIVEin the figure. As a result theAREMAC LASERis directed to form a red dot, denoted WEAR POINT, on the object that the person at the base station is pointing at from her livingroom or office.

TheAREMAC LASERtogether with theGALVO DRIVEand galvosELandAZtogether comprise the device called an aremac, which is generally concealed in a brooch pinned to a shirt, or in a tie clip attached to a necktie, or is built into a necklace.

The author generally wears this device on a necktie. The aremac andWEAR CAM

must be registered, mounted together (e.g., on the same tie clip), and properly calibrated. The aremac and WEAR CAM are typically housed in a hemispherical dome where the two are combined by way of beamsplitterW.B.S.

2.5.2 Computer-Supported Collaborative Living (CSCL)

While much has been written about computer-supported collaborative work (CSCW), there is more to life than work, and more to living than pleasing one’s employer. The apparatus of the invention can be incorporated into ordinary day-to-day living, and used for such “tasks” as buying a house, a used car, a new sofa, or groceries while a remote spouse collaborates on the purchase decision.

Figure 2.6 shows the author wearing theWEAR STATIONin a grocery store where photography and videography are strictly prohibited. Figure 2.7 shows a close-up view of the necktie clip portion of the apparatus.

Figure 2.6 Wearable portion of apparatus, as worn by author. The necktie-mounted visual augmented reality system requires no headwear or eyewear. The apparatus is concealed in a smoked plexiglass dome of wine-dark opacity. The dark dome reduces the laser output to safe levels, while at the same time making the apparatus blatantly covert. The dome matches the decor of nearly any department store or gambling casino. When the author has asked department store security staff what’s inside their dark ceilings domes, he’s been called

‘‘paranoid,’’ or told that they are light fixtures or temperature sensors. Now the same security guards are wondering what’s inside this dome.

Figure 2.7 Necktie clip portion. The necktie-mounted visual augmented reality system. A smoked plexiglass dome of wine-dark opacity is used to conceal the inner components. Wiring from these components to a body-concealed computer runs through the crack in the front of the shirt. The necktie helps conceal the wiring.

2.6 PORTABLE PERSONAL PULSE DOPPLER RADAR VISION