Exact Identity Mapping Over a Variety of Depth Planes

In order to better facilitate rapid switching back and forth between the mediated and unmediated worlds, particularly in the context of a partially mediated reality, it was desired to mediate part of the visual field without alteration in the identity configuration (e.g., when the computer was issued the identity map, equivalent to

(a) (b)

Figure 3.12 Illusory transparency. Examples of a camera supplying a television with an image of subject matter blocked by the television. (a) A television camera on a tripod at left supplies an Apple ‘‘Studio’’ television display with an image of the lower portion of Niagara Falls blocked by the television display (resting on an easel to the right of the camera tripod). The camera and display were carefully arranged by the author, along with a second camera to capture this picture of the apparatus. Only when viewed from the special location of the second camera, does the illusion of transparency exist. (b) Various still cameras set up on a hill capture pictures of trees on a more distant hillside on Christian Island. One of the still cameras having an NTSC output displays an image on the television display.

a direct connection from camera to viewfinder), over a variety of different depth planes.

This was accomplished with a two-sided mirror. In many embodiments a pellicle was used, while sometimes a glass silvered on one or both sides was used, as illustrated in Figure 3.14.

In this way a portion of the wearer’s visual field of view may be replaced by the exact same subject matter, in perfect spatial register with the real world. The image could, in principle, also be registered in tonal range. This is done using the quantigraphic imaging framework for estimating the unknown nonlinear response of the camera, and also estimating the response of the display, and compensating for both [64]. So far focus has been ignored, and infinite depth-of-field has been assumed. In practice, a viewfinder with a focus adjustment is used for the computer screen, and the focus adjustment is driven by a servomechanism controlled by an autofocus camera. Thus the camera automatically focuses on the subject matter of interest, and controls the focus of the viewfinder so that the apparent distance to the object is the same when seen through the apparatus as with the apparatus removed.

Figure 3.13 Various cameras with television outputs are set up on the walkway, but none of them can recreate the subject matter behind the television display in a manner that conveys a perfect illusion of transparency, because the subject matter does not exist in a single depth plane. There exists no choice of camera orientation, zoom setting, and viewer location that creates an exact illusion of transparency for the portion of the Brooklyn Bridge blocked by the television screen. Notice how the railings don’t quite line up correctly as they vary in depth with respect to the first support tower of the bridge.

It is desirable that embodiments of the personal imaging system with manual focus cameras also have the focus of the camera linked to the focus of the viewfinder. Through this linkage both may be adjusted together with a single knob. Moreover a camera with zoom lens may be used together with a viewfinder having zoom lens. The zoom mechanisms are linked in such a way that the viewfinder image magnification is reduced as the camera magnification is increased. This appropriate linkage allows any increase in magnification by the camera to be negated exactly by decreasing the apparent size of the viewfinder image. As mentioned previously, this procedure may seem counterintuitive, given traditional cameras, but it was found to assist greatly in elimination of undesirable long-term effects caused by wearing a camera not implementing the virtual light collinearity principle.

The calibration of the autofocus zoom camera and the zoom viewfinder was done by temporarily removing the double-sided mirror and adjusting the focus and zoom of the viewfinder to maximize video feedback. This must be done for each zoom and focus setting so that the zoom and focus of the viewfinder will properly track the zoom and focus of the camera. In using video feedback as a calibration tool, a computer system can be made to monitor the video output of the camera, adjust the viewfinder, and generate a lookup table for the viewfinder settings corresponding to each camera setting. In this way calibration can be automated during the manufacture of the personal imaging system. Some similar

Rightmost ray of light Leftmost

ray of light

Eye

Leftmost ray of virtual light

Rightmost ray of virtual light Diverter

Aremac

d

d Camera

Figure 3.14 The orthoscopic reality mediator. A double-sided mirror diverts incoming rays of light to a camera while providing the eye with a view of a display screen connected to the wearable computer system. The display screen appears backward to the eye. But, since the computer captures a backward stream of images (the camera’s view of the world is also through a mirror), display of that video stream will create an illusion of transparency. Thus the leftmost ray of light diverted by the mirror, into the camera, may be quantified, and that quantity becomes processed and resynthesized by virtue of the computer’s display output. This way it appears to emerge from the same direction as if the apparatus were absent. Likewise for the rightmost ray of light, as well as any in between. This principle of ‘‘virtual light’’ generalizes to three dimensions, though the drawing has simplified it to two dimensions. Typically such an apparatus may operate with orthoquantigraphic capability through the use of quantigraphic image processing [63].

embodiments of the personal imaging system have used two cameras and two viewfinders. In some embodiments the vergence of the viewfinders was linked to the focus mechanism of the viewfinders and the focus setting of cameras. The result was a single automatic or manual focus adjustment for viewfinder vergence, camera vergence, viewfinder focus, and camera focus. However, a number of these embodiments became too cumbersome for unobtrusive implementation, rendering them unacceptable for ordinary day-to-day usage. Therefore most of what follows will describe other variations of single-eyed (partially mediated) systems.

Partial Mediation within the Mediation Zone

Partially mediated reality typically involves a mediation zone (field of view of the viewfinder) over which visual reality can be completely reconfigured. However, a more moderate form of mediated reality is now described. In what follows, the mediation is partial in the sense that not only it affects only part of the field of view (e.g., one eye or part of one eye) but the mediation is partial within the mediation zone. The original reason for introducing this concept was to make the

apparatus less obtrusive and allow others to see the wearer’s eye(s) unobstructed by the mediation zone.

The apparatus of Figure 3.14 does not permit others to make full eye contact with the wearer. Therefore a similar apparatus was built using a beamsplitter instead of the double-sided mirror. In this case a partial reflection of the display is visible to the eye of the wearer by way of the beamsplitter. The leftmost ray of light of the partial view of the display is aligned with the direct view of the leftmost ray of light from the original scene, and likewise for the rightmost ray, or any ray within the field of view of the viewfinder. Thus the wearer sees a superposition of whatever real object is located in front of the apparatus and a displayed picture of the same real object at the same location. The degree of transparency of the beamsplitter affects the degree of mediation. For example, a half-silvered beamsplitter gives rise to a 50% mediation within the mediation zone.

In order to prevent video feedback, in which light from the display screen would shine into the camera, a polarizer was positioned in front of the camera.

The polarization axis of the polarizer was aligned at right angles to the polarization axis of the polarizer inside the display screen, in situations where the display screen already had a built-in polarizer as is typical of small battery-powered LCD televisions, LCD camcorder viewfinders, and LCD computer displays. In embodiments of this form of partially mediated reality where the display screen did not have a built in polarizer, a polarizer was added in front of the display screen. Thus video feedback was prevented by virtue of the two crossed polarizers in the path between the display and the camera. If the display screen displays the exact same rays of light that come from the real world, the view presented to the eye is essentially the same as it might otherwise be.

In order that the viewfinder provide a distinct view of the world, it was found to be desirable that the virtual light from the display screen be made different in color from the real light from the scene. For example, simply using a black-and-white display, or a black-and-green display, gave rise to a unique appearance of the region of the visual field of the viewfinder by virtue of a difference in color between the displayed image and the real world upon which it is exactly superimposed. Even with such chromatic mediation of the displayed view of the world, it was still found to be far more difficult to discern whether or not video was correctly exposed, than when the double-sided mirror was used instead of the beamsplitter. Therefore, when using these partially see-through implementations of the apparatus, it was found to be necessary to use a pseudocolor image or unique patterns to indicate areas of overexposure or underexposure. Correct exposure and good composition are important, even if the video is only used for object recognition (e.g., if there is no desire to generate a picture as the final result). Thus even in tasks such as object recognition, a good viewfinder system is of great benefit.

In this see-through embodiment, calibration was done by temporarily removing the polarizer and adjusting for maximum video feedback. The apparatus may be concealed in eyeglass frames in which the beamsplitter is embedded in one or

both lenses of the eyeglasses, or behind one or both lenses. In the case in which a monocular version of the apparatus is being used, the apparatus is built into one lens, and a dummy version of the beamsplitter portion of the apparatus may be positioned in the other lens for visual symmetry. It was found that such an arrangement tended to call less attention to itself than when only one beamsplitter was used.

These beamsplitters may be integrated into the lenses in such a manner to have the appearance of the lenses in ordinary bifocal eyeglasses. Moreover magnifica-tion may be unobtrusively introduced by virtue of the bifocal characteristics of such eyeglasses. Typically the entire eyeglass lens is tinted to match the density of the beamsplitter portion of the lens, so there is no visual discontinuity intro-duced by the beamsplitter. It is not uncommon for modern eyeglasses to have a light-sensitive tint so that a slight glazed appearance does not call attention to itself.

3.4 PROBLEMS WITH PREVIOUSLY KNOWN CAMERA VIEWFINDERS