and presentation 2
The basic proposition of digital imaging is summarized in Figure 2.1. Image data is captured, processed, and/or recorded, then presented to a viewer. As outlined in the caption, and detailed later, appearance depends upon display and viewing conditions. Viewing ordinarily takes place in conditions different from those in effect at the time of capture of a scene. If those conditions differ, a nontrivial mapping of the captured image data – picture rendering– must be imposed in order to achieve faithful portrayal, to the ultimate viewer, of the appear-ance of the scene (as opposed to its physical stimulus).
Figure 2.1 Image acquisition takes place in a camera, which
captures light from the scene, converts the light to a signal, and – in most cameras – performs
certain image processing operations. The signal may then be recorded, further processed, and/or distributed. Finally, the signal is converted to light at a display device. The appearance of the displayed image depends upon display conditions (such as peak luminance); upon viewing conditions (such as the surroundings of the display surface); and upon conditions depen-dent upon both the display and its environment (such as contrast ratio). It is common for the scene to be much brighter than the displayed image: The scene may be captured in daylight, with white at 30,000 cd·m-2, but a studio display produces white of just
100 cd·m-2. The usual goal of imaging is not to match the physical stimulus associated with the scene (say, at daylight luminance levels), but to match the viewers’ expectation of the appearance of the scene. Producing an appearance match requires imposing a nontrivial mapping – termed picture rendering– that maps scene luminance to display luminance.
Examine the flowers in a garden at noon on a bright, sunny day. Look at the same garden half an hour after sunset. Physically, the spectra of the flowers have not changed, except by scaling to lower luminance levels.
However, the flowers are markedly less colourful after sunset: Colourfulness decreases as luminance decreases.
This is the Hunt effect, named after the famous colour scientist R.W.G. Hunt. Images are usually viewed at a small fraction, perhaps 1⁄100 or 1⁄1000, of the lumi-nance at which they were captured. If the image is presented with luminance proportional to the scene luminance, the presented image would appear less colourful, and lower in contrast, than the original scene.
Giorgianni, Edward J., and Thomas E. Madden (2008), Digital Color Management: Encoding Solutions, Second Edition (Chichester, U.K.:
Wiley).
To present contrast and colourfulness comparable to the original scene, we must alter the characteristics of the image. An engineer or physicist might strive to achieve mathematical linearity in an imaging system;
however, the required alterations cause the displayed relative luminance to depart from proportionality with scene luminance. The dilemma is this: We can achieve mathematical linearity, or we can achieve correct appearance, but we cannot simultaneously achieve both! Successful commercial imaging systems sacrifice mathematics to achieve the correct perceptual result.
Image state
In many professional imaging applications, imagery is reviewed and/or approved prior to distribution. Even if the image data originated with a colorimetric link from the scene, any technical or creative decision that results in alteration of the image data will break that link.
Consider the movie Pleasantville. Colour is used as a storytelling device. The story hinges upon characters depicted in greyscale and characters depicted in colour.
(See Figure 2.2.) The R’G’B’ values of the final movie do not accurately represent what was in front of the camera! This example is from the entertainment industry, however, examples abound whereever colour is adjusted for æsthetic purposes.
Picture rendering is ordinarily a nonlinear operation, not easily described in a simple equation or even a set of equations. Once picture rendering is imposed, its parameters aren’t usually preserved. In many applica-tions of imaging, image data is manipulated to achieve
an artistic effect – for example, colours in a wedding photograph may be selectively altered by the
photographer. In such cases, data concerning picture rendering is potentially as complex as the whole original image!
The design of an imaging system determines where picture rendering is imposed:
• In consumer digital photography and in video production, picture rendering is typically imposed in the camera.
• In movie making, picture rendering is typically imposed in the processing chain.
ISO 22028-1 (2004), Photography and graphic technology– Extended colour encodings for digital image storage, manipulation and interchange.
If an imaging system has a direct, deterministic link from luminance in the scene to image code values, in colour management terminology the image data is said to have an image state that is scene referred. If there is a direct, deterministic linkage from image code values to the luminance intended to be produced by a display, then image data is said to be display referred.
Video standards such as BT.709 and SMPTE ST 274 (both to be detailed later) are at best unclear and at worst wrong concerning image state. Consequently, video engineers often mistakenly believe that video data is linked colorimetrically to the scene. Users of digital still cameras may believe that their cameras capture “science”; however, when capturing TIFF or JPEG images, camera algorithms perform rendering, so the colorimetric link to the scene is broken. What is important in these applications is not the OECF that once mapped light from the scene to image data values, but rather the EOCF that is expected to map image data values to light presented to the viewer.
Figure 2.2 Colour as a dramatic device. This image mimics the visual style of the 1998 New Line Cinema movie, Pleasant-ville. When the scene was captured, the characters in the background weren’t grey; they were rotoscoped in post-production. Image data has been altered to achieve an artistic goal.
High-end D-SLR cameras have provisions to capture “raw” data that has not been subject to picture rendering operations. These cameras are capable of capturing “science.”
EOCF standards
In imaging systems where imagery is subject to review or approval at origination, faithful presentation requires consistent mapping from image data to light – and in entertainment applications, from audio signal to sound – between the origination environment and the ultimate viewing environment.
Figure 2.3 depicts the basic chain of origination, approval, distribution, and presentation. Origination is depicted as a “black box.” The mapping from image data to displayed light involves an electro-optical conversion function (EOCF). It is clear from the sketch that faithful presentation requires matching EOCFs at the approval display and the presentation display. EOCF is thereby incorporated – explicitly or implicitly – in any image interchange standard. Faithful presentation also requires agreement – again, implicit or explicit – upon reference viewing conditions.
To make the most effective use of limited capacity in the “channel,” the EOCFs common in commercial imaging incorporate perceptual uniformity, a topic to which we now turn.
Entertainment programming
Entertainment represents an economically important application of imaging, so it deserves special mention here. Digital video, HD, and digital cinema all involve acquisition, recording, processing, distribution, and Figure 2.3 Image approval is based upon
the display at the culmination of the origination
process. (The entire origination process is depicted here as
a black box.) Upon approval, image data is mastered, packaged, and
distributed; these operations are transparent or near-transparent. Eventually, imagery is presented to the viewer. Image makers hope for faithful presentation of
what was reviewed and approved. There is not necessarily any reference to the original scene (if indeed there was a physical scene). In principle, the viewer should be able to compare the presented image to that which was approved.
presentation of programs. I’ll use the generic word
“program” as shorthand for a movie, a television show, or a short piece such as a commercial. Figure 2.4 above presents a sketch of the entire chain.
If a movie is “in production,”
then principal photography is not yet complete.
Production refers to acquisition, recording, and processing. In a live action movie, the term production may be limited to just the acquisition of imagery (on set or on location); processes that follow are then postpro-duction (“post”). In the case of a movie whose visual elements are all represented digitally, post production is referred to as the digital intermediate process, or DI.
Production culminates with display and approval of a program on a studio reference display – or, in the case of digital cinema, approval on a cinema reference projector in a review theatre. (If distribution involves compression, then approval properly includes review of compression at the studio and decompression by a ref-erence decompressor.) Following approval, the program is mastered, packaged, and distributed.
The word reproduction, taken literally, suggests production again! I propose presentation.
Professional content creators rarely seek to present, at the viewer’s premises, an accurate representation of the scene in front of the camera. Apart from makers of documentaries, movie makers often make creative choices that alter that reality. They hope that when the program completes its journey through the distribution chain, the ultimate consumer will be presented with a faithful approximation not of the original scene, but rather of what the director saw on his or her studio display when he or she approved the final product of postproduction. In colour management terms, movie and video image data is display-referred.
Production
Appr oval MasteringPackaging Post-production
(Digital intermediate)
Distribution Consumer presentation (Exhibition) Figure 2.4 Stages of production are depicted. In video, the final stage is presentation; in cinema, it’s called exhibition.
Acquisition
A person using a camera to acquire image data from a scene expects that when the acquired material is displayed it will approximately match the appearance of the scene. Luminance of white in an outdoor scene might reach 30,000 cd·m-2, but it is rare to find an electronic display whose luminance exceeds
450 cd·m-2, and professional HD content mastering and approval is performed with a reference white around 100 cd·m-2. Linear transfer of the scene nance to the display – in effect, scaling absolute lumi-nance by a factor of 0.015 or 0.01 – won’t present the same appearance as the outdoor scene. The person using the camera expects an approximate apppearance match upon eventual display; consequently, picture rendering must be imposed. In HD, and in consumer still photography, rendering is imposed at the camera;
in digital cinema and in professional (“raw”) still photography, rendering is imposed in postproduction.
Some people use the phrase “scene-to-screen” to describe the goal of delivering an accurate representa-tion of scene luminance and colour to the display.
Unless proper account is taken of appearance phenomena – that is, unless picture rendering is imposed – this effort is doomed to failure.
Consumer origination
Figure 2.5 summarizes consumer-originated video.
Consumers may exercise creative control through signal processing after acquisition; however, picture rendering is imposed by algorithms in the camera. The camera is engineered to encode signals for presentation in the consumers’ living room. Studio origination is built on an assumption of viewing at 100 cd·m-2 in a dark
surround (today, around 1%). Consumer camcorders incorporate picture rendering based upon comparable parameters.
Consumer electronics (CE) display
In the consumer electronics domain, there is a diversity of display devices (having different contrast ratios, different peak luminance values, and different colour gamuts), and there is a diversity of viewing
environ-ments (some bright, some dark; some having bright surround, some dim, and some dark).
Different consumer display devices have different default EOCFs. The EOCF for a particular product is preset at the factory in a manner suitable for the viewing conditions expected for that product. Tradi-tional domestic television receivers had EOCFs approxi-mating the 2.4-power function used in the studio;
however, modern consumer receivers are considerably brighter than 100 cd·m-2 (up to 350 cd·m-2 today) and the higher brightness necessitates a somewhat lower value of gamma, today typically between 2.1 and 2.3. A home theatre projector used in a rather dark environment will have characteristics comparable to those of a studio reference display (see Reference display and viewing conditions, on page 427), and will typically have “gamma” of about 2.4. A PC has a default sRGB EOCF, with gamma of 2.2.
Consumer television receiver vendors commonly impose signal processing claimed to “improve” the image – often described by adjectives such as “natural-ness” or “vividness.” However, the director may have thoughtful reasons for wanting the picture to look unnatural, pale, or noisy! Creative control is properly Figure 2.5 Consumer origination of
either still photographs or video has all of the issues of image acquisition that I outlined in Figure 2.1, but consumers rarely process or review
imagery before distribution and typically exercise no control over the parameters of image capture or processing. Algorithms in the camera impose picture rendering and incorporate the rendering into the image data. Those operations assume a certain display and viewing envi-ronment. That reference viewing environment is thereby incorporated (explicitly or implictly) into the image exchange standard.
exercised at production, not at presentation. Creative staff generally despise consumer processing that goes by such names as Digital Reality Creation (DRC) or Digital Natural Image engine (DNIe).
Digital Reality Creation (DRC) is a trademark of Sony Electronics Inc.
DNIe is a trademark of Samsung Electronics Co., Ltd.