This chapter continues the discussion of the JPEG format. In it we cover the structure of JPEG files, which includes every part of a file except for the compressed data. The descriptions here take into account only the subset of JPEG covered by this book, so you will find some discrepancies with the JPEG standard.
Markers
Markers are used to break a JPEG stream into its component structures. They are 2 bytes in length, with the first byte always having the value FF16. The second byte contains a code that specifies the marker type. Any number of bytes with the value FF16 may be used as a fill character before the start of any marker. A byte in a JPEG stream with a value of FF16 that is followed by another FF16 byte is always ignored.
JPEG markers can be grouped into two general types. Stand-alone markers consist of no data other than their 2 bytes; markers that do not stand alone are immediately followed by a 2-byte-long value that gives the number of bytes of data the marker contains. The length count includes the 2 length bytes but not the marker length itself. JPEG stand-alone markers are listed in Table 5.1 and mark-ers with data are listed in Table 5.2. You can see from the tables that most of the markers are not used in the modes we are working with.
The JPEG standard is fairly flexible when it comes to the ordering of mark-ers within a file. Its strictest rule is that a file must begin with an SOI marker and end with an EOI marker. In most other cases markers can appear in any order.
47
Chapter 5
48 JPEG File Format
The main requirement is that if the information from one marker is needed to process a second marker, the first marker must appear before the second. The specific rules regarding marker ordering are described in the next section.
Table 5.1
Start of frame, baseline
Start of frame, extended sequential Start of frame, progressive
Start of frame, lossless Define Huffman table
Start of frame, differential sequential Start of frame, differential progressive Start of frame, differential lossless Reserved
Start of frame, extended sequential, arithmetic coding Start of frame, progressive, arithmetic coding
Start of frame, lossless, arithmetic coding Define arithmetic coding conditions
Start of frame, differential sequential, arithmetic coding
Start of frame, differential progressive, arithmetic coding
Start of frame, differential lossless, arithmetic coding Start of scan
Define quantization tables Define number of lines Define restart interval
*Not used by any of the techniques covered in this book.
Value
Temporary for arithmetic coding Restart marker
Start of image End of image
*Not used by any of the techniques covered in this book.
We use the subscript n when referring to certain markers collectively. For example, RSTn means any of the restart markers RST0 to RST7; SOFn means any start of frame marker.
Marker Types
This section describes all of the markers used by the JPEG modes we are work-ing with. The tables that illustrate marker structure omit the 2-byte marker and the 2-byte length field.
The individual marker descriptions specify where in the file the marker can occur. They do not repeat the following restrictions:
• Only RST and DNL markers may appear in compressed data.
• An image must start with an SOI marker followed by an APP0 marker.2
• The last marker in the file must be an EO1, and it must immediately follow the compressed data of the last scan in the image.
APPn
The APP0-APP15 markers hold application-specific data. They are used by image-processing applications to hold additional information beyond what is
1The JPEG standard allows DNL markers to appear with compressed data. A DNL marker is used to define or redefine the image size within the compressed data rather than within the SOFn marker. In prac-tice DNL markers are not used and most applications cannot handle them. Therefore, we will not con-sider them further.
2That an APP0 marker immediately follow the SOI marker is a JFIF requirement. Some applications, such as Adobe Illustrator, can create JPEG files that are not JFIF.
Marker Types 49
Compressed Data
The compressed component data is the only part of a JPEG file that does not occur within a marker. Compressed data always immediately follows an SOS marker. Since it has no length information, in order to find the end of the com-pressed data without expanding it, you have to scan for the next marker (other than RSTn). RSTn markers are the only ones that can occur within the compressed data, and they cannot be placed anywhere else.1
The JPEG compression method is designed so that it will rarely produce the compressed value FF16. When this value is required in the compressed data, it is encoded as the 2-byte sequence FF16 followed by 0016. This makes it easy for applications to scan a JPEG file for specific markers.
JPEG File Format
specified in the JPEG standard. The format of these markers is application spe-cific. The length field after the marker can be used to skip over the marker data.
Except for the APP0 markers used by the JFIF format, an application can ignore the APP markers it does not recognize. If an application needs to store informa-tion beyond the capabilities of JPEG and JFIF, it can create APPn markers to hold this information. An APPn marker may appear anywhere within a JPEG file.
By convention, applications that create APPn markers store their name (zero-terminated) at the start of the marker to prevent conflicts with other applications.
An application that processes APPn markers should check not only the marker identifier but also the application name.
COM
The COM (Comment) marker is used to hold comment strings such as copyright information. Its interpretation is application specific, but a JPEG encoder should assume that the decoder is going to ignore this information. A COM marker, rather than an APPn marker, should be used for plain comment text. It may appear anywhere within a JPEG file.
DHT
The DHT (Define Huffman Table) marker defines (or redefines) Huffman tables, which are identified by a class (AC or DC3) and a number. A single DHT marker can define multiple tables; however, baseline mode is limited to two of each type, and progressive and sequential modes are limited to four. The only restriction on the placement of DHT markers is that if a scan requires a specific table identifier and class, it must have been defined by a DHT marker earlier in a file.
The structure of the DHT marker is shown in Table 5.3. Each Huffman table is 17 bytes of fixed data followed by a variable field of up to 256 additional bytes.
The first fixed byte contains the identifier for the table. The next 16 form an array of unsigned 1-byte integers whose elements give the number of Huffman codes for each possible code length (1-16). The sum of the 16 code lengths is the num-ber of values in the Huffman table. The values are 1 byte each and follow, in order of Huffman code, the length counts. The structure of Huffman tables is described in more detail in Chapter 6.
The number of Huffman tables defined by the DHT marker is determined from the length field. An application needs to maintain a counter that is initial-ized with the value of the length field minus 2. Each time you read a table you subtract its length from the counter. When the counter reaches zero all the tables have been read. No padding is allowed in a DHT marker, so if the counter becomes negative the file is invalid.
3The significance of AC and DC is explained in Chapter 7.
50
Table 5.3 DHT Format
DRI
The DRI (Define Restart Interval) marker specifies the number of MCUs between restart markers within the compressed data. The value of the 2-byte length field following the marker is always 4. There is only one data field in the marker—a 2-byte value that defines the restart interval. An interval of zero means that restart markers are not used. A DRI marker with a nonzero restart interval can be used re-enable restart markers later in the image.
A DRI marker may appear anywhere in the file to define or redefine the restart interval, which remains in effect until the end of the image or until another DRI marker changes it. A DRI marker must appear somewhere in the file for a compressed data segment to include restart markers.
Restart markers assist in error recovery. If the decoder finds corrupt scan data, it can use the restart marker ID and the restart interval to determine where in the image to resume decoding.
The following formula can be used to determine the number of MCUs to skip:
Example Assume that the restart interval is 10. After 80 MCUs have been decoded, the RST7 marker is read from the input stream. If the decoder encounters corrupt data before reading the next restart marker (RST0), it stops decoding and scans the input stream for the next restart marker.
If the next marker is RST3, the decoder skips 40 MCUs
(= 10 × ((8 + 3 - 7)MOD8)) from the last known point (80). The decoder resumes decoding at the 121st MCU in the image.
DQT
The DQT (Define Quantization Table) marker defines (or redefines) the quanti-zation tables used in an image. A DQT marker can define multiple quantiquanti-zation tables (up to 4). The quantization table definition follows the markers length
Field Size 1 byte
16 bytes Variable
Description
The 4 high-order bits specify the table class. A value of 0 means a DC table, a value of 1 means an AC table. The 4 low-order bits specify the table identifier. This value is 0 or 1 for baseline frames and 0, 1, 2, or 3 for progressive and extended frames.
The count of Huffman codes of length 1 to 16. Each count is stored in 1 byte.
The 1-byte symbols sorted by Huffman code. The number of symbols is the sum of the 16 code counts.
MCUs to Skip = Restart Interval × ((8 + Current Marker ID - Last Marker ID)MOD8)
Marker Types 51
52 JPEG File Format
field. The value of the length field is the sum of the sizes of the tables plus 2 (for the length field).
The format of a quantization table definition is shown in Table 5.4. Each table starts with 1 byte that contains information about the table. If the 4 high-order bits of the information byte are zero, the quantization table values are 1 byte each and the entire table definition is 65 bytes long. If the value is 1, the size of each quantization value is 2 bytes and the table definition is 129 bytes long. Two-byte quantization values may be used only with 12-bit sample data.
The 4 low-order bits assign a numeric identifier to the table, which can be 0, 1, 2, or 3. The information byte is followed by 64 quantization values that are stored in JPEG zigzag order (defined in Chapter 7).
DQT markers can appear anywhere in an image file. The one restriction is that if a scan requires a quantization table it must have been defined in a previ-ous DQT marker.
Table 5.4
Quantization Table Definition in a DQT Marker
EOI
The EOI (End of Image) marker marks the end of a JPEG image. An EOI marker must be at the end of a JPEG file and there can only be one EOI marker per file and no other markers may follow the EOI marker. The EOI marker stands alone.
RSTn
The restart markers RST0-RST7 are used to mark blocks of independently encoded compressed scan data. They have no length field or data and may only occur within compressed scan data.
Restart markers can be used to implement error recovery. The interval between them is defined by the DRI (Define Restart Interval) marker. Thus, if the restart interval is zero, then restart markers are not used. Restart markers must occur in the sequence RST0, RST1, ... RST7, RST0, ... in the scan data.
SOI
The SOI (Start of Image) marker marks the start of a JPEG image. It must be at the very beginning of the file and there can only be one per file. The SOI marker stands alone.
Field Size 1 byte
64 or 128 bytes
Description
The 4 low-order bits are the table identifier (0, 1, 2, or 3).
The 4 high-order bits specify the quanization value size (0 = 1 byte, 1 = 2 bytes).
64 1- or 2-byte unsigned quantization values
Marker Types 53
SOFn
The SOFn (Start of Frame) marker defines a frame. Although there are many frame types, all have the same format. The SOF marker consists of a fixed header after the marker length followed by a list of structures that define each compo-nent used by the frame. The structure of the fixed header is shown in Table 5.5, and the structure of a component definition is shown in Table 5.6.
Components are identified by an integer in the range 0 to 255. The JFIF stan-dard is more restrictive and specifies that the components be defined in the order {Y, Cb, Cr} with the identifiers {1, 2, 3} respectively. Unfortunately, some encoders do not follow the standard and assign other identifiers to the compo-nents. The most inclusive way for a decoder to match the colorspace component with the identifier is to go by the order in which the components are defined and to accept whatever identifier- the encoder assigns.
There can be only one SOFn marker per JPEG file and it must precede any SOS markers.
Table 5.5
Fixed Portion of an SOF Marker
Table 5.6 Component-Specific Area of an SOF Marker
SOS
The SOS (Start of Scan) marker marks the beginning of compressed data for a scan in a JPEG stream. Its structure is illustrated in Table 5.7. After the compo-nent count comes a compocompo-nent descriptor for each compocompo-nent (shown in Table 5.8). This is followed by 3 bytes of data used only in progressive mode. The com-pressed scan data immediately follows the marker.
Field Size
Sample precision in bits (can be 8 or 12) Image height in pixels
Image width in pixels
Number of components in the image
Field Size 1 byte 1 byte
1 byte
Description
Component identifier. JPEG allows this to be 0 to 255. JFIF restricts it to 1 (Y), 2 (Cb), or 3 (Cr).
The 4 high-order bits specify the horizontal sampling for the compo-nent. The 4 low-order bits specify the vertical sampling. Either value can be 1, 2, 3, or 4 according to the standard. We do not support val-ues of 3 in our code.
The quantization table identifier for the component. Corresponds to the identifier in a DQT marker. Can be 0, 1, 2, or 3.
54 JPEG File Format
The component descriptors are ordered in the same sequence in which the components are ordered within MCUs in the scan data. While not all of the com-ponents from the SOFn marker must be present, their order in the SOS marker and in the SOFn marker must match. The component identifier in the scan des-criptor must match a component identifier value defined in the SOFn marker. The AC and DC Huffman table identifiers must match those of Huffman tables defined in a previous DHT marker.
The JPEG standard allows 1 to 4 components in a scan, but there are some other restrictions on the number. JFIF limits an image to 3 components. In pro-gressive scans there can be only 1 component if the spectral selection start is not zero. The JPEG standard also places a rather low arbitrary limit of 10 on the num-ber of data units that can be in an MCU, which can limit the numnum-ber of compo-nents in a scan.
In a sequential scan, the spectral selection start must be zero, the spectral selection end 63, and the successive approximation zero. In a progressive scan, if the spectral selection start is zero the spectral selection end must also be zero. Otherwise, the spectral selection end must be greater than or equal to the spectral selection start. These values will be explained in more detail in Chapter 10.
An SOS marker must occur after the SOFn marker in the file. It must be preceded by DHT markers that define all of the Huffman tables used by the scan and DQT markers that define all the quantization tables used by the scan components.
Field Size 1 byte
2 component count bytes 1 byte
1 byte 1 byte
Description Component count
Scan component descriptors (see Table 5.8) Spectral selection start (0-63)
Spectral selection end (0-63)
Successive approximation (two 4-bit fields, each with a value in the range 0-13)
Field Size 1 byte 1 byte
Description
Component identifier
The 4 high-order bits specify the DC Huffman table and the 4 low-order bits specify the AC Huffman table.
JFIF Format 55
JFIF Format
Figure 5.1 Structure of a JFIF File
For all practical purposes a "JPEG file" means "a JPEG file in the JFIF format."
The JFIF standard defines the following:
• A signature for identifying JPEG files
• Colorspace
• Pixel density for device independent display of an image
• Thumbnails
• The relationship of pixels to sampling frequency
The general layout of a JFIF file is shown in Figure 5.1. Because all of the JFIF-specific information is stored within APP0 markers, JFIF files are com-pletely compatible with the JPEG standard. JFIF requires that the SOI that starts the file be immediately followed by an APP0 marker in the format shown in Table 5.9.
JFIF specifies that the YCbCr colorspace be used. The conversion between RGB and YCbCr specified by JFIF was shown in Chapter 1. Color images should have all three components, and their component identifiers should be Y=l, Cb=2, and Cr=3. Grayscale images should have only the Y component.
A JFIF file can be identified by reading the first 11 bytes and ensuring that the first 4 bytes are FF16 D816 FF16 E016 (SOI marker followed by APP0
marker). After skipping 2 bytes the next 5 bytes should be J, F, I, F followed by 0016. A file that starts with this pattern is unlikely to be anything other than a JFIF file.
The JFIF APP0 header may be followed by another APP0 used to embed a thumbnail in a format other than the RGB triple format defined by the header.
The format of the data following the length field for the optional APP0 header is shown in Table 5.10.
If the JFIF extension header has an extension_code of 1016 the extension_data field contains a JPEG encoded image. Colors are encoded according to the JFIF standard, but the compressed data does not contain a JFIF APP0 header. The format for the extension_data when the extension_code is 1116 is shown in Table 5.11; the format for 1216 is shown in Table 5.12.
Your applications should avoid creating APP0 markers other than those required by the JFIF standard. There are fifteen other APPn markers available, and this should be sufficient for any conceivable situation. When applications create APPn markers, it should place the application name at the start of the data area. This allows the marker's creator to be identified.
56 JPEG File Format Extension Type 1116
Field Name
Array of 3-byte RGB color values Indices into the palette for each pixel in the thumbnail
Field Name
The zero-terminated string JFIF
The major ID of the file version. For the current JFIF version (1.2) this value is 1.
The minor ID of the file version. For the current
The minor ID of the file version. For the current