MEASURES OF COMPRESSION PERFORMANCE

As in any other system, the metrics of performance of a data compression algorithm is an important criteria for its selection. The performance measures of data compression algorithms can be looked at from different perspectives depending upon the application requirements, namely, amount of compression

MEASURES OF COMPRESSION PERFORMANCE 97

achieved, objective and subjective quality of the reconstructed data, relative complexity of the algorithm, speed of execution, etc. We explain some of these below.

3.5.1 Compression ratio and bits per sample

The most popular performance measure of a data compression algorithm is the 'compression ratio1. It is defined as the ratio of the number of bits in the original data to the number of bits in the compressed data. Consider a gray scale image of size 256 x 256. If each pixel is represented by a single byte, the image needs 65536 bytes of storage. If the compressed version of the image can be stored in 4096 bytes, the compression ratio achieved by the compression algorithm will be 16:1.

A variation of the compression ratio is ''bits per sample¹. This metric indicates the average number of bits to represent a single sample of the data -for example, bits per pixel -for image coding. In case of an image, each pixel represents a sample. On the other hand, in case of a text file, each sample corresponds to a character in the text. If 65536 pixels of an image are com-pressed to 4096 bytes, we can say that the compression algorithm achieved 0.5 bits per pixel on the average. Hence the bits per sample can be measured by the ratio of the number of bits of a single uncompressed sample to the compression ratio.

3.5.2 Quality metric

The quality or fidelity metric is important for lossy compression algorithms used in video, image, voice, etc., because here the reconstructed data differs from the original one and the human perceptual system is the ultimate judge of the reconstructed quality. For example, if there is no perceivable difference between the reconstructed data and the original version, then the compression algorithm can be claimed to have achieved a very high quality or fidelity. The difference of the reconstructed data from the original one is called the distor-tion, and a lower distortion implies a higher quality of the reconstructed data.

A quality measure can either be very subjective based on human perception, or be objectively defined using mathematical or statistical evaluation.

3.5.2.1 Subjective quality metric: There is no universally accepted mea-sure for the subjective quality metrics. Often the subjective quality metric is defined as the mean observer score (MOS). Sometimes it is also called the mean opinion score. There are different statistical ways to compute MOS. In one of the simplest methods, a statistically significant number of observers are randomly chosen to evaluate the visual quality of the reconstructed im-ages. All the images are compressed and decompressed by the same algorithm.

Each observer assigns a numeric score to each reconstructed image based on

98 MULTIMEDIA DATA COMPRESSION

his or her perception of quality of the image, say, within a range 1-5 to de-scribe the quality of the image, with 5 and 1 being the best and worst quality, respectively. The average of the scores assigned by all the observers is the MOS, and it is considered as a viable subjective metric if all the observers are unbiased and evaluate the images under the same viewing or experimental conditions. There are different variations of this approach to calculate MOS, namely, absolute comparison, paired comparison, blind evaluation, etc.

The techniques of measurement of MOS could well be different for different perceptual data. For example, the methodology to evaluate the subjective quality of a still image could be entirely different from that for video or voice data.

3.5.2.2 Objective quality metric: There is no universally accepted mea-sure for the objective quality of data compression algorithms either. The most widely used objective quality metrics are root-mean-squared error (RMSE), signal-to-noise ratio (SNR), and peak signal-to-noise ratio (PSNR). If / is an M x N image and 7 is the corresponding reconstructed image after compression and decompression, RMSE is calculated as

RMSE =

\

M N

MN ^(3-3)

where i,j refer to the pixel position in the image. The SNR in decibel unit (dB) is expressed as SNR =

20 log

i IUMOC, i \Di=iUy=i |/(*»J')-/(*»

(3.4) In case of an 8-bit image, the corresponding PSNR in dB is computed as

(3.5) where 255 is the maximum possible pixel value in 8 bits.

It should be noted that a lower RMSE (or equivalently, higher SNR or PSNR) does not necessarily always indicate a higher subjective quality. In fact these objective error metrics do not always correlate well with the subjec-tive quality metrics. There are many cases where the PSNR of a reconstructed image can be reasonably high, but the subjective quality is really bad when visualized by human eyes. Hence the choice of the objective or subjective met-rics, to evaluate a compression and decompression algorithm, often depends upon the application criteria.

Similar objective quality metrics are used for audio and speech signals as well.

SOURCE CODING ALGORITHMS 99

3.5.3 Coding complexity

When the computational requirement to implement the CODEC in a particu-lar computing platform is an important criterion, we often consider the coding complexity and computation time of a compression algorithm to be a perfor-mance measure. Implementation of a compression algorithm using special purpose digital signal processor (DSP) architectures is common in communi-cation systems. In portable systems, the coding complexity is an important criterion from the perspective of low-power hardware implementation. The computational requirement is usually measured in terms of the number of arithmetic operations and the memory requirement. Usually, the number of arithmetic operations is described by MOPS (millions of operations per sec-ond). But in compression literature, the term MIPS (millions of instructions per second) is often used to measure the compression performance of a specific computing engine's architecture.