FIGURE 4.5: Transcoding architectures for bit-rate reduction [72]:
4.4.5 Overview of Optimization Techniques for Bandwidth Adaptation
Recall that bandwidth adaptation requires (i) observing the state of the network, (ii) estimating or observing the state of the decoder, and then (iii) based on band-width availability and decoder state, deciding what information should be sent next to the decoder. In this section we discuss briefly this decision process. Our focus here is in highlighting the challenges involved and how these have to be addressed by proposed techniques.
Ideally the goal in deciding what information is sent to the decoder should be to maximize the expected quality at the decoder. Note that we consider expected quality because there is uncertainty about the actual quality available at the de-coder; changes in bandwidth, packet losses, and so forth will affect the resulting quality.
To facilitate the discussion, in what follows we assume that information avail-able for transmission has already been packetized. The role of the decision mecha-nisms under consideration is essentially to prioritize the transmission so that most
“important” information is sent first.
Optimization of expected quality at the decoder is complex because of multiple factors:
• The expected distortion is hard to estimate.
• The candidate packets may depend on each other.
• At any given time there are many candidate packets.
Estimating the expected distortion at the transmitter requires first determining both the current “state” of the transmission channel and its expected behavior in the near future. Various types of channel models are considered in Chapters 7 and 11. The type of channel models available, for example, with memory [29] or without it [16,48], depends on the systems being considered. Observations may include packet receipt feedback, received power measurements, etc. While the ac-curacy of the models may be questionable, it is also likely that even an inaccurate model will provide enough information to improve on a system that makes no assumptions about the transmission channel.
In addition, estimations of expected distortion are based on the reconstruction quality achievable when different sets of packets are received. In cases where pre-encoded data is being transmitted it is possible, in theory, to quantify ex-actly achievable distortion in each scenario. In practice, however, techniques that require less computation and provide estimates of expected distortion may be preferable. For example, some methods may attach some importance to each packet, where the importance is based on some simplifications about the decoding process (e.g., frames that depend on frames received in error are not decoded, no error concealment is applied); see, for example, [16,48]. Then optimization tech-niques would seek to maximize the expected “importance” of packets received.
Most widely used video coding techniques make use of prediction across frames. This complicates distortion estimation, since a packet loss may affect multiple future frames. A very powerful technique used to capture the dependen-cies is that formalized by Chou and Miao [16], which leads to the creation of a directed acyclic graph to represent all the packets being transmitted. With this type of technique it is possible to attach more importance to packets from which multiple other packets depend. As we had indicated earlier for the channel model, even a rough model of these dependencies (which may not provide exact dis-tortion values) is likely to provide better results than techniques that completely ignore the existence of these dependencies.
Optimization complexity should definitely be of concern. As has been demon-strated by various authors (see [9–11,14,16,17,32,48,49,73,80,81]) efficient tech-niques can be developed once knowledge of the structure of the media stream (including dependencies) and an estimate of the channel state are available. This can be done by estimating the expected distortions if several different candidate packets (not necessarily all available ones) were transmitted. This distortion can be estimated for one decision (the next packet to be transmitted) or more than one.
After this evaluation, the packet leading to a lower expected distortion is chosen, and this decision process is repeated for the next packet.
4.5 SUMMARY AND FURTHER READING
The heterogeneous and time-variant nature of today’s networks imposes a num-ber of challenges for real-time video communication. In this chapter, we have discussed alternative techniques for bandwidth adaptation and their relative mer-its. The main points made in this chapter are summarized as follows.
• We classify bandwidth adaptation architectures based on three basic de-sign decisions, namely selection of adaptation points, decision agents, and source coding techniques. Bandwidth adaptation is made based on available source and channel information. The source-related information is known more accurately at the sender, while channel information is more accurate at the client. A proxy, located in the middle of the network, can achieve a good compromise between server and client adaptation.
• When the sender acts as the adaptation point, the highest degree of flex-ibility is possible in terms of source coding, which facilitates achieving finer granularity rate adaptation, reducing the quality penalty at the receiver.
However, this may lead to a longer reaction time if network state informa-tion is provided by the receiver. Adaptainforma-tion decisions may be inefficient if, instead, the sender itself has to estimate the state of the network without waiting for receiver feedback. Adaptation at the sender makes scaling to a large number of receivers more difficult, as it increases the computation load at the sender. Adaptation at the client can reduce decoding complexity, but will have no impact on the network traffic.
• If the sender is the decision agent, it will have access to more accurate source information, but may not have reliable or timely information about the network state near the receiver. This approach helps improve overall bandwidth utilization when multiple receivers are served by the sender. In contrast, if the client acts as the decision agent, there is potential for better adaptation decisions given the higher accuracy network and packet arrival information. However, when decisions made by the receiver have to be put in place by the sender, the latency involved can lead to lower adaptation efficiency.
• Rate control techniques are used during the encoding process to adjust cod-ing parameters to meet a target encodcod-ing rate. Transcodcod-ing techniques, of-ten used at either the server or the proxy, take a compressed media stream as an input and convert it to another compressed stream. Scalable coding provides flexible bandwidth adaptation over a given bit rate range rather than at a fixed bit rate. Different from the aforementioned techniques, bit
stream switching techniques encode the same media content into multiple versions at different bit rates and dynamically switch among them to ac-commodate the bandwidth variations. In this chapter we have discussed several switching techniques: multiple bit rate coding, SP/SI pictures, and stream morphing. The trade-off between coding efficiency (to reduce over-head) and switching flexibility is a main consideration on the design of various switching techniques.
Further details on many of the bandwidth adaptation techniques described in this chapter can be found in other literature, as well as in other chapters in this book. For example, Ortega and Ramchandran [53] and Sullivan and Wiegand [65]
discuss rate–distortion optimization for image and video compression; Vetro et al. [72] and Xin et al. [78] provide overviews of transcoding; and Goyal [25]
and Wang et al. [74] review state-of-the-art multiple description coding. For more details on rate–distortion-optimized streaming, the article by Chou and Miao [16]
can serve as a starting point. Although this chapter focused on the fundamentals of bandwidth adaptation on a simple client–server system, there is considerable interest in more complex systems with multiple paths used for media transport, such as content delivery networks and P2P networks. The interested reader is referred to the work of Apostolopoulos et al. [4], Padmanabhan et al. [54], and Rejaie and Ortega [57].
REFERENCES
[1] ISO/IEC 13818-2. Generic coding of moving pictures and associated audio, part-2 video. November 1994.
[2] ISO/IEC 14496-10 and ITU-T Rec. H.264. Advanced video coding. 2003.
[3] ISO/IEC 14496-2/FPDAM4. Coding of audio-visual objects, part-2 visual, amend-ment 4: streaming video profile. July 2000.
[4] J. Apostolopoulos, T. Wong, W.-T. Tan, and S. Wee. On multiple description stream-ing with content delivery networks. In Proc. Conf. Computer Communications (INFOCOM), June 2002.
[5] J. F. Arnold, M. R. Frater, and Y. Wang. Efficient drift-free signal-to-noise ratio scala-bility. IEEE Trans. Circuits and Systems for Video Technology, 10(1):70–82, February 2000.
[6] B. Birney. Intelligent streaming. http://www.microsoft.com/windows/windowsmedia/
howto/articles/intstreaming.aspx, May 2003.
[7] P. Assunção and M. Ghanbari. Post-processing of MPEG2 coded video for transmis-sion at lower bit rates. In Proc. Int’l Conf. Acoustics, Speech, and Signal Processing, volume 3, pages 1998–2001, May 1996.
[8] P. Assunção and M. Ghanbari. A frequency-domain video transcoder for dynamic bit-rate reduction of MPEG-2 bit streams. IEEE Trans. Circuits and Systems for Video Technology, 8(8):953–967, December 1998.
[9] J. Chakareski, J. Apostolopoulos, S. Wee, W. Tan, and B. Girod. Rate-distortion hint tracks for adaptive video streaming. IEEE Trans. Circuits and Systems for Video Tech-nology, 15(10):1257–1269, October 2005.
[10] J. Chakareski, P. A. Chou, and B. Girod. Rate-distortion optimized streaming from the edge of the network. In Proc. Workshop on Multimedia Signal Processing, De-cember 2002.
[11] J. Chakareski and B. Girod. Rate-distortion optimized packet scheduling and rout-ing for media streamrout-ing with path diversity. In Proc. Data Compression Conference, March 2003.
[12] S.-F. Chang and D. G. Messerschmitt. Manipulation and compositing of MC-DCT compressed video. IEEE J. Selected Areas in Communications, 13(1):1–11, January 1995.
[13] M. C. Chen and A. N. Willson. Rate-distortion optimal motion estimation algorithms for motion-compensated transform video coding. IEEE Trans. Circuits and Systems for Video Technology, 8(2):147–158, April 1998.
[14] G. Cheung and W. Tan. Directed acyclic graph based source modeling for data unit selection of streaming media over QoS networks. In Proc. Int’l Conf. Multimedia and Exhibition, August 2002.
[15] P. A. Chou, A. E. Mohr, A. Wang, and S. Mehrotra. Error control for receiver-driven layered multicast of audio and video. IEEE Trans. Multimedia, 3(1):108–122, March 2001.
[16] P. A. Chou and Z. Miao. Rate-distortion optimized streaming of packetized media.
IEEE Trans. Multimedia, 8(2):390–404, April 2006.
[17] P. A. Chou and A. Sehgal. Rate-distortion optimized receiver-driven streaming over best-effort networks. In Proc. Int’l Packet Video Workshop, volume 1, April 2002.
[18] P. A. Chou, H. J. Wang, and V. N. Padmanabhan. Layered multiple description cod-ing. In Proc. Int’l Packet Video Workshop, volume 1, April 2003.
[19] G. J. Conklin, G. S. Greenbaum, K. O. Lillevold, A. F. Lippman, and Y. A. Reznik.
Video coding for streaming media delivery on the internet. IEEE Trans. Circuits and Systems for Video Technology, 11(3):269–281, March 2001.
[20] W. Ding and B. Liu. Rate control of MPEG video coding and recording by rate-quantization modeling. IEEE Trans. Circuits and Systems for Video Technology, 6(1):12–20, February 1996.
[21] M. Domanski, A. Luczak, and S. Mackowiak. Spatio-temporal scalability for MPEG video coding. IEEE Trans. Circuits and Systems for Video Technology, 10(7):1088–
1093, October 2000.
[22] A. Eleftheriadis and D. Anastassiou. Constrained and general dynamic rate shaping of compressed digital video. In Proc. Int’l Conf. Image Processing, volume 3, pages 396–399, October 1995.
[23] N. Farber and B. Girod. Robust H.263 compatible video transmission for mobile access to video servers. In Proc. Int’l Conf. Image Processing, volume 2, pages 73–
76, October 1997.
[24] B. Girod. Rate-constrained motion estimation. In Proc. Visual Communications and Image Processing, pages 1026–1034, September 1994.
[25] V. K. Goyal. Multiple description coding: Compression meets the network. IEEE Signal Processing Magazine, 18(5):74–93, September 2001.
[26] Z. Guo, O. C. Au, and K. B. Letaief. Parameter estimation for image/video transcod-ing. In Proc. Int’l Symp. Circuits and Systems, volume 2, pages 269–272, May 2000.
[27] H.-M. Hang and J.-J. Chen. Source model for transform video coder and its appli-cation. IEEE Trans. Circuits and Systems for Video Technology, 7(2):287–311, April 1997.
[28] Z. He and S. K. Mitra. From rate–distortion analysis to resource-distortion analysis.
IEEE Circuits and Systems Magazine, 5(3):6–18, 2005.
[29] C.-Y. Hsu, A. Ortega, and M. Khansari. Rate control for robust video transmis-sion over burst-error wireless channels. IEEE J. Selected Areas in Communications, 17(5):1–18, May 1999.
[30] C.-Y. Hsu, A. Ortega, and A. Reibman. Joint selection of source and channel rate for VBR video transmission under ATM policing constraints. IEEE J. Selected Areas in Communications, 15(6):1016–1028, August 1997.
[31] H.-C. Huang, C.-N. Wang, and T. Chiang. A robust fine granularity scalability using trellis-based predictive leak. IEEE Trans. Circuits and Systems for Video Technology, 12(6):372–385, June 2002.
[32] M. Kalman and B. Girod. Rate-distortion optimized streaming of video with multiple independent encodings. In Proc. Int’l Conf. Image Processing, volume 1, October 2004.
[33] M. Karczewicz and R. Kurceren. The SP- and SI-frames design for H.264/AVC. IEEE Trans. Circuits and Systems for Video Technology, 13(7):637–644, July 2003.
[34] S. A. Karunasekera and N. G. Kingsbury. A distortion measure for image artifacts based on human visual sensitivity. In Proc. Int’l Conf. Image Processing, April 1994.
[35] K. Lengwehasatit and A. Ortega. Probabilistic partial distance fast matching for mo-tion estimamo-tion. IEEE Trans. Circuits and Systems for Video Technology, 11(2):139–
152, February 2001.
[36] K. Lengwehasatit and A. Ortega. Scalable variable complexity approximate forward DCT. IEEE Trans. Circuits and Systems for Video Technology, 14(11):1236–1248, November 2004.
[37] A. Leontaris and A. R. Reibman. Comparison of blocking and blurring metrics for video compression. In Proc. Int’l Conf. Acoustics, Speech, and Signal Processing, March 2005.
[38] W. Li. Overview of fine granularity scalalability in MPEG-4 video standard. IEEE Trans. Circuits and Systems for Video Technology, 11(3):301–317, March 2001.
[39] Y. J. Liang, N. Farber, and B. Girod. Adaptive playout scheduling and loss conceal-ment for voice communication over IP networks. IEEE Transactions on Multimedia, 5(4), December 2003.
[40] Yi J. Liang and B. Girod. Prescient R-D optimized packet dependency management for low-latency video streaming. In Proc. Int’l Conf. Image Processing, September 2003.
[41] Yi J. Liang, E. Steinbach, and B. Girod. Multi-stream voice transmission over the internet using path diversity. In Proc. ACM Multimedia, September 2001.
[42] C.-W. Lin and Y.-R. Lee. Fast algorithms for DCT-domain video transcoding. In Proc. Int’l Conf. Image Processing, volume 1, pages 421–424, October 2001.
[43] L.-J. Lin and A. Ortega. Bit-rate control using piecewise approximated rate–
distortion characteristics. IEEE Trans. Circuits and Systems for Video Technology, 8(4):446–459, August 1998.
[44] J. Macnicol, J. Arnold, and M. Frater. Scalable video coding by stream morphing.
IEEE Trans. Circuits and Systems for Video Technology, 15(2):306–319, February 2005.
[45] N. Magharei and R. Rejaie. Adaptive receiver-driven streaming from multi-ple senders. Proceedings of ACM Multimedia Systems Journal, Springer-Verlag, 11(6):1–18, April 2006.
[46] S. McCanne, V. Jacobson, and M. Vetterli. Receiver-driven layered multicast. In ACM SIGCOMM, August 1996.
[47] N. Merhav. Multiplication-free approximate algorithms for compressed-domain lin-ear operations on images. IEEE Trans. Image Processing, 8(2):247–254, February 1999.
[48] Z. Miao and A. Ortega. Expected run-time distortion based scheduling for delivery of scalable media. In Proc. Int’l Packet Video Workshop, volume 1, April 2002.
[49] Z. Miao and A. Ortega. Fast adaptive media scheduling based on expected run-time distortion. In Proc. Asilomar Conf. Signals, Systems, and Computers, volume 1, No-vember 2002.
[50] Z. Miao and A. Ortega. Scalable proxy caching of video under storage constraints.
IEEE J. Selected Areas in Communications, 20(7):1315–1327, September 2002.
[51] Y. Nakajima, H. Hori, and T. Kanoh. Rate conversion of MPEG coded video by re-quantization process. In Proc. Int’l Conf. Image Processing, volume 3, pages 408–
411, October 1995.
[52] T. Nguyen and A. Zakhor. Multiple sender distributed video streaming. IEEE Trans.
Multimedia, 6(2):315–326, April 2004.
[53] A. Ortega and K. Ramchandran. Rate-distortion methods for image and video com-pression. IEEE Signal Processing Magazine, 15(6):23–50, November 1998.
[54] V. N. Padmanabhan, H. J. Wang, and P. A. Chou. Resilient peer-to-peer streaming. In Proc. Int’l Conf. Network Protocols, November 2003.
[55] W. Pan and A. Ortega. Complexity-scalable transform coding using variable com-plexity algorithms. In Proc. Data Compression Conference, pages 263–272, March 2000.
[56] A. R. Reibman and M. T. Sun, editors. Compressed Video over Networks. “Variable bit rate video coding.” Marcel Dekker, New York (NY), 2001.
[57] R. Rejaie and A. Ortega. Pals: Peer-to-peer adaptive layered streaming. In Proc. Int’l Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV), June 2003.
[58] K. Rose and S. Regunathan. Toward optimality in scalable predictive coding. IEEE Transactions on Image Processing, 10:965–976, July 2001.
[59] B. Shen, S.-J. Lee, and S. Basu. Caching strategies in transcoding-enabled proxy sys-tems for streaming media distribution networks. IEEE Trans. Multimedia, 6(2):375–
386, April 2004.
[60] R. Singh and A. Ortega. Erasure recovery in predictive coding environments using multiple description coding. In IEEE Workshop on Multimedia Signal Processing, 1999.
[61] I. Sodagar, H.-J. Lee, P. Hatrack, and Y.-Q. Zhang. Scalable wavelet coding for syn-thetic/natural hybrid images. IEEE Trans. Circuits and Systems for Video Technology, 9(2):244–254, March 1999.
[62] J. Song and B.-L. Yeo. A fast algorithm for DCT-domain inverse motion compensa-tion based on shared informacompensa-tion in a macroblock. IEEE Trans. Circuits and Systems for Video Technology, 10(5):767–775, August 2000.
[63] H. Sorial, W. E. Lynch, and A. Vincent. Selective requantization for transcoding of MPEG compressed video. In Proc. Int’l Conf. Multimedia and Exhibition, volume 1, pages 217–220, August 2000.
[64] N. Srinivasamurthy, A. Ortega, and S. Narayanan. Efficient scalable encoding for distributed speech recognition. Speech Communication, 48:888–902, 2006.
[65] G. J. Sullivan and T. Wiegand. Rate-distortion optimization for video compression.
IEEE Signal Processing Magazine, 15(6):74–90, November 1998.
[66] H. Sun, W. Kwok, and J. W. Zdepski. Architectures for MPEG compressed bitstream scaling. IEEE Trans. Circuits and Systems for Video Technology, 6(2):191–199, April 1996.
[67] X. Sun, F. Wu, S. Li, W. Gao, and Y.-Q. Zhang. Seamless switching of scalable video bitstreams for efficient streaming. IEEE Trans. Multimedia, 6(2):291–303, April 2004.
[68] K. T. Tan and M. Ghanbari. A multi-metric objective picture-quality measurement model for MPEG video. IEEE Trans. Circuits and Systems for Video Technology, 10(7):1208–1213, October 2000.
[69] M. van der Schaar and Y. Andreopoulos. Rate-distortion-complexity modeling for network and receiver aware adaptation. IEEE Trans. Multimedia, 7(3):471–479, June 2005.
[70] M. van der Schaar and P. H. N. de With. Near-lossless complexity-scalable embedded compression algorithm for cost reduction in DTV receivers. IEEE Trans. on Con-sumer Electronics, 46(4):923–933, November 2000.
[71] M. van der Schaar and H. Radha. Adaptive motion-compensation fine-granular-scalability (AMC-FGS) for wireless video. IEEE Trans. Circuits and Systems for Video Technology, 12(6):360–371, June 2002.
[72] A. Vetro, C. Christopoulos, and H. Sun. Video transcoding achitectures and tech-niques: an overview. IEEE Signal Processing Magazine, 20(2):18–29, March 2003.
[73] H. Wang and A. Ortega. Robust video communication by combining scalability and multiple description coding techniques. In Proc. Symp. Electronic Imaging, volume 1, January 2003.
[74] Y. Wang, A. R. Reibman, and S. Lin. Multiple description coding for video delivery.
Proceedings of the IEEE, 93(1):57–70, January 2005.
[75] O. Werner. Requantization for transcoding of MPEG-2 intraframes. IEEE Trans. Im-age Processing, 8(2):179–191, February 1999.
[76] T. Wiegand, M. Lightstone, D. Mukherjee, T. G. Campbell, and S. K. Mitra. Rate-distortion optimized mode selection for very low bit rate video coding and the emerging H.263 standard. IEEE Trans. Circuits and Systems for Video Technology, 6(2):182–190, April 1996.
[77] F. Wu, S. Li, and Y.-Q. Zhang. A framework for efficient progressive fine granular-ity scalable video coding. IEEE Trans. Circuits and Systems for Video Technology, 11(3):332–344, March 2001.
[78] J. Xin, C.-W. Lin, and M.-T. Sun. Digital video transcoding. Proceedings of the IEEE, 93(1):84–97, January 2005.
[79] J. Youn, J. Xin, and M.-T. Sun. Fast video transcoding architectures for networked multimedia applications. In Proc. Int’l Symp. Circuits and Systems, volume 4, pages 25–28, May 2000.
[80] R. Zhang, S. Regunathan, and K. Rose. Optimized video streaming over lossy net-works with real-time estimation of end-to-end distortion. In Proc. Int’l Conf. Multi-media and Exhibition, volume 1, August 2002.
[81] R. Zhang, S. L. Regunathan, and K. Rose. Video coding with optimal inter/intra-mode switching for packet loss resilience. IEEE J. Selected Areas in Communica-tions, 18(6):966–976, June 2000.