Algorithms and Engineering Applications
Algorithms and Engineering Applications
Andreas Antoniou Wu-Sheng Lu
Department of Electrical and Computer Engineering University of Victoria, Canada
Spri inger
University of V ictoria University of V ictoria British Columbia British Columbia
Canada Canada [email protected] [email protected],ca
Library of Congress Control Number: 2007922511
Practical Optimization: Algorithms and Engineering Applications by Andreas Antoniou and Wu-Sheng Lu
ISBN-10: 0-387-71106-6 e-ISBN-10: 0-387-71107-4 ISBN-13: 978-0-387-71106-5 e-ISBN-13: 978-0-387-71107-2
Printed on acid-free paper.
© 2007 Springer Science+Business Media, LLC
All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.
9 8 7 6 5 4 3 2 1 springer.com
Lynne and Chi'Tang Catherine
with our love
Andreas Antoniou received the Ph.D. degree in Electrical Engineering from the University of London, UK, in 1966 and is a Fellow of the lET and IEEE.
He served as the founding Chair of the Department of Electrical and Computer Engineering at the University of Victoria, B.C., Canada, and is now Professor Emeritus in the same department. He is the author of Digital Filters: Analysis, Design, and Applications (McGraw-Hill, 1993) and Digital Signal Processing:
Signals, Systems, and Filters (McGraw-Hill, 2005). He served as Associate Editor/Editor of IEEE Transactions on Circuits and Systems from June 1983 to May 1987, as a Distinguished Lecturer of the IEEE Signal Processing Society in 2003, as General Chair of the 2004 International Symposium on Circuits and Systems, and is currently serving as a Distinguished Lecturer of the IEEE Circuits and Systems Society. He received the Ambrose Fleming Premium for 1964 from the lEE (best paper award), the CAS Golden Jubilee Medal from the IEEE Circuits and Systems Society, the B.C. Science Council Chairman's Award for Career Achievement for 2000, the Doctor Honoris Causa degree from the Metsovio National Technical University of Athens, Greece, in 2002, and the IEEE Circuits and Systems Society 2005 Technical Achievement Award.
Wu-Sheng Lu received the B.S. degree in Mathematics from Fudan University, Shanghai, China, in 1964, the M.E. degree in Automation from the East China Normal University, Shanghai, in 1981, the M.S. degree in Electrical Engineer- ing and the Ph.D. degree in Control Science from the University of Minnesota, Minneapolis, in 1983 and 1984, respectively. He was a post-doctoral fellow at the University of Victoria, Victoria, BC, Canada, in 1985 and Visiting Assistant Professor with the University of Minnesota in 1986. Since 1987, he has been with the University of Victoria where he is Professor. His current teaching and research interests are in the general areas of digital signal processing and application of optimization methods. He is the co-author with A. Antoniou of Two-Dimensional Digital Filters (Marcel Dekker, 1992). He served as an As- sociate Editor of the Canadian Journal of Electrical and Computer Engineering in 1989, and Editor of the same journal from 1990 to 1992. He served as an Associate Editor for the IEEE Transactions on Circuits and Systems, Part II, from 1993 to 1995 and for Part I of the same journal from 1999 to 2001 and from 2004 to 2005. Presently he is serving as Associate Editor for the Inter- national Journal of Multidimensional Systems and Signal Processing. He is a Fellow of the Engineering Institute of Canada and the Institute of Electrical and Electronics Engineers.
Biographies of the authors vii
Preface xv Abbreviations xix
1. THE OPTIMIZATION PROBLEM 1
1.1 Introduction 1 1.2 The Basic Optimization Problem 4
1.3 General Structure of Optimization Algorithms 8
1.4 Constraints 10 1.5 The Feasible Region 17
1.6 Branches of Mathematical Programming 22
References 24 Problems 25 2. BASIC PRINCIPLES 27
2.1 Introduction 27 2.2 Gradient Information 27
2.3 The Taylor Series 28 2.4 Types of Extrema 31 2.5 Necessary and Sufficient Conditions for
Local Minima and Maxima 33 2.6 Classification of Stationary Points 40 2.7 Convex and Concave Functions 51 2.8 Optimization of Convex Functions 58
References 60 Problems 60 3. GENERAL PROPERTIES OF ALGORITHMS 65
3.1 Introduction 65 3.2 An Algorithm as a Point-to-Point Mapping 65
3.3 An Algorithm as a Point-to-Set Mapping 67
3.4 Closed Algorithms 68 3.5 Descent Functions 71 3.6 Global Convergence 72
References 79 Problems 79 4. ONE-DIMENSIONAL OPTIMIZATION 81
4.1 Introduction 81 4.2 Dichotomous Search 82
4.3 Fibonacci Search 85 4.4 Golden-Section Search 92 4.5 Quadratic Interpolation Method 95
4.6 Cubic Interpolation 99 4.7 The Algorithm of Davies, Swann, and Campey 101
4.8 Inexact Line Searches 106
References 114 Problems 114 5. BASIC MULTIDIMENSIONAL GRADIENT METHODS 119
5.1 Introduction 119 5.2 Steepest-Descent Method 120
5.3 Newton Method 128 5.4 Gauss-Newton Method 138
References 140 Problems 140 6. CONJUGATE-DIRECTION METHODS 145
6.1 Introduction 145 6.2 Conjugate Directions 146
6.3 Basic Conjugate-Directions Method 149
6.4 Conjugate-Gradient Method 152 6.5 Minimization of Nonquadratic Functions 157
6.6 Fletcher-Reeves Method 158 6.7 Powell's Method 159 6.8 Partan Method 168
References 172
Problems 172 7. QUASI-NEWTON METHODS 175
7.1 Introduction 175 7.2 The Basic Quasi-Newton Approach 176
7.3 Generation of Matrix Sk 177 7.4 Rank-One Method 181 7.5 Davidon-Fletcher-Powell Method 185
7.6 Broyden-Fletcher-Goldfarb-Shanno Method 191
7.7 Hoshino Method 192 7.8 The Broyden Family 192 7.9 The Huang Family 194 7.10 Practical Quasi-Newton Algorithm 195
References 199 Problems 200 8. MINIMAX METHODS 203
8.1 Introduction 203 8.2 Problem Formulation 203
8.3 Minimax Algorithms 205 8.4 Improved Minimax Algorithms 211
References 228 Problems 228 9. APPLICATIONS OF UNCONSTRAINED OPTIMIZATION 231
9.1 Introduction 231 9.2 Point-Pattern Matching 232
9.3 Inverse Kinematics for Robotic Manipulators 237
9.4 Design of Digital Filters 247
References 260 Problems 262 10. FUNDAMENTALS OF CONSTRAINED OPTIMIZATION 265
10.1 Introduction 265 10.2 Constraints 266
10.3 Classification of Constrained Optimization Problems 273
10.4 Simple Transformation Methods 277
10.5 Lagrange Multipliers 285 10.6 First-Order Necessary Conditions 294
10.7 Second-Order Conditions 302
10.8 Convexity 308 10.9 Duality 311
References 312 Problems 313 11. LINEAR PROGRAMMING PART I: THE SIMPLEX METHOD 321
11.1 Introduction 321 11.2 General Properties 322 11.3 Simplex Method 344
References 368 Problems 368 12. LINEAR PROGRAMMING PART II:
INTERIOR-POINT METHODS 373
12.1 Introduction 373 12.2 Primal-Dual Solutions and Central Path 374
12.3 Primal Affine-Scaling Method 379 12.4 Primal Newton Barrier Method 383 12.5 Primal-Dual Interior-Point Methods 388
References 402 Problems 402 13. QUADRATIC AND CONVEX PROGRAMMING 407
13.1 Introduction 407 13.2 Convex QP Problems with Equality Constraints 408
13.3 Active-Set Methods for Strictly Convex QP Problems 411 13.4 Interior-Point Methods for Convex QP Problems 417
13.5 Cutting-Plane Methods for CP Problems 428
13.6 Ellipsoid Methods 437 References 443
Problems 444 14. SEMIDEFINITE AND SECOND-ORDER CONE
PROGRAMMING 449 14.1 Introduction 449 14.2 Primal and Dual SDP Problems 450
14.3 Basic Properties of SDP Problems 455 14.4 Primal-Dual Path-Following Method 458
14.5 Predictor-Corrector Method 465 14.6 Projective Method of Nemirovski and Gahinet 470
14.7 Second-Order Cone Programming 484 14.8 A Primal-Dual Method for SOCP Problems 491
References 496 Problems 497 15. GENERAL NONLINEAR OPTIMIZATION PROBLEMS 501
15.1 Introduction 501 15.2 Sequential Quadratic Programming Methods 501
15.3 Modified SQP Algorithms 509 15.4 Interior-Point Methods 518
References 528 Problems 529 16. APPLICATIONS OF CONSTRAINED OPTIMIZATION 533
16.1 Introduction 533 16.2 Design of Digital Filters 534
16.3 Model Predictive Control of Dynamic Systems 547 16.4 Optimal Force Distribution for Robotic Systems with Closed
Kinematic Loops 558 16.5 Multiuser Detection in Wireless Communication Channels 570
References 586 Problems 588 Appendices 591 A Basics of Linear Algebra 591
A. 1 Introduction 591
A.2 Linear Independence and Basis of a Span 592
A.3 Range, Null Space, and Rank 593 A.4 Sherman-Morrison Formula 595 A.5 Eigenvalues and Eigenvectors 596
A.6 Symmetric Matrices 598
A.7 Trace 602 A.8 Vector Norms and Matrix Norms 602
A.9 Singular-Value Decomposition 606
A. 10 Orthogonal Projections 609 A.l 1 Householder Transformations and Givens Rotations 610
A. 12 QR Decomposition 616 A. 13 Cholesky Decomposition 619 A. 14 Kronecker Product 621 A. 15 Vector Spaces of Symmetric Matrices 623
A. 16 Polygon, Polyhedron, Polytope, and Convex Hull 626
References 627 B Basics of Digital Filters 629
B.l Introduction 629 B.2 Characterization 629 B. 3 Time-Domain Response 631 B.4 Stability Property 632 B.5 Transfer Function 633 B.6 Time-Domain Response Using the Z Transform 635
B.7 Z-Domain Condition for Stability 635 B.8 Frequency, Amplitude, and Phase Responses 636
B.9 Design 639 Reference 644
Index 645
The rapid advancements in the efficiency of digital computers and the evo- lution of reliable software for numerical computation during the past three decades have led to an astonishing growth in the theory, methods, and algo- rithms of numerical optimization. This body of knowledge has, in turn, mo- tivated widespread applications of optimization methods in many disciplines, e.g., engineering, business, and science, and led to problem solutions that were considered intractable not too long ago.
Although excellent books are available that treat the subject of optimization with great mathematical rigor and precision, there appears to be a need for a book that provides a practical treatment of the subject aimed at a broader au- dience ranging from college students to scientists and industry professionals.
This book has been written to address this need. It treats unconstrained and constrained optimization in a unified manner and places special attention on the algorithmic aspects of optimization to enable readers to apply the various algo- rithms and methods to specific problems of interest. To facilitate this process, the book provides many solved examples that illustrate the principles involved, and includes, in addition, two chapters that deal exclusively with applications of unconstrained and constrained optimization methods to problems in the areas of pattern recognition, control systems, robotics, communication systems, and the design of digital filters. For each application, enough background information is provided to promote the understanding of the optimization algorithms used to obtain the desired solutions.
Chapter 1 gives a brief introduction to optimization and the general structure of optimization algorithms. Chapters 2 to 9 are concerned with unconstrained optimization methods. The basic principles of interest are introduced in Chap- ter 2. These include the first-order and second-order necessary conditions for a point to be a local minimizer, the second-order sufficient conditions, and the optimization of convex functions. Chapter 3 deals with general properties of algorithms such as the concepts of descent function, global convergence, and
rate of convergence. Chapter 4 presents several methods for one-dimensional optimization, which are commonly referred to as line searches. The chapter also deals with inexact line-search methods that have been found to increase the efficiency in many optimization algorithms. Chapter 5 presents several basic gradient methods that include the steepest descent, Newton, and Gauss- Newton methods. Chapter 6 presents a class of methods based on the concept of conjugate directions such as the conjugate-gradient, Fletcher-Reeves, Powell, and Partan methods. An important class of unconstrained optimization meth- ods known as quasi-Newton methods is presented in Chapter 7. Representa- tive methods of this class such as the Davidon-Fletcher-Powell and Broydon- Fletcher-Goldfarb-Shanno methods and their properties are investigated. The chapter also includes a practical, efficient, and reliable quasi-Newton algorithm that eliminates some problems associated with the basic quasi-Newton method.
Chapter 8 presents minimax methods that are used in many applications in- cluding the design of digital filters. Chapter 9 presents three case studies in which several of the unconstrained optimization methods described in Chap- ters 4 to 8 are applied to point pattern matching, inverse kinematics for robotic manipulators, and the design of digital filters.
Chapters 10 to 16 are concerned with constrained optimization methods.
Chapter 10 introduces the fundamentals of constrained optimization. The con- cept of Lagrange multipliers, the first-order necessary conditions known as Karush-Kuhn-Tucker conditions, and the duality principle of convex program- ming are addressed in detail and are illustrated by many examples. Chapters 11 and 12 are concerned with linear programming (LP) problems. The gen- eral properties of LP and the simplex method for standard LP problems are addressed in Chapter 11. Several interior-point methods including the primal affine-scaling, primal Newton-barrier, and primal dual-path following meth- ods are presented in Chapter 12. Chapter 13 deals with quadratic and general convex programming. The so-called active-set methods and several interior- point methods for convex quadratic programming are investigated. The chapter also includes the so-called cutting plane and ellipsoid algorithms for general convex programming problems. Chapter 14 presents two special classes of con- vex programming known as semidefinite and second-order cone programming, which have found interesting applications in a variety of disciplines. Chapter 15 treats general constrained optimization problems that do not belong to the class of convex programming; special emphasis is placed on several sequential quadratic programming methods that are enhanced through the use of efficient line searches and approximations of the Hessian matrix involved. Chapter 16, which concludes the book, examines several applications of constrained opti- mization for the design of digital filters, for the control of dynamic systems, for evaluating the force distribution in robotic systems, and in multiuser detection for wireless communication systems.
The book also includes two appendices, A and B, which provide additional support material. Appendix A deals in some detail with the relevant parts of linear algebra to consolidate the understanding of the underlying mathematical principles involved whereas Appendix B provides a concise treatment of the basics of digital filters to enhance the understanding of the design algorithms included in Chaps. 8, 9, and 16.
The book can be used as a text for a sequence of two one-semester courses on optimization. The first course comprising Chaps. 1 to 7, 9, and part of Chap. 10 may be offered to senior undergraduate or first-year graduate students.
The prerequisite knowledge is an undergraduate mathematics background of calculus and linear algebra. The material in Chaps. 8 and 10 to 16 may be used as a text for an advanced graduate course on minimax and constrained optimization. The prerequisite knowledge for thi^ course is the contents of the first optimization course.
The book is supported by online solutions of the end-of-chapter problems under password as well as by a collection of MATLAB programs for free access by the readers of the book, which can be used to solve a variety of optimiza- tion problems. These materials can be downloaded from the book's website:
http://www.ece.uvic.ca/~optimization/.
We are grateful to many of our past students at the University of Victoria, in particular, Drs. M. L. R. de Campos, S. Netto, S. Nokleby, D. Peters, and Mr. J. Wong who took our optimization courses and have helped improve the manuscript in one way or another; to Chi-Tang Catherine Chang for typesetting the first draft of the manuscript and for producing most of the illustrations; to R. Nongpiur for checking a large part of the index; and to R Ramachandran for proofreading the entire manuscript. We would also like to thank Professors M. Ahmadi, C. Charalambous, P. S. R. Diniz, Z. Dong, T. Hinamoto, and P. P.
Vaidyanathan for useful discussions on optimization theory and practice; Tony Antoniou of Psicraft Studios for designing the book cover; the Natural Sciences and Engineering Research Council of Canada for supporting the research that led to some of the new results described in Chapters 8, 9, and 16; and last but not least the University of Victoria for supporting the writing of this book over anumber of years.
Andreas Antoniou and Wu-Sheng Lu
AWGN additive white Gaussian noise BER bit-error rate
BFGS Broyden-Fletcher-Goldfarb-Shanno CDMA code-division multiple access
CMBER constrained minimum BER CP convex programming
DPP Davidon-Fletcher-Powell D-H Denavit-Hartenberg DNB dual Newton barrier
DS direct sequence
FDMA frequency-division multiple access FIR finite-duration impulse response FR Fletcher-Reeves GCO general constrained optimization GN Gauss-Newton IIR infinite-duration impulse response IP integer programming
KKT Karush-Kuhn-Tucker LCP linear complementarity problem LMI linear matrix inequality LP linear programming
LSQI least-squares minimization with quadratic inequality LU lower-upper
MAI multiple access interference ML maximum likelihood MPC model predictive control PAS primal affine-scaling PCM predictor-corrector method PNB primal Newton barrier QP quadratic programming SD steepest descent
SDP semidefinite programming SDPR-D SDP relaxation-dual SDPR-P SDP relaxation-primal SNR signal-to-noise ratio
SOCP second-order cone programming SQP sequential quadratic programming SVD singular-value decomposition TDMA time-division multiple access
THE OPTIMIZATION PROBLEM
1.1 Introduction
Throughout the ages, man has continuously been involved with the process of optimization. In its earliest form, optimization consisted of unscientific rituals and prejudices like pouring libations and sacrificing animals to the gods, con- sulting the oracles, observing the positions of the stars, and watching the flight of birds. When the circumstances were appropriate, the timing was thought to be auspicious (or optimum) for planting the crops or embarking on a war.
As the ages advanced and the age of reason prevailed, unscientific rituals were replaced by rules of thumb and later, with the development of mathematics, mathematical calculations began to be applied.
Interest in the process of optimization has taken a giant leap with the advent of the digital computer in the early fifties. In recent years, optimization techniques advanced rapidly and considerable progress has been achieved. At the same time, digital computers became faster, more versatile, and more efficient. As a consequence, it is now possible to solve complex optimization problems which were thought intractable only a few years ago.
The process of optimization is the process of obtaining the ‘best’, if it is pos- sible to measure and change what is ‘good’ or ‘bad’. In practice, one wishes the
‘most’ or ‘maximum’ (e.g., salary) or the ‘least’ or ‘minimum’ (e.g., expenses).
Therefore, the word ‘optimum’ is taken to mean ‘maximum’ or ‘minimum’ de- pending on the circumstances; ‘optimum’ is a technical term which implies quantitative measurement and is a stronger word than ‘best’ which is more appropriate for everyday use. Likewise, the word ‘optimize’, which means to achieve an optimum, is a stronger word than ‘improve’. Optimization theory is the branch of mathematics encompassing the quantitative study of optima and methods for finding them. Optimization practice, on the other hand, is the
collection of techniques, methods, procedures, and algorithms that can be used to find the optima.
Optimization problems occur in most disciplines like engineering, physics, mathematics, economics, administration, commerce, social sciences, and even politics. Optimization problems abound in the various fields of engineering like electrical, mechanical, civil, chemical, and building engineering. Typical areas of application are modeling, characterization, and design of devices, circuits, and systems; design of tools, instruments, and equipment; design of structures and buildings; process control; approximation theory, curve fitting, solution of systems of equations; forecasting, production scheduling, quality control;
maintenance and repair; inventory control, accounting, budgeting, etc. Some recent innovations rely almost entirely on optimization theory, for example, neural networks and adaptive systems.
Most real-life problems have several solutions and occasionally an infinite number of solutions may be possible. Assuming that the problem at hand admits more than one solution, optimization can be achieved by finding the best solution of the problem in terms of some performance criterion. If the problem admits only one solution, that is, only a unique set of parameter values is acceptable, then optimization cannot be applied.
Several general approaches to optimization are available, as follows:
1. Analytical methods 2. Graphical methods 3. Experimental methods 4. Numerical methods
Analytical methods are based on the classical techniques of differential cal- culus. In these methods the maximum or minimum of a performance criterion is determined by finding the values of parametersx1, x2, . . . , xnthat cause the derivatives off(x1, x2, . . . , xn)with respect tox1, x2, . . . , xnto assume zero values. The problem to be solved must obviously be described in mathematical terms before the rules of calculus can be applied. The method need not entail the use of a digital computer. However, it cannot be applied to highly nonlinear problems or to problems where the number of independent parameters exceeds two or three.
A graphical method can be used to plot the function to be maximized or min- imized if the number of variables does not exceed two. If the function depends on only one variable, say,x1, a plot off(x1)versusx1will immediately reveal the maxima and/or minima of the function. Similarly, if the function depends on only two variables, say,x1andx2, a set of contours can be constructed. A contouris a set of points in the(x1, x2)plane for whichf(x1, x2)is constant, and so a contour plot, like a topographical map of a specific region, will reveal readily the peaks and valleys of the function. For example, the contour plot of f(x1, x2)depicted in Fig. 1.1 shows that the function has a minimum at point
A. Unfortunately, the graphical method is of limited usefulness since in most practical applications the function to be optimized depends on several variables, usually in excess of four.
A 10
f(x , x ) = 01 2 f(x , x ) = 501 2
x1 x2
20 30
40 50
Figure 1.1. Contour plot off(x1, x2).
The optimum performance of a system can sometimes be achieved by direct experimentation. In this method, the system is set up and the process variables are adjusted one by one and the performance criterion is measured in each case. This method may lead to optimum or near optimum operating conditions.
However, it can lead to unreliable results since in certain systems, two or more variables interact with each other, and must be adjusted simultaneously to yield the optimum performance criterion.
The most important general approach to optimization is based on numerical methods. In this approach, iterative numerical procedures are used to generate a series of progressively improved solutions to the optimization problem, starting with an initial estimate for the solution. The process is terminated when some convergence criterion is satisfied. For example, when changes in the indepen- dent variables or the performance criterion from iteration to iteration become insignificant.
Numerical methods can be used to solve highly complex optimization prob- lems of the type that cannot be solved analytically. Furthermore, they can be readily programmed on the digital computer. Consequently, they have all but replaced most other approaches to optimization.
The discipline encompassing the theory and practice of numerical optimiza- tion methods has come to be known as mathematical programming [1]–[5].
During the past 40 years, several branches of mathematical programming have evolved, as follows:
1. Linear programming 2. Integer programming 3. Quadratic programming 4. Nonlinear programming 5. Dynamic programming
Each one of these branches of mathematical programming is concerned with a specific class of optimization problems. The differences among them will be examined in Sec. 1.6.
1.2 The Basic Optimization Problem
Before optimization is attempted, the problem at hand must be properly formulated. A performance criterionFmust be derived in terms ofnparameters x1, x2, . . . , xnas
F =f(x1, x2, . . . , xn)
Fis a scalar quantity which can assume numerous forms. It can be the cost of a product in a manufacturing environment or the difference between the desired performance and the actual performance in a system. Variablesx1, x2, . . . , xn
are the parameters that influence the product cost in the first case or the actual performance in the second case. They can be independent variables, like time, or control parameters that can be adjusted.
The most basic optimization problem is to adjust variablesx1, x2, . . . , xn
in such a way as to minimize quantityF. This problem can be stated mathe- matically as
minimizeF =f(x1, x2, . . . , xn) (1.1) QuantityFis usually referred to as theobjectiveorcost function.
The objective function may depend on a large number of variables, sometimes as many as 100 or more. To simplify the notation, matrix notation is usually employed. Ifxis a column vector with elementsx1, x2, . . . , xn, the transpose ofx, namely,xT, can be expressed as the row vector
xT = [x1x2 · · · xn]
In this notation, the basic optimization problem of Eq. (1.1) can be expressed as
minimizeF =f(x) for x∈En whereEnrepresents then-dimensional Euclidean space.
On many occasions, the optimization problem consists of finding the maxi- mum of the objective function. Since
max[f(x)] =−min[−f(x)]
the maximum of F can be readily obtained by finding the minimum of the negative ofF and then changing the sign of the minimum. Consequently, in this and subsequent chapters we focus our attention on minimization without loss of generality.
In many applications, a number of distinct functions ofxneed to be optimized simultaneously. For example, if the system of nonlinear simultaneous equations
fi(x) = 0 for i= 1,2, . . . , m
needs to be solved, a vector xis sought which will reduce all fi(x) to zero simultaneously. In such a problem, the functions to be optimized can be used to construct a vector
F(x) = [f1(x)f2(x) · · · fm(x)]T
The problem can be solved by finding a pointx = x∗ such thatF(x∗) = 0.
Very frequently, a pointx∗ that reduces all the fi(x) to zero simultaneously may not exist but an approximate solution, i.e.,F(x∗) ≈0, may be available which could be entirely satisfactory in practice.
A similar problem arises in scientific or engineering applications when the function of x that needs to be optimized is also a function of a continuous independent parameter (e.g., time, position, speed, frequency) that can assume an infinite set of values in a specified range. The optimization might entail adjusting variablesx1, x2, . . . , xn so as to optimize the function of interest over a given range of the independent parameter. In such an application, the function of interest can be sampled with respect to the independent parameter, and a vector of the form
F(x) = [f(x, t1)f(x, t2) · · · f(x, tm)]T
can be constructed, wheretis the independent parameter. Now if we let fi(x)≡f(x, ti)
we can write
F(x) = [f1(x)f2(x) · · · fm(x)]T
A solution of such a problem can be obtained by optimizing functionsfi(x) for i = 1,2, . . . , m simultaneously. Such a solution would, of course, be
approximate because any variations inf(x, t)between sample points are ig- nored. Nevertheless, reasonable solutions can be obtained in practice by using a sufficiently large number of sample points. This approach is illustrated by the following example.
Example 1.1The step responsey(x, t)of annth-order control system is re- quired to satisfy the specification
y0(x, t) =
⎧⎪
⎪⎨
⎪⎪
⎩
t for 0≤t <2 2 for 2≤t <3
−t+ 5 for 3≤t <4
1 for 4≤t
as closely as possible. Construct a vectorF(x) that can be used to obtain a functionf(x, t)such that
y(x, t)≈y0(x, t) for 0≤t≤5
SolutionThe difference between the actual and specified step responses, which constitutes theapproximation error, can be expressed as
f(x, t) =y(x, t)−y0(x, t) and iff(x, t)is sampled att= 0, 1,2, . . . , 5, we obtain
F(x) = [f1(x)f2(x) · · · f6(x)]T where
f1(x) = f(x, 0) =y(x, 0) f2(x) = f(x, 1) =y(x, 1)−1 f3(x) = f(x, 2) =y(x, 2)−2 f4(x) = f(x, 3) =y(x, 3)−2 f5(x) = f(x, 4) =y(x, 4)−1 f6(x) = f(x, 5) =y(x, 5)−1
The problem is illustrated in Fig. 1.2. It can be solved by finding a pointx=x∗ such thatF(x∗)≈0. Evidently, the quality of the approximation obtained for the step response of the system will depend on the density of the sampling points and the higher the density of points, the better the approximation.
Problems of the type just described can be solved by defining a suitable objec- tive function in terms of the element functions ofF(x). The objective function
0 1 2 3 4 5 1
2 3
y (x,t)
y (x,t) 0 f (x,t)
t
Figure 1.2. Graphical construction for Example 1.1.
must be ascalar quantityand its optimization must lead to the simultaneous optimization of the element functions ofF(x)in some sense. Consequently, a norm of some type must be used. An objective function can be defined in terms of theLpnormas
F ≡Lp= m
i=1
|fi(x)|p 1/p
wherepis an integer.1
Several special cases of theLp norm are of particular interest. Ifp= 1 F ≡L1 =
m i=1
|fi(x)|
and, therefore, in a minimization problem like that in Example 1.1, the sum of the magnitudes of the individual element functions is minimized. This is called anL1 problem.
Ifp= 2, theEuclidean norm F ≡L2=
m
i=1
|fi(x)|2 1/2
is minimized, and if the square root is omitted, the sum of the squares is mini- mized. Such a problem is commonly referred as aleast-squares problem.
1See Sec. A.8 for more details on vector and matrix norms. Appendix A also deals with other aspects of linear algebra that are important to optimization.
In the case wherep=∞, if we assume that there is a unique maximum of
|fi(x)|designatedFˆsuch that
Fˆ = max
1≤i≤m|fi(x)| then we can write
F ≡L∞ = lim
p→∞
m
i=1
|fi(x)|p 1/p
= ˆF lim
p→∞
m
i=1
|fi(x)| Fˆ
p1/p
Since all the terms in the summation except one are less than unity, they tend to zero when raised to a large positive power. Therefore, we obtain
F = ˆF = max
1≤i≤m|fi(x)|
Evidently, if theL∞normis used in Example 1.1, the maximum approximation error is minimized and the problem is said to be aminimax problem.
Often the individual element functions ofF(x)are modified by using con- stants w1, w2, . . . , wm asweights. For example, the least-squares objective function can be expressed as
F = m i=1
[wifi(x)]2
so as to emphasize important or critical element functions and de-emphasize unimportant or uncritical ones. IfFis minimized, the residual errors inwifi(x) at the end of the minimization would tend to be of the same order of magnitude, i.e.,
error in |wifi(x)| ≈ε and so
error in |fi(x)| ≈ ε
|wi|
Consequently, if a large positive weightwiis used withfi(x), a small residual error is achieved in|fi(x)|.
1.3 General Structure of Optimization Algorithms
Most of the available optimization algorithms entail a series of steps which are executed sequentially. A typical pattern is as follows:
Algorithm 1.1 General optimization algorithm Step 1
(a) Setk= 0and initializex0. (b) ComputeF0 =f(x0).
Step 2
(a) Setk=k+ 1.
(b) Compute the changes inxkgiven by column vector∆xkwhere
∆xTk = [∆x1∆x2 · · · ∆xn] by using an appropriate procedure.
(c) Setxk=xk−1+∆xk
(d) ComputeFk=f(xk)and∆Fk =Fk−1−Fk. Step 3
Check if convergence has been achieved by using an appropriate crite- rion, e.g., by checking∆Fkand/or∆xk. If this is the case, continue to Step 4; otherwise, go to Step 2.
Step 4
(a) Outputx∗=xkandF∗=f(x∗).
(b) Stop.
In Step 1, vectorx0is initialized by estimating the solution using knowledge about the problem at hand. Often the solution cannot be estimated and an arbitrary solution may be assumed, say, x0 = 0. Steps 2 and 3 are then executed repeatedly until convergence is achieved. Each execution of Steps 2 and 3 constitutes one iteration, that is,kis the number of iterations.
When convergence is achieved, Step 4 is executed. In this step, column vector
x∗ = [x∗1 x∗2 · · · x∗n]T =xk and the corresponding value ofF, namely,
F∗ =f(x∗)
are output. The column vectorx∗is said to be theoptimum,minimum,solution point, or simply theminimizer, andF∗ is said to be the optimum or minimum value of the objective function. The pairx∗ andF∗ constitute the solution of the optimization problem.
Convergence can be checked in several ways, depending on the optimization problem and the optimization technique used. For example, one might decide to stop the algorithm when the reduction inFkbetween any two iterations has become insignificant, that is,
|∆Fk|=|Fk−1−Fk|< εF (1.2)
whereεF is anoptimization tolerancefor the objective function. Alternatively, one might decide to stop the algorithm when the changes in all variables have become insignificant, that is,
|∆xi|< εx for i= 1, 2, . . . , n (1.3) where εx is an optimization tolerance for variablesx1, x2, . . . , xn. A third possibility might be to check if both criteria given by Eqs. (1.2) and (1.3) are satisfied simultaneously.
There are numerous algorithms for the minimization of an objective function.
However, we are primarily interested in algorithms that entail the minimum amount of effort. Therefore, we shall focus our attention on algorithms that are simple to apply, are reliable when applied to a diverse range of optimization problems, and entail a small amount of computation. A reliable algorithm is often referred to as a ‘robust’ algorithm in the terminology of mathematical programming.
1.4 Constraints
In many optimization problems, the variables are interrelated by physical laws like the conservation of mass or energy, Kirchhoff’s voltage and current laws, and other system equalities that must be satisfied. In effect, in these problems certain equality constraints of the form
ai(x) = 0 for x∈En
wherei= 1,2, . . . , pmust be satisfied before the problem can be considered solved. In other optimization problems a collection of inequality constraints might be imposed on the variables or parameters to ensure physical realizability, reliability, compatibility, or even to simplify the modeling of the problem. For example, the power dissipation might become excessive if a particular current in a circuit exceeds a given upper limit or the circuit might become unreliable if another current is reduced below a lower limit, the mass of an element in a specific chemical reaction must be positive, and so on. In these problems, a collection of inequality constraints of the form
cj(x)≥0 for x∈En
wherej = 1,2, . . . , q must be satisfied before the optimization problem can be considered solved.
An optimization problem may entail a set of equality constraints and possibly a set of inequality constraints. If this is the case, the problem is said to be a constrained optimization problem. The most general constrained optimization problem can be expressed mathematically as
minimizef(x) for x∈En (1.4a) subject to: ai(x) = 0 for i= 1,2, . . . , p (1.4b) cj(x) ≥ 0 for j= 1,2, . . . , q (1.4c) A problem that does not entail any equality or inequality constraints is said to be anunconstrained optimization problem.
Constrained optimization is usually much more difficult than unconstrained optimization, as might be expected. Consequently, the general strategy that has evolved in recent years towards the solution of constrained optimization problems is to reformulate constrained problems as unconstrained optimiza- tion problems. This can be done by redefining the objective function such that the constraints are simultaneously satisfied when the objective function is minimized. Some real-life constrained optimization problems are given as Examples 1.2 to 1.4 below.
Example 1.2Consider a control system that comprises a double inverted pen- dulum as depicted in Fig. 1.3. The objective of the system is to maintain the pendulum in the upright position using the minimum amount of energy. This is achieved by applying an appropriate control force to the car to damp out any displacementsθ1(t)andθ2(t). Formulate the problem as an optimization problem.
θ1
θ2
Μ u(t)
Figure 1.3. The double inverted pendulum.
SolutionThe dynamic equations of the system are nonlinear and the standard practice is to apply a linearization technique to these equations to obtain a small-signal linear model of the system as [6]
˙
x(t) =Ax(t) +fu(t) (1.5)
where
x(t) =
⎡
⎢⎢
⎣ θ1(t) θ˙1(t) θ2(t) θ˙2(t)
⎤
⎥⎥
⎦, A=
⎡
⎢⎢
⎣
0 1 0 0
α 0 −β 0
0 0 0 1
−α 0 α 0
⎤
⎥⎥
⎦, f =
⎡
⎢⎢
⎣ 0
−1 0 0
⎤
⎥⎥
⎦ with α > 0, β > 0,and α = β. In the above equations, x(t),˙ θ˙1(t), and θ˙2(t)represent the first derivatives ofx(t),θ1(t), andθ2(t), respectively, with respect to time, θ¨1(t)andθ¨2(t)would be the second derivatives ofθ1(t) and θ2(t), and parametersαandβdepend on system parameters such as the length and weight of each pendulum, the mass of the car, etc. Suppose that at instant t= 0small nonzero displacementsθ1(t)andθ2(t)occur, which would call for immediate control action in order to steer the system back to the equilibrium state x(t) = 0 at timet = T0. In order to develop a digital controller, the system model in (1.5) is discretized to become
x(k+ 1) =Φx(k) +gu(k) (1.6) where Φ = I+ ∆tA, g = ∆tf, ∆t is the sampling interval, and I is the identity matrix. Letx(0)=0be given and assume thatT0is a multiple of∆t, i.e.,T0 =K∆twhereKis an integer. We seek to find a sequence of control actionsu(k)fork = 0, 1, . . . , K−1such that the zero equilibrium state is achieved att=T0, i.e.,x(T0) =0.
Let us assume that the energy consumed by these control actions, namely, J =
K−1 k=0
u2(k)
needs to be minimized. This optimal control problem can be formulated ana- lytically as
minimizeJ =
K−1 k=0
u2(k) (1.7a)
subject to: x(K) = 0 (1.7b)
From Eq. (1.6), we know that the state of the system att=K∆tis determined by the initial value of the state and system model in Eq. (1.6) as
x(K) = ΦKx(0) +
K−1 k=0
ΦK−k−1gu(k)
≡ −h+
K−1 k=0
gku(k)
whereh=−ΦKx(0)andgk =ΦK−k−1g. Hence constraint (1.7b) is equiv- alent to
K−1 k=0
gku(k) =h (1.8)
If we defineu = [u(0) u(1) · · · u(K −1)]T andG = [g0 g1 · · · gK−1], then the constraint in Eq. (1.8) can be expressed asGu =h, and the optimal control problem at hand can be formulated as the problem of finding authat solves the minimization problem
minimizeuTu (1.9a)
subject to: a(u) =0 (1.9b)
where a(u) = Gu−h. In practice, the control actions cannot be made arbitrarily large in magnitude. Consequently, additional constraints are often imposed on|u(i)|, for instance,
|u(i)| ≤m for i= 0, 1, . . . , K−1 These constraints are equivalent to
m+u(i) ≥ 0 m−u(i) ≥ 0 Hence if we define
c(u) =
⎡
⎢⎢
⎢⎢
⎢⎣
m+u(0) m−u(0)
... m+u(K−1) m−u(K−1)
⎤
⎥⎥
⎥⎥
⎥⎦
then the magnitude constraints can be expressed as
c(u)≥0 (1.9c)
Obviously, the problem in Eq. (1.9) fits nicely into the standard form of opti- mization problems given by Eq. (1.4).
Example 1.3High performance in modern optical instruments depends on the quality of components like lenses, prisms, and mirrors. These components have reflecting or partially reflecting surfaces, and their performance is limited by the reflectivities of the materials of which they are made. The surface reflectivity
can, however, be altered by the deposition of a thin transparent film. In fact, this technique facilitates the control of losses due to reflection in lenses and makes possible the construction of mirrors with unique properties [7][8].
As is depicted in Fig. 1.4, a typicalN-layer thin-film system consists ofN layers of thin films of certain transparent media deposited on a glass substrate.
The thickness and refractive index of the ith layer are denoted asxi andni, respectively. The refractive index of the medium above the first layer is denoted asn0. Ifφ0is the angle of incident light, then the transmitted ray in the(i−1)th layer is refracted at an angleφiwhich is given by Snell’s law, namely,
nisinφi=n0sinφ0
n0
n3 n2
nN
nN+1 layer 1
layer 2
layer 3
x2
x1 n1
Substrate φ1
φ0
φ2
. . .
layerN xn
φN
Figure 1.4. AnN-layer thin-film system.
Given angleφ0and the wavelength of light,λ, the energy of the light reflected from the film surface and the energy of the light transmitted through the film surface are usually measured by the reflectanceRand transmittanceT which satisfy the relation
R+T = 1
For anN-layer system,Ris given by (see [9] for details)
R(x1, . . . , xN, λ) = η0−y η0+y
2 (1.10)
y = c
b (1.11)
b
c =
N
k=1
cosδk (jsinδk)/ηk
jηksinδk cosδk
1 ηN+1
(1.12) wherej=√
−1and δk = 2πnkxkcosφk
λ (1.13)
ηk =
⎧⎪
⎪⎪
⎪⎪
⎪⎪
⎨
⎪⎪
⎪⎪
⎪⎪
⎪⎩
nk/cosφk for light polarized with the electric vector lying in the plane of incidence nkcosφk for light polarized with the electric
vector perpendicular to the plane of incidence
(1.14)
The design of a multilayer thin-film system can now be accomplished as follows:
Given a range of wavelenghsλl≤λ≤λu and an angle of incidenceφ0, find x1, x2, . . . , xN such that the reflectanceR(x, λ)best approximates a desired reflectance Rd(λ) for λ ∈ [λl, λu]. Formulate the design problem as an optimization problem.
SolutionIn practice, the desired reflectance is specified at grid pointsλ1, λ2, . . . , λKin the interval[λl, λu]; hence the design may be carried out by selecting xisuch that the objective function
J = K i=1
wi[R(x, λi)−Rd(λi)]2 (1.15) is minimized, where
x= [x1 x2 · · · xN]T
andwi >0is a weight to reflect the importance of term[R(x, λi)−Rd(λi)]2 in Eq. (1.15). If we letη= [1ηN+1]T, e+ = [η01]T, e−= [η0 −1]T, and
M(x, λ) = N k=1
cosδk (jsinδk)/ηk
jηksinδk cosδk
thenR(x, λ)can be expressed as R(x, λ) =bη0−c
bη0+c 2 =
eT−M(x, λ)η eT+M(x, λ)η
2
(1.16)
Finally, we note that the thickness of each layer cannot be made arbitrarily thin or arbitrarily large and, therefore, constraints must be imposed on the elements ofxas
dil≤xi≤diu for i= 1,2, . . . , N (1.17) The design problem can now be formulated as the constrained minimization problem
minimizeJ = K i=1
wi
⎡
⎣
eT−M(x, λi)η eT+M(x, λi)η
2
−Rd(λi)
⎤
⎦
2
(1.18a) subject to: xi−dil ≥ 0 for i= 1,2, . . . , N (1.18b) diu−xi ≥ 0 for i= 1,2, . . . , N (1.18c)
Example 1.4Quantitiesq1, q2, . . . , qm of a certain product are produced by mmanufacturing divisions of a company, which are at distinct locations. The product is to be shipped tondestinations that require quantitiesb1, b2, . . . , bn. Assume that the cost of shipping a unit from manufacturing divisionito des- tinationjiscij withi= 1,2, . . . , mandj = 1,2, . . . , n. Find the quantity xij to be shipped from divisionito destinationjso as to minimize the total cost of transportation, i.e.,
minimizeC= m i=1
n j=1
cijxij
This is known as the transportation problem. Formulate the problem as an optimization problem.
Solution Note that there are several constraints on variablesxij. First, each division can provide only a fixed quantity of the product, hence
n j=1
xij =qi for i= 1,2, . . . , m
Second, the quantity to be shipped to a specific destination has to meet the need of that destination and so
m i=1
xij =bj for j= 1,2, . . . , n In addition, the variablesxij are nonnegative and thus, we have
xij ≥0 for i= 1,2, . . . , m and j= 1,2, . . . , n
If we let
c = [c11 · · · c1nc21 · · · c2n · · · cm1 · · · cmn]T x = [x11 · · · x1nx21 · · · x2n · · · xm1 · · · xmn]T
A =
⎡
⎢⎢
⎢⎢
⎢⎢
⎢⎢
⎢⎢
⎢⎣
1 1 · · · 1 0 0 · · · 0 · · · · 0 0 · · · 0 1 1 · · · 1 · · · ·
· · · · 0 0 · · · 0 0 0 · · · 0 · · · 1 1 · · · 1 1 0 · · · 0 1 0 · · · 0 · · · 1 0 · · · 0 0 1 · · · 0 0 1 · · · 0 · · · 0 1 · · · 0
· · · · 0 0 · · · 1 0 0 · · · 1 · · · 0 0 · · · 1
⎤
⎥⎥
⎥⎥
⎥⎥
⎥⎥
⎥⎥
⎥⎦
b = [q1 · · · qm b1 · · · bn]T
then the minimization problem can be stated as
minimizeC = cTx (1.19a)
subject to: Ax =b (1.19b)
x ≥0 (1.19c)
wherecTxis the inner product of c andx. The problem in Eq. (1.19) like those in Examples 1.2 and 1.3 fits into the standard optimization problem in Eq. (1.4). Since both the objective function in Eq. (1.19a) and the constraints in Eqs. (1.19b) and (1.19c) are linear, the problem is known as alinear program- ming (LP) problem(see Sect. 1.6.1).
1.5 The Feasible Region
Any pointxthat satisfies both the equality as well as the inequality constraints is said to be afeasible pointof the optimization problem. The set of all points that satisfy the constraints constitutes thefeasible domain regionoff(x). Evidently, the constraints define a subset of En. Therefore, the feasible region can be defined as a set2
R={x: ai(x) = 0for i= 1,2, . . . , pandcj(x)≥0for j= 1,2, . . . , q} whereR ⊂En.
The optimum pointx∗ must be located in the feasible region, and so the general constrained optimization problem can be stated as
minimizef(x) for x∈ R
2The above notation for a set will be used consistently throughout the book.
Any pointxnot inRis said to be anonfeasible point.
If the constraints in an optimization problem are all inequalities, the con- straints divide the points in theEnspace into three types of points, as follows:
1. Interior points 2. Boundary points 3. Exterior points
Aninterior pointis a point for whichcj(x)>0for allj. Aboundary pointis a point for which at least onecj(x) = 0, and anexterior pointis a point for which at least onecj(x)<0. Interior points are feasible points, boundary points may or may not be feasible points, whereas exterior points are nonfeasible points.
If a constraintcm(x)is zero during a specific iteration, the constraint is said to beactive, and ifcm(x∗)is zero when convergence is achieved, the optimum pointx∗is located on the boundary. In such a case, the optimum point is said to be constrained. If the constraints are all equalities, the feasible points must be located on the intersection of all the hypersurfaces corresponding toai(x) = 0 fori= 1,2, . . . , p. The above definitions and concepts are illustrated by the following two examples.
Example 1.5 By using a graphical method, solve the following optimization problem
minimizef(x) = x21+x22−4x1+ 4 subject to: c1(x) = x1−2x2+ 6≥0
c2(x) = −x21+x2−1≥0 c3(x) = x1 ≥0
c4(x) = x2 ≥0 SolutionThe objective function can be expressed as
(x1−2)2+x22 =f(x)
Hence the contours off(x) in the(x1, x2) plane are concentric circles with radiusf(x)centered atx1 = 2, x2 = 0. Constraintsc1(x)andc2(x)dictate that
x2≤ 12x1+ 3 and
x2≥x21+ 1
respectively, while constraintsc3(x)andc4(x)dictate thatx1andx2be positive.
The contours off(x)and the boundaries of the constraints can be constructed as shown in Fig. 1.5.
The feasible region for this problem is the shaded region in Fig. 1.5. The solution is located at point A on the boundary of constraint c2(x). In effect,
