OPTIMIZATION Algorithms and Applications

(1)

OPTIMIZATION

Algorithms and

Applications

(2)

(3)

OPTIMIZATION

Algorithms and Applications

Rajesh Kumar Arora

Senior Engineer

Vikram Sarabhai Space Centre

Indian Space Research Organization

Trivandrum, India

(4)

pedagogical approach or particular use of the MATLAB® software.

CRC Press

Taylor & Francis Group

6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742

CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works

Version Date: 20150205

International Standard Book Number-13: 978-1-4987-2115-8 (eBook - PDF)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmit- ted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.

com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.

Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com

(5)

Dedicated to my mother

(6)

(7)

Preface

There are numerous books on the subject of optimization, attributable to a number of reasons. First, the subject itself is mathematically rigorous and there are a number of solution methods that need to be examined and understood. No single solution method can be applied to all types of optimization problems. Thus a clear understanding of the problem, as well as solution techniques, is required to obtain a proper and meaningful solution to the optimization problem. With the progression of time, optimization problems have also become complex. It is necessary not only to obtain the global optimum solution, but to find local optima as well. Today’s problems are also of the multiobjective type, where conflicting objective functions are to be handled. There is also a need to simultaneously handle objective functions and constraints of different disciplines, resulting in multidisciplinary design optimization (MDO) problems that are handled using different architectures. Gradient-based methods were popular until the 1990s. At present, a large number of complex problems are solved using guided random search methods such as genetic algorithm, simulated annealing, and particle swarm optimization (PSO) techniques. Even hybrid algorithms, that use a combination of gradient-based and stochastic methods, are also very popular. Different authors have addressed these issues separately, resulting in a number of books in this area.

So how does this book differ from the others? The solution techniques are detailed in such a way that more emphasis is given to the concepts and rigorous mathematical details and proofs are avoided. It is observed that a method can be understood better if different parameters in the algorithm are plotted or printed over different iterations while solving a problem. This can be accomplished by writing a software code for the method or the algorithm.

It is often difficult for a newcomer to write a software code if the algorithm such as, say, Broyden–Fletcher–Goldfarb–Shanno (BFGS) or PSO is given to him or her. In this book, a step-by-step approach is followed in developing the software code from the algorithm. The codes are then applied to solve some standard functions taken from the literature. This creates understanding and confidence in handling different solution methods. The software codes are then suitably modified to solve some real-world problems. A few books on optimization have also followed this approach. However, the software code in these books is hard to correlate with the corresponding algorithms mentioned in the book and readers are forced to use them as black box optimization tools. The codes presented in this book are user friendly in the sense that they can be easily understood. A number of practical problems are solved using these codes.

(12)

The codes are written in the MATLAB^® environment and the use of ready- made optimization routines available in MATLAB is avoided. The algorithms are developed right from computing the gradient or Hessian of a function to a complex algorithm such as for solving a constraint optimization problem.

MATLAB is a software package for technical computing that performs both computing and visualization with ease. It has a number of built-in functions that can be used by an individual’s application. The main advantage of MATLAB is the ease with which readers can translate their ideas into an application.

The book covers both gradient and stochastic methods as solution techniques for unconstrained and constrained optimization problems. A separate chapter (Chapter 5) is devoted to stochastic methods, where genetic algorithm, PSO, simulated annealing, ant colony optimization, and tabu search methods are discussed. With simple modifications of the basic PSO code, one can also solve nonconvex multiobjective optimization problems.

This is probably the first optimization book in which MDO architectures are introduced (Chapter 9). Software codes are also developed for the simplex method and affine-scaling interior point method for solving linear programming problems. Gomory’s cutting plane method, branch-and-bound method, and Balas’ algorithm are also discussed in the chapter on integer programming (Chapter 10). A number of real-world problems are solved using the MATLAB codes given in the book. Some applications that are included in this book are solving a complex trajectory design problem of a robot (Chapter 3), multiobjective shape optimization problem of a reentry body (Chapter 7), portfolio optimization problem (Chapter 4), and so forth.

I thank my organization, Vikram Sarabhai Space Centre (a lead center of Indian Space Research Organisation [ISRO]), for giving permission to publish this book. The book has been reviewed internally by Dr. Mohankumar D., Head, Computer Division. I thank him for his suggestions and corrections. I thank Mr. Pandian, S., Deputy Director, VSSC for supporting the idea to write this book. I am ever grateful to Prof. M Seetharama Bhat from IISc, Bangalore and Dr. Adimurthy, V. for their support during the last ten years.

I thank my colleagues Dr. Jayakumar K., Mr. Priyankar, B., Mr. Sajan Daniel and Mr. Amit Sachdeva for many hours of discussions on book-related aspects.

I am grateful to Taylor & Francis Group for agreeing to publish this book and agreeing to most of my suggestions. Much credit should be given to Ms. Aastha Sharma, Editor, for her prompt actions and follow-up with the reviewers. Thanks are also due to three anonymous reviewers for their criti- cal remarks, corrections, and suggestions. I thank Mr. Sarfraz Khan, assistant to Ms. Aastha Sharma, for providing online support in signing the contract.

I also thank Mr. David Fausel for coordinating and reviewing the style and art files of the book. My sincere thanks to Mr. Ed Curtis and Ms. Amor Nanas for language corrections, copyediting, and other production related works.

The cover page is designed by Mr. Kevin Craig.

(13)

xiii Preface

I thank the MATLAB book program for supporting the idea of this book on optimization with MATLAB codes. They have also agreed to give wide publicity to this book on their website, for which I am grateful.

I thank my wife, Manju, and children, Abhinav and Aditi, for their patience during the last two years. In fact my whole family—father, brothers, sister, and in-laws—are eagerly waiting for the launch of this book.

This is the first edition of this book. Some errors and omissions are expected. The MATLAB codes are validated with a number of test problems taken from the literature. It is still possible that some pathways in the codes would not have been exercised during this validation. As a result, no claim is made that these codes are bug-free. I request readers to report corrections and suggestions on this book at [email protected] or arora_rajesh@rediffmail .com.

The MATLAB codes mentioned in this book can be downloaded from the weblink http://www.crcpress.com/product/isbn/9781498721127.

Rajesh Kumar Arora, PhD, MBA, FIE MATLAB^® is a registered trademark of The MathWorks, Inc. For product information, please contact:

The MathWorks, Inc.

3 Apple Hill Drive

Natick, MA 01760-2098 USA Tel: 508 647 7000

Fax: 508-647-7001

E-mail: [email protected] Web: www.mathworks.com

(14)

(15)

Author

Rajesh Kumar Arora is a senior engineer with the Indian Space Research Organization (ISRO), where he has been working for more than two decades.

He earned his PhD in aerospace engineering from the Indian Institute of Science, Bangalore. He has published a book titled Fundamentals of Aerospace Engineering. His area of research includes mission design, simulation of launch vehicle systems, and trajectory optimization. He also has a master’s degree in business administration and is a Fellow of Institution of Engineers (India).

(16)

(17)

1 Introduction

1.1 Historical Review

Optimization means finding the best solution among many feasible solutions that are available to us. Feasible solutions are those that satisfy all the constraints in the optimization problem. The best solution could be mini- mizing the cost of a process or maximizing the efficiency of a system. Some simple optimization problems that come to mind are machine allocation and diet problems. In the machine allocation problem, one has to find how jobs are to be allocated to different machines of different capacities and with different operating costs so as to meet the production target with minimum cost. In the diet problem, different food types are available with different nutritional contents at different costs. The aim is to estimate different quantities of food so that nutritional requirements are met for an individual at minimum cost.

Though rigorous mathematical analysis of the optimization problems was carried out during the 20th century, the roots can be traced back to about 300 b.c., when the Greek mathematician Euclid evaluated the minimum dis- tance between a point and a line. Another Greek mathematician, Zenedorous, showed in 200 b.c. that a figure bounded by a line that has a maximum area for a given perimeter is a semicircle.

In the 17th century, Pierre de Fermat, a French mathematician, laid the foundation of calculus. He showed that the gradient of a function vanishes at the maximum or minimum point. Moving further in the timeline, Newton and Leibniz laid mathematical details for the calculus of variations. This method deals with maxima or minima of functionals. The foundation for the calculus of variations is credited to Euler and Lagrange (in the 18th century), as they provided rigorous mathematical details on this topic. Subsequently, Gauss and Legendre developed the least squares method, which is exten- sively used even today. Cauchy used the steepest descent method to solve unconstrained optimization problems.

(18)

The first textbook on optimization was authored by Harris Hancock and published in 1917. In 1939, Leonid Kantorovich presented the linear programming (LP) model and an algorithm for solving it. A few years later in 1947, George Dantzig presented a simplex method for solving LP problems.

Kantorovich and Dantzig are regarded as pioneers who provided break- throughs in the development of optimization techniques. The conditions for constrained optimization were brought together by Harold Kuhn and Albert Tucker in 1951 and also earlier by William Karush in 1939. Richard Bellman laid the principles of dynamic programming problems in which a complex problem is broken down into smaller subproblems. Ralph Gomory’s contribution to the development of integer programming is worth noting, as in this type of optimization problem, design variables can take integer values such as 0 and 1.

With the advent of computers in the 1980s, subsequently many large-scale problems were solved. Present-day problems in the optimization area are of the multidisciplinary and multiobjective type. The solution techniques that are employed today to solve complex optimization problems are not just gradient-based algorithms, but also include nontraditional methods such as genetic algorithms, ant colony optimization, and particle swarm optimization that mimic natural processes.

Today, optimization methods are required to solve problems from all disciplines, whether economics, sciences, or engineering. As a result of stiff competition in virtually all disciplines, the role of optimization has become still more substantial as one aims to minimize the cost of a product or wants to allocate resources judiciously. A simple example from the subject field of aerospace engineering can prove this point. The cost of putting 1 kilogram of payload in a low Earth orbit is typically about US$15,000. The fuel and structural weight of the different stages of the rocket strongly influence the payload mass, as does the trajectory of the rocket. Of course, one can reduce the structural weight of a stage only to the extent it should not fail because of aerodynamic and other loads. The optimization problem that aims at maximizing the payload mass is highly complex and requires algorithms that run on high-speed computers. Even if the optimization technique results in few extra kilograms in payload, it represents large revenue for the space agency.

In the next section, we introduce to the optimization problem design variables, constraints, and applications of optimization in different domains.

Further in the chapter, modeling aspects of a physical problem are explained that convert the verbal problem to a mathematical form. The solution of simple optimization problems with up to two design variables is explained by the graphical method. The importance of convex function in optimization is then explained. The chapter concludes with an introduction to the mathematical preliminaries of the gradient vector, Hessian matrix, directional directive, and linear and quadratic approximation of the function. The road map of this chapter is given in Figure 1.1.

(19)

3 Introduction

1.2 Optimization Problem

In an optimization problem, a function is to be maximized or minimized.

The function that is being optimized is referred to as the objective function or the performance index. The function is a quantity such as cost, profit, efficiency, size, shape, weight, output, and so on. It goes without saying that cost minimization or profit maximization are prime considerations for most organizations. Certain types of equipment, such as air conditioners or refrig- erators, are designed with different optimization criteria to have higher efficiency in terms of reducing energy consumption requirements of the user.

However, this higher efficiency evidently comes at a higher cost to the user.

Weight minimization is a prime consideration for aerospace applications.

The variables in the objective function are denoted the design variables or decision variables. Typically it could be the dimensions of a structure or its material attributes, for a structure optimization problem. From practical considerations, design variables can take values within a lower and an upper limit only. For instance, the maximum capacity of a machine is limited to a certain value. The design variables can be a real or a discrete number, binary, or integer type. Though a majority of the design variables in the optimization problems are real, some variables can also be discrete. For example, pipe sizes come in standard numbers such as 1, 2, or 5 inches. If pipe size is

Optimization problem

Modeling of the problem

Convexity

Gradient vector, Hessian matrix, and directional derivative

Linear and quadratic approximation Solution with graphical method

Book layout

FIGURE 1.1

Road map of Chapter 1.

(20)

used as a design variable in an optimization problem, it has to be treated as discrete only. There is no point in selecting it as a real number and getting a solution such as, say, 3.25 inches, a pipe dimension that really does not exist.

See Table 1.1 for some typical objective functions and design variables for optimization problems from different disciplines.

The optimization problem can be mathematically expressed as follows.

Minimize

f(x) (1.1)

subject to

g_i(x) ≤ 0 i = 1, 2,…, m < n (1.2) h_j(x) = 0 j = 1, 2,…, r < n (1.3)

x_l ≤ x ≤ x_u where x is a vector of n design variables given by

x=















 x x xn

1 2



The functions f, g_i, and h_j are all differentiable. The design variables are bounded by x_l and x_u. The constraints g_i are called inequality constraints and TablE 1.1

Typical Optimization Problems

Discipline Design Variables Objective Function

Manufacturing Productivity from different machines Minimize cost

Corporate Different capitals from projects Maximize the net present value Airline Different aircrafts, different routes Maximize the profit

Aerospace Propellant fraction in different stages Maximize the payload

Agriculture Different crops Maximize the yield

Biology Gene interaction Network stability

Electronics Size of the devices Minimize the power consumption Portfolio Investment in stocks/bonds Maximize the return

Thermal Dimensions and material properties Minimize the heat load

(21)

5 Introduction

hj are called equality constraints. An example in the aerospace industry is to restrict the dimensions of the spacecraft so that in can be accommodated inside the payload fairing of a rocket. These restrictions are the constraints of the optimization problem. The constraints are functions of the design variables. In addition, the number of equality or inequality constraints is lower than the number of design variables (n). If the design variables satisfy all the constraints, they represent a feasible set and any element from the set is called a feasible point. The design variables at which the minimum of f(x) is reached are given by x*. If the optimization problem does not have any constraints, it is referred to as an unconstrained optimization problem. If the objective function and constraints are linear functions in x then the optimi- zation problem is termed a linear programming problem (LPP).

1.3 Modeling of the Optimization Problem

Modeling refers to expressing observations about a problem in mathematical form using basic building blocks of mathematics such as addition, sub- traction, multiplication, division, functions, and numbers with proper units.

Observations refer to data obtained for the problem in hand, by varying certain parameters of the problem through experiments. Further, mathematical models provide predictions of the behavior of the problem for different inputs. If the model does not yield expected results, it has to be refined by conducting further experiments. The mathematical model is not unique for different problems, as observed data can be discrete (defined at select inter- vals) or continuous and can vary in different fashion (say, linear or quadratic) with change in input parameters. Some simple mathematical models of different physical phenomena are presented next.

The pressure (P), volume (V), and temperature (T) relationship of a gas is given by Boyle’s law as

PV = kT (1.4)

where k is a constant. Using this mathematical model, the behavior of a gas can be predicted (say, pressure) for the different input parameters (say, temperature), keeping the volume of the gas constant.

An example from economics could be constructing a mathematical model for the demand–supply problem. The price of a product is to be calculated so as to maximize the profit. It is well known that if the price of the merchan- dise is kept high, profit per unit will increase but then demand for the product may be low. Likewise, if the price of the product is kept low, profit per unit will decrease, but then demand for the product may be higher. Typically, demand (D) varies with price (P) as

(22)

D c

c P

= +

1

2 2 (1.5)

where c₁ and c₂ are constants.

Some problems can be written mathematically in differential equation form.

A differential equation contains an unknown function and its derivatives.

As the derivative represents the rate of change of a function, the differential equation represents the continuously varying quantity and its rate of change. For example, the temperature change (with respect to time) of an object is proportional to the difference between the temperature (T) of the object and that of its surroundings (T_s) and can be represented in differential equation form as

dT

dt = −k T T

(

− s

)

(1.6)

Equation 1.6 is also referred to as Newton’s law of cooling. The solution of the differential equation is a function that satisfies the differential equation for all values of the independent variable in the domain. As the name suggests, independent variables are changed during an experiment and the dependent variable responds to this depending on on the type of the experiment being conducted. A differential equation can have many solutions (referred to as general solution). A particular solution is one such solution. Often, a differential equation has a closed form solution. For example, the solution for the differential equation representing Newton’s law of cooling is

T(t) = Ts + (To − Ts)e^−kt (1.7) Not all problems have closed form solutions and such problems have to be numerically simulated to arrive at the solutions.

Therefore, using modeling, one can construct the objective function as well as the constraint functions for the optimization problem. One can then use different optimization techniques for solving such problems. The following examples illustrate how to formulate an optimization problem by constructing the objective and constraint functions.

Example 1.1

In a diet problem, an individual has to meet his daily nutritional requirements from a menu of available foods at a minimum cost. The available food items are milk, juice, fish, fries, and chicken. The nutrient requirements to be met are for proteins, vitamins, calcium, calories, and carbohydrates. Table 1.2 shows the cost in dollars of the food items per serving, nutrient values are shown in rows against their names (such as

(23)

7 Introduction

proteins, vitamins, etc.) per serving, and the last column indicates the minimum daily requirements of the nutrients.

The first step is to select the design variables for the problem. It appears obvious to select quantities of food items such as fish, fries, and so on as the design variables. Let us represent the design variables by x₁, x₂, x₃, x₄, and x₅ for quantities in milk, juice, fish, fries, and chicken respectively.

As discussed earlier, the objective function and constraints are a function of these design variables. In this particular problem, the objective function is to minimize the cost of the food items purchased. If x₃ is the quantity of fish ordered and $2 is its unit price, then the total cost of the fish item is 2x₃. In a similar way, we can evaluate the cost of other items such as milk, juice, and so on. Hence the total cost of the food items is

1.1x₁ + 1.2x₂ + 2x₃ + 1.3x₄ + 3x₅

Note that the cost function or the objective function is linear; that is, x₁ is not dependent on x₂ or any other variable. Having defined the objective function, let us define the constraints for the problem. In the problem it is clearly mentioned that the nutritional needs of the individual have to be met. For example, a minimum protein requirement of 60 units is to be met. Similarly, minimum requirements of other nutrients such as vitamins, calcium, and so forth are also to be met. Now, we can write the first constraint as

8x₁ + 2x₂ + 15x₃ + 4x₄ + 30x₅ ≥ 60 (1.8) Note that this constraint is an inequality. In a similar fashion, we can write other constraints. We are now ready to write the objective function and constraints for the diet problem.

Minimize

1.1x₁ + 1.2x₂ + 2x₃ + 1.3x₄ + 3x₅ (1.9) TablE 1.2

Data for the Diet Problem

Milk Juice Fish Fries Chicken Required

Cost 1.1 1.2 2.0 1.3 3.0

Proteins 8 2 15 4 30 60

Vitamins 9 3 3 1 9 100

Calcium 35 3 17 1 16 120

Calories 100 90 350 200 410 2100

Carbohydrates 10 20 40 25 40 400

Note: Construct the objective function and the constraints for this optimization problem.

(24)

subject to

8x₁ + 2x₂ + 15x₃ + 4x₄ + 30x₅ ≥ 60 (1.10) 9x₁ + 3x₂ + 3x₃ + x₄ + 9x₅ ≥ 100 (1.11) 35x₁ + 3x₂ + 17x₃ + x₄ + 16x₅ ≥ 120 (1.12) 100x1 + 90x2 + 350x3 + 200x4 + 410x5 ≥ 2100 (1.13) 10x₁ + 20x₂ + 40x₃ + 25x₄ + 40x₅ ≥ 400 (1.14) Once the optimization problem is defined, one has to use standard optimization techniques in evaluating the design variables x₁, x₂, x₃, x₄, and x₅. These methods are described in the later chapters. In this chapter, we are focusing on the formulation of the optimization problem.

Example 1.2

A soft drink manufacturer needs to produce a cylindrical can that can hold 330 mL of a soft drink. He wants to make the dimensions of the container such that the amount of material used in its construction is minimized. Formulate the optimization problem by writing down the objective function and the constraint.

The design variables for the optimization problem are the radius and the height of the can. Let these variables be denoted by x₁ and x₂ with units in millimeters (Figure 1.2). The cylindrical can consists of a curved portion and two circular ends. The area of the curved portion is given by 2πx1x₂ and the area of two circular lids is given by 2πx . Hence, the total 12

area that needs to be minimized is 2πx x_{1 2}+2πx₁². The volume of the can is given by πx x12

2. This volume is to be limited to 330 mL or 330,000 mm³. Now we are ready to formulate the optimization problem.

Minimize

2πx x_{1 2}+2πx₁² (1.15)

x1

x₂

FIGURE 1.2 Cylindrical can.

(25)

9 Introduction

subject to

πx x12

2=330 000, (1.16)

Note that in this optimization problem, the constraint is an equality.

Example 1.3

The shape of a reentry body is a spherical nose, a conical body, and a flared bottom (see Figure 1.3). The design variables through which the configura- tion of the reentry body can be altered are nose radius (R_n), cone length (l₁), cone angle (θ1), flare length (l2), and flare angle (θ2). By varying the design variables, the area (A) of the reentry capsule is to be minimized. As the reentry capsule has to house electronic packages and other instruments, it must have a certain minimum volume (V), which is specified as 1 m³.

The design variables are bounded between a minimum and maximum value. Rn can take a value between 0.4 and 0.6 m, l1 and l2 can take a value between 0.4 and 0.8 m, θ1 can take a value between 22 and 27 degrees, and θ2 can take a value between (θ1 + 5) and (θ1 + 10) degrees. Formulate the shape optimization problem of the reentry capsule.

The total surface area and volume of the capsule are computed using the equations

A R R R R R l

R R

n

B

= − + + − +

+ +

2 ² 1 1 1 2 2 1 2

12

2

π θ π

π

( sin ) ( ) ( )

( ) (RR_B−R2 2+ +l R_B

22 2

) π (1.17)

V R

R R

l R R

= n −

(

+ −

)

+ +

π θ θ

π

( sin )

( sin ) 1

6 3 1

1 3

1 12

12

1 2

1 12 222

1 2 2 2

22 2

1 + 3

(

^{R R}

)

⁺ π^{l R}

(

B⁺^R ⁺^{R R}B

)

(1.18)

Flare Conical body

Spherical nose

θ1

θ₂ l₂ l₁ R

FIGURE 1.3 Reentry capsule.

(26)

where

R₁ = R_n cos θ1 (1.19)

R₂ = R_n cos θ1 + l₁ tan θ1 (1.20) RB = R₂ + l₂ tan θ2 (1.21) The optimization problem can now be written as

Minimize

2 ² 1 1 1 2 2 1

2 12

2

π θ π

π

R R R R R l

R R R

n

B B

( sin ) ( )

( )

− + +

(

−

)

⁺

+ +

(

−RR₂

)

²^{+ + π}l₂² R_B² ^(1.22) subject to

πR θ θ π

R R l R R

n( sin )

( sin ) 1

6 3 1 1

3

1 12

12

1 2

1 12 22

−

(

+ −

)

⁺ ⁺ ⁺^R^{R R}

l R_B R R R_B

1 2

2 2

22 2

1

3 1

( )

+ ^π

(

+ +

)

^≥ ^(1.23)

0.4 < R_n < 0.6 (1.24) 22 < θ1 < 27 (1.25) θ1 + 5 < θ2 < θ1 + 10 (1.26) 0.4 < l₁ < 0.8 (1.27) 0.4 < l2 < 0.8 (1.28)

Example 1.4

It is required to find the optimum diameter (d) of a solid steel shaft whose mass (M) is to be minimized and the first cantilever frequency has to be greater than 20 Hz. Formulate this as an optimization problem by writing down the objective function and the constraint.

If L is the length of the rod (Figure 1.4) and ρ is its density, then the mass of the rod is given by

M= πd Lρ 4

2 (1.29)

(27)

11 Introduction

For this problem, L = 1 m and ρ = 7800 kg/m³. The first cantilever frequency is given by

f L

EI

1 2 k

1 2

3 5156

= π

. (1.30)

where E is Young’s modulus of steel and its value is 2 × 10¹¹ N/m². The variable k is mass per unit length. The moment of inertia I for the rod is given by

I= π d 64

4 (1.31)

The optimization problem can be written as follows.

Minimize

π ρ

4

d L2 (1.32)

subject to

1 2

3 5156

2 20 π

. L

EI

k ≥ (1.33)

1.4 Solution with the Graphical Method

Having formulated the optimization problems in the previous section, it is tempting for readers to get solutions for these problems. The graphical method is a simple technique for locating the optimal solution for problems with up to two to three design variables. Beyond three variables, the rep- resentation of the optimization problem through graphs becomes complex.

L

d

FIGURE 1.4 Cantilever rod.

(28)

The optimization problem mentioned in Example 1.2 requires two design variables, x₁ and x₂, to be evaluated such that it minimizes the total surface area and at the same time satisfying the equality constraint. A MATLAB^® code, graph_examp12.m, given at the end of this book, is used for drawing the graph for this problem. For a quick introduction to MATLAB, see Appendix A.

The variable x₁ is varied from 1 to 100 mm and the variable x₂ is varied from 1 to 200 mm in the code. The surface area is calculated based on the val- ues x₁ and x₂ and contour of the objective function is plotted (Figure 1.5) for different values of x₁ and x₂. The constraint function is then plotted (marked with *). Because this is an equality constraint optimization problem, the minimum value of the objective function is the contour curve that touches the constraint curve and has the lowest value. The minimum value of the objective function is 26,436 mm² corresponding to design variables x₁ as 37.45 mm and x₂ as 74.9 mm. Note that the length of the can is two times its radius at the minimum point. This can be proved analytically using elementary calculus.

Similarly, the optimization problem mentioned in Example 1.4 has only one design variable, the diameter d of the rod. Again, we can use a graphical method to solve this problem. A MATLAB code, graph_examp14.m, is written for this problem.

On executing the code, the output is in the form of a graph or a plot as shown in Figure 1.6. The value of the objective function (along the y-axis) decreases with the reduction in the value of the design variable (along the x-axis). However, the constraint value (also plotted along the y-axis) also decreases with the reduction in the value of the design variable. In the optimization problem, it is given that the constraint should have a value that is equal to or greater than 20 Hz. Hence the optimum solution corresponds to

90,000 70,000 26,436

15,000

50,000 15,000

50,000 70,000 90,000

15,000 50,000 70,000

26,436

90000

x₁ (mm) x2 (mm)

10 20 30 40 50 60 70 80 90 100

20 40 60 80 100 120 140 160 180

200 Objective function

Constraint 26,436

FIGURE 1.5

Function contours for the optimization problem in Example 1.2.

(29)

13 Introduction

d = 0.0283 m, where the value of the objective function mass is 4.94 kg and the constraint value is 20 Hz.

1.5 Convexity

Consider two design points, x₁ and x₂, that belong to a set S. If the line join- ing these two points is also within the set S, then the set S is a convex set. If the line joining the design points x₁ and x₂ does not belong to the set S, then the set S is a nonconvex set. See Figure 1.7 for convex and nonconvex sets. In optimization, often we have to check a function for its convexity. Consider a single variable function f(x) as shown in Figure 1.8 and two points x₁ and x₂ at which the value of the function is f(x₁) and f(x₂) respectively. Consider any point x on the line joining x₁ and x₂. If f x

( )

 is less than the value of the func- tion at the corresponding point x` on the line joining f(x₁) and f(x₂) then f(x) is a convex function, that is, for convexity

f x

( )

 ≤ f x( )` (1.34) Examples of convex functions are x², e^x, etc. If f(x) is a convex function then e^f(x) is also a convex function. Hence, e^x² and e^e^x are also convex functions.

Let us plot (Figure 1.9) these functions in MATLAB (convexity.m) to show that these functions are indeed convex.

0 0.01 0.02 0.03 0.04 0.05

0 5 10 15 20 25 30 35 40

d (m)

Objective function (kg), constraint (Hz)

Objective function Constraint

(0.0283, 4.94)

FIGURE 1.6

Objective function and constraint plot for the problem in Example 1.4.

(30)

The concept of convexity is important in declaring that a function has one minimum only. A convex function thus has a global minimum. If a function is nonconvex, the optimum reached might be a local one (see Figure 1.10).

Such functions with more than one minimum or maximum are referred to as multimodal functions. Traditional gradient-based algorithms have difficulty in locating a global optimum solution. In addition, a designer often has to look for an alternative solution to a global optimum because of the pres- ence of the constraints. For example, at a global optimum solution, the design variables may be such that it might be difficult to manufacture the product or the particular material might be very costly. The task of the designer is thus difficult. He not only has to find a global optimum solution, but also locate local optimum solutions.

x₂ S

S Convex set

Nonconvex set x₁

x1

x₂

FIGURE 1.7

Convex and nonconvex sets.

x₁ x^~ x₂

~

f (x₂)

f (x₁) f (x`) f (x)

FIGURE 1.8 Convex function.

(31)

15 Introduction

If f(x) is a convex function, then −f(x) is a concave function. Similarly, if f(x) is a concave function, then −f(x) is a convex function. Figure 1.11 shows both convex and concave functions for y = e^x.

Typically, optimization algorithms are developed to minimize the objective function. As discussed earlier, convexity plays an important role for functions where their minima are to be located. However, there can be opti- mization problems where one needs to maximize the objective function f(x).

The maximization problem can be converted to the minimization type by changing the sign of the objective function to −f(x). Mathematically,

–2 0 2 –2 0 2

0 2 4

0 5 10

0 1000 2000

0 20 40 60

y = x² y = e ^x

y = e ^x² y y = e ^e^x

y

y y

x x

FIGURE 1.9

Examples of convex functions.

Local optimum

Global optimum

x₁ x₂

FIGURE 1.10

Local and global optima for a nonconvex function.

(32)

Maximize

f(x) is the same as

Minimize

−f(x)

1.6 Gradient Vector, Directional Derivative, and Hessian Matrix

The derivative or gradient of a function f(x) at a point x, generally denoted by f′(x), is the slope of the tangent (see Figure 1.12) at that point. That is,

f′(x) = tan θ (1.35)

where θ is the angle measured by the tangent with respect to the horizon- tal. Along the gradient direction, there is the maximum change in the value of the function. Thus, gradient information provides the necessary search direction to locate the maximum or minimum of the function.

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−8

−6

−4

−2 0 2 4 6 8

x y

Convex

Concave y = e ^x

y = −e ^x

FIGURE 1.11

Concave and convex functions.

(33)

17 Introduction

In most optimization problems, which are generally nonlinear, f′(x) has to be evaluated numerically. We can use forward difference, backward difference, and central difference methods to find the derivative of a function at a point. If the value of a function f(x) is known at a point x, then the value of the func- tion at its neighboring point x + Δx can be computed using Taylor’s series as

f x x f x xf x x

f x x

f x

( ) ( ) ( )

! ( )

! ( ) +∆ = +∆ ′ +∆ ² ′′ +∆ ³ ′′′ +

2 3  (1.36)

Rearranging Equation 1.36 gives f x x f x

x f x x

f x x

( ) ( ) ( ) f x

! ( )

+∆ − = ′ + ′′ + ′′′ +

∆

∆ ∆

2 3

2  (1.37)

The forward difference formula for evaluating the derivative of a function can be written as

′ = + − +

f x f x x f x

x O x

( ) ( ∆ ) ( ) ( )

∆ ∆ (1.38)

The quantity O(Δx) represents that this formula is first-order accurate. In a similar fashion, the backward difference formula can be written as

′ = − − +

f x f x f x x

x O x

( ) ( ) ( )

∆ ( )

∆ ∆ (1.39)

Tangent

f (x) x

0 θ

FIGURE 1.12

Concept of derivative.

(34)

Using the forward and backward difference formulas, one can derive the central difference formula as

′ = + − − +

f x f x x f x x

x O x

( ) ( ∆ ) ( ∆ ) ( )

∆ ∆

2

2 (1.40)

Because the central difference formula for computing the derivative of a function is of second order, it is more accurate than forward/backward difference method. Again, the second derivative can be evaluated using the equation

′′ = + − + −

f x f x x f x f x x

( ) ( ∆ ) x( ) ( ∆ )

∆ 2

2 (1.41)

Let us take a function

f(x) = 2 sin 5x + 3x³ − 2x² + 3x − 5 (1.42) and compute the first and second derivatives using the central difference formula for x ranging from 0.1 to 1.0 with Δx as 0.01. A MATLAB code, derivative.m, is written and the output is plotted in Figure 1.13.

The top plot in the Figure 1.13 is f(x) varying with x. Note that the function has one maximum and one minimum and these points are shown with *.

The derivative of the function is plotted in the second plot. Note that f′(x) = 0 at the maximum and minimum of the function. From the third plot, observe

0 0.2 0.4 0.6 0.8 1

–4 –3 –2

f (x)

–20

–100 0 20

f ´(x)

0 100

x

f ˝(x)

FIGURE 1.13

Plot of a function with its first and second derivative.

(35)

19 Introduction

that f″(x) ≥ 0 at the minimum and f″(x) ≤ 0 at the maximum of the function.

The second derivative provides curvature information of the function.

For certain functions such as f(x) = x³, both f′(x) = f″(x) = 0 at x* = 0. In such instances, one has to look for higher order derivatives. Here f‴(x) = 6, which is nonzero. If the first nonzero higher order derivative is denoted by n, then x* is an inflection point (or a saddle point) if n is odd and x* is local optimum if n is even. Therefore, x* is an inflection point for the function f(x) = x³, as the first nonzero higher order derivative is odd (third derivative). Similarly, it can be shown that the function f(x) = x⁴ has a local minimum at x* = 0. These two functions are plotted in Figure 1.14.

So far we considered the derivative of a function with one variable only.

The gradient is a vector that contains partial derivatives of the function with respect to the design variables (x1, x2, ⋯, x_n) and is mathematically written as

∇ =

∂













 f

f x f x f xn

1

2



(1.43)

Let us plot a tangent and gradient at a given point (x₁, x₂) on the function contours for Example 1.2. For a single-variable case, we observed that the tangent at any point for a function and its gradient are the same (Figure 1.12).

–2 0 2

–8 –6 –4 –2 0 2 4 6 8

0 2 4 6 8 10 12 14 16

y y

x –2 0 2

x

y = x ³ y = x ⁴

FIGURE 1.14

Saddle point and local minimum functions.

OPTIMIZATION Algorithms and Applications

OPTIMIZATION

Algorithms and

Applications

OPTIMIZATION

Algorithms and Applications

Rajesh Kumar Arora

Senior Engineer

Vikram Sarabhai Space Centre

Indian Space Research Organization

Trivandrum, India

Contents

Preface

Author

1

Introduction

(

)

(

)

(

)

(

)

(

)

(

)

(

)

( )

(

)

( )

( )