Optimization
Didier Auroux (auroux@unice.fr) Polytech’Nice-Sophia
Contents
1 Introduction to optimization 3
1.1 Notations and definitions . . . 3
1.2 Examples . . . 3
1.2.1 Example 1: linear problem . . . 3
1.2.2 Example 2: least squares problem . . . 4
1.2.3 Calculus of variations . . . 4
1.2.4 Example 3: optimal trajectory . . . 4
1.2.5 Example 4: geodesic . . . 5
1.2.6 Other examples . . . 5
2 Optimality conditions 7 2.1 Fr´echet and Gˆateaux differentiability . . . 7
2.2 Convexity and Gˆateaux differentiability . . . 9
2.3 Optimality conditions . . . 11
2.3.1 First-order necessary optimality conditions . . . 11
2.3.2 Convex case: sufficient optimality conditions . . . 12
2.3.3 Second-order optimality conditions . . . 13
3 Existence of a minimum 15 3.1 Existence of minima in finite dimension . . . 15
3.2 Infinite dimension . . . 15
3.2.1 Convex analysis: existence of a minimum in a Hilbert space . . . 15
3.2.2 Application to quadratic functions: . . . 15
3.2.3 Weak convergence . . . 16
3.2.4 Lower semicontinuous functions and epigraphs . . . 16
3.2.5 Proof of the existence of a minimum in a Hilbert space . . . 17
4 Lagrange multipliers and duality for constrained optimization 19 4.1 Minimization under equality constraints . . . 19
4.2 Saddle points . . . 19
4.2.1 Definition . . . 19
4.2.2 Inequality constraints . . . 20
4.3 Convex optimization: the Kuhn-Tucker theorem . . . 20
4.4 Duality . . . 21
1
5 Optimization algorithms 23
5.1 Unconstrained optimization . . . 23
5.1.1 Gradient algorithm with fixed step . . . 23
5.1.2 Gradient algorithm with optimal step . . . 24
5.1.3 Conjugate gradient . . . 25
5.2 Constrained optimization . . . 26
5.2.1 Gradient algorithm with projection . . . 26
5.2.2 Identification of a saddle-point: Uzawa’s algorithm . . . 27
2
