RESEARCH OUTPUTS / RÉSULTATS DE RECHERCHE
Author(s) - Auteur(s) :
Publication date - Date de publication :
Permanent link - Permalien :
Rights / License - Licence de droit d’auteur :
Bibliothèque Universitaire Moretus Plantin
Institutional Repository - Research Portal
Dépôt Institutionnel - Portail de la Recherche
researchportal.unamur.be
University of Namur
On the complexity of finding first-order critical points in constrained nonlinear
optimization
Cartis, Coralia; Gould, Nicholas I M; Toint, Philippe L.
Published in: Mathematical Programming DOI: 10.1007/s10107-012-0617-9 Publication date: 2014 Document Version
Early version, also known as pre-print Link to publication
Citation for pulished version (HARVARD):
Cartis, C, Gould, NIM & Toint, PL 2014, 'On the complexity of finding first-order critical points in constrained nonlinear optimization', Mathematical Programming, vol. 144, no. 1-2, pp. 93-106.
https://doi.org/10.1007/s10107-012-0617-9
General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain
• You may freely distribute the URL identifying the publication in the public portal ?
Take down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.
On the complexity of finding first-order critical points in constrained nonlinear optimization
by C. Cartis, N. I. M. Gould and Ph. L. Toint Report NAXYS-13-2011 13 April 2011
University of Namur
61, rue de Bruxelles, B5000 Namur (Belgium)
http://www.fundp.ac.be/sciences/naxys
On the complexity of finding first-order critical points in
constrained nonlinear optimization
C. Cartis
∗, N. I. M. Gould
†and Ph. L. Toint
‡13 April 2011
Abstract
The complexity of finding ǫ-approximate first-order critical points for the general smooth con-strained optimization problem is shown to be no worse that O(ǫ−2) in terms of function and con-straints evaluations. This result is obtained by analyzing the worst-case behaviour of a first-order short-step homotopy algorithm consisting of a feasibility phase followed by an optimization phase, and requires minimal assumptions on the objective function. Since a bound of the same order is known to be valid for the unconstrained case, this leads to the conclusion that the presence of possibly nonlinear/nonconvex inequality/equality constraints is irrelevant for this bound to apply.
Keywords: evaluation complexity, worst-case analysis, constrained nonlinear optimization.
1
Introduction
Evaluation complexity analysis for nonconvex smooth optimization problems has recently been a very ac-tive area of research and has covered both standard methods for the unconstrained case, such as steepest-descent (see Vavasis, 1993, Nesterov, 2004, Cartis, Gould and Toint, 2010b), trust-region methods (see Gratton, Sartenaer and Toint, 2008), Newton’s algorithm (see Cartis et al., 2010b) or finite-difference and derivative-free approaches (see Vicente, 2010, Cartis, Gould and Toint, 2010c), along with newer ap-proaches involving regularization (see Nesterov and Polyak, 2006, Cartis, Gould and Toint, 2010a). The issue considered in this paper is to bound the number of objective function (and gradient) evaluations that are necessary to find an approximate first-order critical point for the problem
minimize f(x) such that x∈ IRn
,
where f is a continuously differentiable possibly nonconvex function from IRn to IR with Lipschitz con-tinuous gradient. Such an approximate critical point is defined as a point x such that
kg(x)k ≤ ǫ, (1.1) where ǫ ∈ (0, 1) is a user-specified accuracy, k · k is the Euclidean norm and g(x) def= ∇xf(x). For
first-order methods, i.e., for steepest-descent and trust-region algorithm with linear models, it has been shown that this maximum number of objective function (and gradient) evaluations is bounded above by
lκ ǫ2
m
(1.2) for some constant κ > 0 independent of n and ǫ (Nesterov, 2004, Gratton et al., 2008). Moreover, Cartis et al. (2010b) proved that this order in ǫ is sharp. A first extension of this kind of results to constrained
∗School of Mathematics, University of Edinburgh, The King’s Buildings, Edinburgh, EH9 3JZ, Scotland, UK. Email:
coralia.cartis@ed.ac.uk
†Computational Science and Engineering Department, Rutherford Appleton Laboratory, Chilton, Oxfordshire,
OX11 0QX, England, UK. Email: nick.gould@sftc.ac.uk
‡Namur Center for Complex Systems (naXys) and Department of Mathematics, FUNDP-University of Namur, 61, rue
de Bruxelles, B-5000 Namur, Belgium. Email: philippe.toint@fundp.ac.be
Cartis, Gould, Toint: Complexity of constrained nonlinear optimization 2 problems was provided by Cartis, Gould and Toint (2009), where it is shown that (1.2) also holds for a first-order projection-based method for the more general problem
minimize f(x)
such that x∈ C, (1.3) where C is a convex set and where (1.1) is suitably adapted to define an ǫ-approximate first-order critical point for constrained problem (1.3). Alternatively, Cartis, Gould and Toint (2011b) considered a first-order exact penalty function algorithm for solving the completely general nonlinearly constrained nonconvex optimization problem
minimize f(x) such that cE(x) = 0,
and cI(x) ≥ 0,
(1.4) where cE and cI are continuously differentiable functions from IRn to IRm and IRp, respectively, having
Lipschitz continuous Jacobians. They proved that the complexity of finding an ǫ-approximate first-order critical point for (2.1) is given by an appropriate variant of (1.2) when the penalty parameters remain finite and is bounded above by O(ǫ−4) otherwise. Unfortunately, this last result requires the undesirable
assumption that the objective function f is bounded below on the whole of IRn.
In this paper, we provide a theoretical approach to the same problem and improve the results of Cartis et al. (2011b) by showing that the complexity of achieving ǫ-approximate first-order criticality for (1.4) is bounded by (1.2) for a first-order algorithm. Morever, this stronger result only require f to be bounded in an ǫ-neighbourhood of the feasible set, which is considerably weaker than assuming this property on the whole space.
The paper is organized as follows. The SSSD Algorithm is introduced in Section 2 for approximately solving the equality constrained problem and its complexity is shown in Section 3 to be bounded above by (1.2). Section 4 briefly covers the simple extension of this result to the general problem (1.4). Some conclusions and perspectives are finally proposed in Section 5.
2
The SSSD Algorithm for the equality constrained problem
For the sake of simplicity, we start by considering the equality constrained problem minimize f(x)
such that c(x) = 0, (2.1) where c is a continuously differentiable function from IRn to IRm with Lipschitz continuous Jacobian. The algorithm we now describe consists of two phases. In the first, a first-order algorithm is applied to minimize kc(x)k (independently of the objective function f ), resulting in a point which is either (approximately) feasible, or is an approximate infeasible stationary point of kc(x)k. This last outcome is not desirable if one wishes to solve (2.1), but cannot be avoided by any algorithm not relying on global minimization. If an (approximate) feasible point has been found, Phase 2 of the algorithm then performs short steps along generalized steepest-descent directions so long as first-order criticality is not satisfied. These steps are computed by attempting to preserve feasibility of the iterates while producing values of the objective function that are close to a sequence of decreasing “targets”.
Both phases rely on the first-order trust-region algorithm(1)proposed in Cartis et al. (2011b), which
can be used to solve the problem
minimize θ u(x) such that x∈ IRn
, (2.2)
where θ is a (potentially nonsmooth) convex and globally Lipschitz function from IRpinto IR and u(x) is
a continuously differentiable function from IRninto IRp with Lipschitz continuous Jacobian A(x). In this
(1)We make this choice for simplicity of exposition, but other methods can be considered with similar results. In particular,
the quadratic regularization technique of Cartis et al. (2011b) or the trust-region technique proposed by Byrd, Gould, Nocedal and Waltz (2005) are also adequate.
Cartis, Gould, Toint: Complexity of constrained nonlinear optimization 3 algorithm, a “Cauchy step” sk is obtained from the iterate xk by solving the linearized model problem
minimize θ u(xk) + A(xk)s
such that ksk ≤ ∆k, (2.3)
where ∆k is a trust-region radius. Because θ is convex and its argument in (2.3) linear, this
prob-lem is computationally tractable. The rest of the algorithm specification follows standard trust-region technology.
We now return to the solution of problem (2.1) proper, and define the merit function
φ(x, t)def= kc(x)k + |f (x) − t|, (2.4) where t is meant as a “target” for f (x). We also define the local linearizations of kc(x)k and φ(x, t) given by ℓc(x, s) def = kc(x) + J(x)sk and ℓφ(x, t, s) def = ℓc(x, s) + |f (x) + hg(x), si|,
(where h·, ·i is the Euclidean inner product). The value of the decrease of the linearized model in a ball of unit radius may then be considered as a first-order criticality measure for the problems of minimizing kc(x)k and φ(x, t), yielding the measures
ψ(x)def= ℓc(x, 0) − min
kdk≤1ℓc(x, d) and χ(x, t) def
= ℓφ(x, t, 0) − min
kdk≤1ℓφ(x, t, d).
Note that ψ(x) is zero if and only if x is first-order critical for the problem of minimizing kc(x)k, while χ(x, t) is zero if and only if (x, t) is a first-order critical point for the problem
minimize φ(x, t) such that x∈ IRn
, (2.5)
(t fixed). In Phase 1 of the SSSD algorithm (aiming for feasiblility), we apply the first-order trust-region algorithm of Cartis et al. (2011b) by identifying
p= m, u(x) = c(x), and θ(·) = k · k, (2.6) in (2.2), yieding θ u(y) = kc(x)k. For Phase 2 (the optimality phase), we choose in (2.2), for t fixed,
p= m + 1, u(x) = (c(x), f (x) − t) and θ(·) = k · k + | · |, (2.7) which gives θ u(x) = φ(x, t). Note that θ(·) is clearly convex with global Lipschitz constant equal to one in both cases.
We are now ready to formalize our Short Step Steepest-Descent (SSSD) Algorithm as presented on the following page.
Since the SSSD algorithm makes no pretense of being practical, we have written Steps 2.2.b and 2.2.c by only using the constants
0 < η < 1, and 0 < γ < 1,
instead of the more usual η1≤ η2and γ1≤ γ2, a simplified choice which is allowed in the standard
trust-region case, including that studied in Cartis et al. (2011b)(2). Note that the SSSD algorithm requires
one evaluation of the objective function and its gradient and one evaluation of the constraint’s function and its Jacobian per iteration.
Note also that one could also consider using the ARC algorithm (see Cartis, Gould and Toint, 2011a) to minimize kc(x)k2 to find x
1 such that kJ(x1)Tc(x1)k ≤ ǫ. We do not consider this (potentially more
efficient) possibility here because it would require stronger assumptions on the constraint function c.
(2)By selecting η
Cartis, Gould, Toint: Complexity of constrained nonlinear optimization 4
Algorithm 2.1: The SSSD algorithm
Let κf ∈ (0, 1) and ∆1>0 be given, together with a starting point x0.
Phase 1:
Starting from x0, minimize kc(x)k (using (2.2) and (2.6) and the trust-region method of Cartis
et al., 2011b) until a point x1 is found such that
ψ(x1) ≤ ǫ.
If kc(x1)k > κfǫ, terminate [locally infeasible].
Phase 2:
1. Set t1= kc(x1)k + f (x1) − ǫ and k = 1.
2. While χ(xk, tk) ≥ ǫ,
(a) Compute a first-order step sk by solving
minimize ℓφ(xk, tk, s)
such that ksk ≤ ∆k, (2.8)
(b) Compute φ(xk+ sk, tk) and define
ρk = φ(xk, tk) − φ(xk+ sk, tk) ℓφ(xk, tk,0) − ℓφ(xk, tk, sk) . (2.9) If ρk≥ η, then xk+1= xk+ sk; else xk+1= xk. (c) Set ∆k+1= ∆k if ρk ≥ η [k successful] γ∆k if ρk < η, [k unsuccessful] (2.10) (d) If ρk≥ η, set tk+1= tk− φ(xk, tk) + φ(xk+1, tk) if f (xk+1) ≥ tk, 2f (xk+1) − tk− φ(xk, tk) + φ(xk+1, tk) if f (xk+1) < tk. (2.11) Otherwise, set tk+1= tk.
(e) Increment k by one and return to 2. 3. Terminate [(approximately) first-order critical]
3
Complexity of the SSSD Algorithm for the equality
constrained problem
Before analyzing the complexity of Algorithm SSSD, we state our assumptions formally.
A.1: The function c is continuously differentiable on IRn and f is continuously differentiable in an open neighbourhood of
Cǫ= {x ∈ IRn | kc(x)k ≤ ǫ}.
A.2: J(x) is globally Lipschitz continuous in IRn with Lipschitz constant bounded above by LJ >0,
and g(x) is Lispchitz continuous in Cǫ with Lipschitz constant bounded above by Lg>0.
A.3: The objective function is bounded above and below in the neighbourhood of the feasible set, that is there exist constants flowand fup≥ flow− 1 such that
Cartis, Gould, Toint: Complexity of constrained nonlinear optimization 5 We start our analysis by exploiting the results of Cartis et al. (2011b) and bounding the number of Phase 1 iterations.
Lemma 3.1 Suppose that A.1 and A.2 hold. Then, at most l kc(x0)k κ1 ǫ2 m (3.12) evaluations of c(x) and its derivatives are needed to complete Phase 1, for some κ1 >0 independent of
n, ǫ and x0.
Proof. See Theorem 2.4 in Cartis et al. (2011b). 2 We next extract from the same reference a property which is crucial for Phase 2.
Lemma 3.2 Suppose that A.1 and A.2 hold. Suppose also that χ(xk, tk) ≥ ǫ and that
∆k ≤
(1 − η)ǫ Lg+12LJ
. (3.13)
Then iteration k is successful and
φ(xk+ sk, tk) ≤ φ(xk, tk) − κCǫ 2 , (3.14) where κC def = η min ∆1, (1 − η)γ Lg+12LJ . (3.15)
is a constant independent of n and ǫ.
Proof. This follows by applying Lemmas 2.1 and 2.3 in Cartis et al. (2011b) to the objective φ(x, tk)
considered as a function of x only. 2 We now bound the total number of unsuccessful iterations in the course of Phase 2.
Lemma 3.3 There are at most O(| log ǫ|) unsuccessful iterations in Phase 2 of the SSSD algorithm. Proof. Note that (2.10) implies that the trust-region radius is never increased, and therefore Lemma 3.2 guarantees that all iterations must be successful once ∆1 has been reduced (by a factor γ)
enough times to ensure (3.13). Hence there are at most 1
| log γ||log ǫ + log(1 − η) − log ∆1− log(Lg+
1 2LJ)|
= O(| log ǫ|) (3.16) unsuccessful iterations during the complete execution of the Phase 2. 2 The next lemma proves the crucial observation that all Phase 2 iterates remain (approximately) feasible, and that the targets tk decrease by a quantity bounded below by a multiple of ǫ−2 at every successful
iteration.
Lemma 3.4 For every k ≥ 1, we have that
f(xk+1) − tk+1>0, (3.17)
φ(xk, tk) = ǫ (3.18)
kc(xk)k ≤ ǫ, (3.19)
|f (xk) − tk| ≤ ǫ. (3.20)
Moreover, if iteration k is successful, then
tk− tk+1≥ κCǫ
2 (3.21)
Cartis, Gould, Toint: Complexity of constrained nonlinear optimization 6 Proof. We start by observing that (3.17) immediately follows from (2.11) and (3.14) and the fact that f (xk) and tkremain unchanged on unsuccessful iterations. We now prove (3.18) by induction on k.
We first note that this inequality holds by construcion for k = 1. We now show that this bound remains valid for k > 1. Assume that iteration k is successful and that
φ(xk, tk) = ǫ. (3.22)
Using (2.4) and (3.17), we observe that
φ(xk+1, tk+1) = kc(xk+1)k + f (xk+1) − tk+ (tk− tk+1). (3.23)
Consider the case where f (xk+1) ≥ tk first. Then, using (3.23) and (2.11), we obtain that
φ(xk+1, tk+1) = φ(xk+1, tk) + φ(xk, tk) − φ(xk+1, tk) = φ(xk, tk).
If f (xk+1) < tk, we have that
φ(xk+1, tk+1) = kc(xk+1)k − f (xk+1) + tk+ φ(xk, tk) − φ(xk+1, tk)
= φ(xk+1, tk) + φ(xk, tk) − φ(xk+1, tk)
= φ(xk, tk),
where we again used (3.23) and (2.11). Combining the two cases and using (3.14) and (3.22), we then deduce that
φ(xk+1, tk+1) = φ(xk, tk) = ǫ.
By induction, and since tk and f (xk) (and hence φ) are unmodified at unsuccessul iterations, (3.18)
therefore holds for all k ≥ 1. Relations (3.19) and (3.20) immediately follow from (2.4). Finally, (3.21) results from (2.11) and (3.14) at successful iterations. 2 Lemma 3.5 Assume that kc(xk)k ≤ ǫ and χ(xk, tk) ≤ ǫ. Then xk is an approximate critical point in
the sense that
kc(xk)k ≤ ǫ and kJ(xk)Ty− g(xk)k ≤ ǫ (3.24)
for some vector of multipliers y ∈ IRm. Similarly, assume that ψ(x) ≤ ǫ. Then
kJ(x)Tzk ≤ ǫ (3.25) for some vector of multipliers z ∈ IRm.
Proof. See Theorem 3.1 in Cartis et al. (2011b) and the comments thereafter. 2 Theorem 3.6 Assume A.1-A.3 hold. Then the SSSD algorithm generates an ǫ-first-oder critical point for problem (2.1), that is an iterate xk satisfying either
(3.24) or (3.25) with kc(xk)k > κfǫ, in at most kc(x0)k + fup− flow κ2 ǫ2 (3.26) evaluations of c and f (and their derivatives), where κ2>0 is a constant independent of n, ǫ and x0.
Proof. We have seen in Lemma 3.1 that the complexity of obtaining x1 is bounded above by
O(⌈kc(x0)kǫ−2⌉). Thus, as ψ(x1) ≤ ǫ, Lemma 3.5 ensures that (3.25) holds. If the algorithm terminates
at this stage, then both (3.25) and kc(xk)k > κfǫhold, as requested. Assume now that Phase 2 of the
algorithm is entered. We then observe that Lemma 3.2 implies that successful iterations must happen as long as χ(xk, tk) ≥ ǫ. Moreover, we have that
flow≤ f (xk) ≤ tk+ ǫ ≤ t1− σkκCǫ
2+ ǫ ≤ f (x
1) − σkκCǫ 2+ ǫ
Cartis, Gould, Toint: Complexity of constrained nonlinear optimization 7 where σk is the number of these successful iterations from iterations 1 to k of Phase 2, and where we
have successively used A.3, (3.20) and (3.21). Hence, we obtain from the inequality f (x1) ≤ fup (itself
implied by A.3 again) that
σk≤
fup− flow+ ǫ
κCǫ2
. (3.27)
The number of Phase 2 iterations satisfying χ(xk, tk) ≥ ǫ is therefore bounded above, and the algorithm
must terminate after (3.27) such iterations at most, yielding, because of Lemma 3.5, an ǫ-first-order critical point satisfying (3.24). Remembering that only one evaluation of c and f (and their derivatives, if successful) occurs per iteration, we therefore conclude from (3.27) and Lemma 3.3 that the total number of such evaluations in Phase 2 is bounded above by
fup− flow+ ǫ
κCǫ2
+ O(| log ǫ|)
Summing this upper bound with that for the number of iterations in Phase 1 given by Lemma 3.1 then
yields (3.26). 2
4
Including general inequality constraints
If we now return to the solution of problem (1.4), we may consider defining c(x) =
cE(x)
min[ 0, cI(x) ]
in the above. The quantity kc(x)k can again be considered as the composition of a nonsmooth convex function with the smooth function (cE(x)T, cI(x)T)T and the theory developed above applies without
modification, except that Lemma 3.5 must be adapted for the presence of inequality constraints. If an inequality constraint is active at an approximate critical point, then its multiplier has to be non-negative because y ∈ ∂(k min[0, · ]k) implies that y ≥ 0. If it is inactive, then it may as well be absent from the problem (and its multiplier must be zero). Hence Lemma 3.5 generalizes to the inequality constraints case (1.4) without difficulty.
5
Conclusions
We have shown that the complexity to achieve either a ǫ-first-order critical point or an infeasible ǫ-critical point for the infeasibilities of the general smooth nonlinear optimization problem (1.4) is O(ǫ−2), where
the constant involved is independent of problem dimension. This is a marked improvement over the results presented in Cartis et al. (2011b), where the same complexity was achieved only if the penalty parameter of the minimization scheme discussed in this reference remains bounded, the complexity being O(ǫ−4) othrewise. Moreover, the results obtained in the present paper only assume boundedness of the
objective function on an ǫ-neighbourhood of the feasible set, rather than on the whole space.
Since Cartis et al. (2010b) have shown that the O(ǫ−2) bound can be effectively achieved by steepest
descent in the unconstrained case, improving the same bound in the constrained case is also impossible for methods of the same type.
We fully accept that the SSSD algorithm discussed in Section 2 is most likely to be extremely inefficient in practice, because it amounts to following the constraints manifold with very small steps. “Long steps” variants may be considered in which the setting of the target tk is more aggressively geared towards
minimizing the objective function. Whether such variants can be numerically effective remains to be seen, but their complexity will be difficult to guarantee with the kind of technique used here, as this would rely on global optimization of the constraint violation.
That we expect Algorithm SSSD to be outperformed in practice is to be welcomed, indicating that the O(ǫ−2) evaluation bound may be as pessimistic for the contrained case as it is for the unconstrained
one. But it remains remarkable that this pessimistic bound is unaffected by the presence of possibly nonlinear and nonconvex constraints.
Cartis, Gould, Toint: Complexity of constrained nonlinear optimization 8
Acknowledgements
The work of the second author is funded by EPSRC Grant EP/E053351/1. All three authors are grateful to the Royal Society for its support through the International Joint Project 14265.
References
R. H. Byrd, N. I. M. Gould, J. Nocedal, and R. A. Waltz. On the convergence of successive linear-quadratic programming algorithms. SIAM Journal on Optimization, 16(2), 471–489, 2005.
C. Cartis, N. I. M. Gould, and Ph. L. Toint. An adaptive cubic regularization algorithm for nonconvex optimization with convex constraints and its function-evaluation complexity. Technical Report 08/05R, Department of Mathematics, FUNDP - University of Namur, Namur, Belgium, 2009. C. Cartis, N. I. M. Gould, and Ph. L. Toint. Adaptive cubic overestimation methods for unconstrained
optimization. Part II: worst-case function-evaluation complexity. Mathematical Programming, Se-ries A, 2010a. DOI: 10.1007/s10107-009-0337-y.
C. Cartis, N. I. M. Gould, and Ph. L. Toint. On the complexity of steepest descent, Newton’s and regular-ized Newton’s methods for nonconvex unconstrained optimization. SIAM Journal on Optimization, 20(6), 2833–2852, 2010b.
C. Cartis, N. I. M. Gould, and Ph. L. Toint. On the oracle complexity of first-order and derivative-free algorithms for smooth nonconvex minimization. Technical Report naXys-03-2010, Namur Centre for Complex Systems (naXys), FUNDP-University of Namur, Namur, Belgium, 2010c.
C. Cartis, N. I. M. Gould, and Ph. L. Toint. Adaptive cubic overestimation methods for unconstrained optimization. Part I: motivation, convergence and numerical results. Mathematical Programming, Series A, 127(2), 245–295, 2011a.
C. Cartis, N. I. M. Gould, and Ph. L. Toint. On the evaluation complexity of composite function minimization with applications to nonconvex nonlinear programming. Technical Report naXys-06-2011, Namur Centre for Complex Systems (naXys), FUNDP-University of Namur, Namur, Belgium, 2011b.
S. Gratton, A. Sartenaer, and Ph. L. Toint. Recursive trust-region methods for multiscale nonlinear optimization. SIAM Journal on Optimization, 19(1), 414–444, 2008.
Yu. Nesterov. Introductory Lectures on Convex Optimization. Applied Optimization. Kluwer Academic Publishers, Dordrecht, The Netherlands, 2004.
Yu. Nesterov and B. T. Polyak. Cubic regularization of Newton method and its global performance. Mathematical Programming, Series A, 108(1), 177–205, 2006.
S. A. Vavasis. Black-box complexity of local minimization. SIAM Journal on Optimization, 3(1), 60–80, 1993.
L. N. Vicente. Worst case complexity of direct search. Technical report, Department of Mathematics, University of Coimbra, Coimbra, Portugal, May 2010.
