f convex on I (Definition 1 .1.1 )
f convex on 1= dom f
extension
restriction
----
f (Definition 1.3.1) E Con v R Fig.l.3.1. "Classical" and extended-valued convex functions- Let x be a real parameter and consider the simple optimization problem
inf {-y :
l
~ x}. (1.3.2)It is meaningless if x <
°
but, for x ;;:: 0, the optimal value is -..jX, a convex function of x. In view of the convention inff2l = +00, we do have a convex function in the sense of Definition 1.3.1. It is good to know that problems ofthe type (1.3.2) yield convex functions of x (this will be confirmed in Chap. IV), even though they may not be meaningful for all values ofx.- Associated to a given f is the so-called conjugate function R ;:) x ~ sup {xy - f(y) : y E R} .
Here again, the values of x for which the supremum is finite are not necessarily known beforehand. This supremum is thus an extended-valued function of x, a function which turns out to be of utmost importance.
- Suppose that a function g, convex on I, must be minimized on some nonempty subinterval eel. The constraint x E C can be included in the objective function by setting
f(x):= {g(X) if x E C,
+00 ifnot.
The resulting f is in Conv R and minimizing it (on the whole ofR) is just equivalent to the original problem.
Remark 1.3.3 The price to pay when accepting f(x)
=
+00 is alluded to in §A.2: some care must be exercised when doing algebraic manipUlations; essentially, multiplications of function-values by nonpositive numbers should be avoided whenever possible. This was done already in (1.1.1) or (1.3.1), where the requirement a E ]0, 1[ (rather than a E [0, 1]) was nottotally innocent. 0
2 First Properties
2.1 Stability Under Functional Operations
In this section, we list some of the operations which can be proved to preserve con-vexity, simply in view of the definitions themselves.
Proposition 2.1.1 Let II, ... , 1m be m convex functions and tl, ... , tm be positive numbers. If there exists Xo such that /j(xo) < +00, j = 1, ... ,m, then thefunction 1:= Ej=1 tjlj is in ConvR
PROOF. Immediate from the relation of definition (1.3.1 ). o Note the precaution above: when adding two functions II and
h,
we have to make sure that their sum is not identically +00, i.e. that dom IIn
domh
=1= 0.Proposition 2.1.2 Let {/j }jEJ be afamily of convexfunctions.Ifthere exists Xo E IR such that SUPjEJ /j (xo) < +00, then the function I := SUPjEJ /j is in Conv R PROOF. Observe that the epigraph of
I
is the intersection over J of the convex setsepi /j. 0
The minimum of two convex functions is certainly not convex in general (draw a picture). However, the inf-operation does preserve convexity in a slightly more elaborate setting.
Proposition 2.1.3 Let II and h be convex and set for all x E IR I(x) := (fl
t
h)(x) := inf{fl (XI)+
h(X2) : XI+
X2 = X}= inf {f1(Y)
+
h(x - y) : Y E 1R}.If there exist two real numbers So and ro such that
/j(x) ~ SoX - ro for j
=
1, 2 and all x E IR(2.1.1)
(in other words; the affine function x ~ sox - ro minorizes II and h), then I E
ConvR
EXPLANATION. The domain of
I
in (2.1.1) is dom/l+
domh: by construction, I(x) < +00 if XI and X2 can be found such that XI + X2=
x and II (XI) + h(X2) <+00. On the other hand, I(x) is minorized by SOX - 2ro > -00 for all x. Now, an algebraic proof of convexity, based on (1.3.1), would be cumbersome. The key is actually to realize that the strict epigraph of
I
is the sum (in 1R2) of the strict epigraphsof II and h: see Definitions 1.0.1 and 1.1.2. 0
The operation described by (2.1.1) is called the infimal convolution of II and
h.
It is admittedly complex but important and will be encountered on many occasions. Let us observe right here that it corresponds to the (admittedly simple) addition of epigraphs - barring some technicalities. It is a good exercise to visualize the infimal convolution of an arbitrary convex II and- h (x) = r ifx = 0,
+00
if not (shift epi II vertically by r);- h(x) = 0 if x = Xo,
+00
if not (horizontal shift);- h(x) = 0 if Ixl :( r,
+00
if not (horizontal smear);- h(x)
=
sx - r (it is gr h that wins);2 First Properties 11
Fig.2.1.1. The ball-pen function
-h(x)
=
1-.J1=X2forx E [-1,+1] (the "ball-pen function" of Fig. 2.1.1); translate the bottom of the ball-pen (the origin ofll~2) to each point in gr II;- h(x)
=
1/2X2 (similar operation).Remark 2.1.4 The classical (integral) convolution between two functions FI and F2 is (F]
*
F2)(X) :=1
F] (y)F2(X - y)dy for all x E JR.For nonnegative functions, we can consider the "convolution of order p" (p > 0):
(F] *p F2)(X):=
{L[F]
(y)F2(X - y)]Pdyrp for all X E JR.It is reasonable to claim that this integral converges to SUPy FI (y) F2 (x - y) when p ~ +00.
Now take Fi := exp(-Ii), i = 1,2; we have
(Fj *00 F2)(X)
=
supe-!I(Y)-!2(x-y)=
e-infY[!I(Y)+ !2(x-y)].Y
Thus, the infimal convolution appears as a "convolution of infinite order", combined with an exponentiation.
2.2 Limits of Convex Functions
Proposition 2.2.1 Consider a sequence Uk} kEN of functions in Conv R Assume that, when k ~ +00, Uk} converges pointwise (in JR U {+oo}) to a function
f :
JR ~ IR U {+oo} which is not identically +00. Thenf
E Conv RPROOF. Apply (1.3.1) to fk and let k ~ +00.
o
Remark 2.2.2 The interval dom!k may depend on k. Special attention is often paid to the behaviour of !k on some fixed interval 1 contained in dom
ik
for all k. If, in addition, 1 is contained in the domain of the limit-functionI,
then a stronger result can be proved: the convergence of !k toi
is uniform on any compact subinterval of int I. 0It is usual in analysis to approximate a given function
f
by a sequence of more"regular" functions fk. In the presence of convexity, we give two examples of regu-larization, based on the convolution operation.
Our first example is classical. Choose a "kernel function" K : lR --+ lR+, which is continuous, vanishes outside some compact interval, and is such that
J
R K (y )dy = 1;define for positive integer k the function
lR:3 Y t-+ Kk(y) := kK(ky).
Given a function
f
(to simplify notation, suppose domf
= 1R), the convolution A{x) := f*
Kk =fa
f{x - y)Kk{y)dy (2.2.1) is an approximation off,
and its smoothness properties just depend on those of K.If K E Coo{lR), a COO regularization is obtained; such is the case for example with
1 .
K{y) :=cexp-2-- forlyl < 1 (0 outsIde) , y -1
where c > 0 is chosen so that K has integral 1.
Proposition 2.2.3 Let {Kk}kEN be a sequence of COO kernel-functions as defined above, and let f : lR --+ IR be convex. Then
A
of (2.2.1) is a Coo convex function on lR, and {A} converges to f uniformly on any compact subset ofRPROOF. The convexity of
A
comes immediately from the analyti<:al definition (1.1.1 ).The Coo-property of fk and the convergence result of {A} to f are classical in real
analysis. 0
Another type of regularization uses the infimal convolution with a convex kernel K k. It plays an important role in convex analysis and optimization, for both theoretical and algorithmic aspects. We give two examples of kernel functions:
Kk{Y) := 4kl and Kk(y):= klyl ,
which have the following effects (the proofs are omitted and will be given later in
§XY.4.1 and §XI.3.4):
Proposition 2.2.4 Let f E Conv R For all positive k, define
f(k){X) := inf {J(y)
+
4k(x - y)2 : y E IR} ; (2.2.2) then:(i) f(k) is convexfrom IR to IR and f(k) (x) ~ f(k+I)(X) ~ f(x)for all x E lR;
(ii) ifxo minimizes f on
R
it also minimizes f(k) and then f(k)(XO) = f(xo); the converse is true whenever Xo is in the interior of dom f;(iii) f(k) is differentiable and its derivative is Lipschitz-continuous:
IfCk)(Xl) - f Ck)(X2)1 ~ klxl - x21 for all (Xl, X2) E IR x IR;
2 First Properties 13 (iv) except possibly on the boundary of dom I. {f(k)} converges pointwise to I when
k -+
+00.
For k large enough. define
l[k](X) := inf {fey)
+
klx - yl : y E JR.} ; (2.2.3) then:(j) I[k] is convexfrom JR. to JR. and I[k] (x) ~ l[k+l](X) ~ I(x) for all x E
R
(jj) ifxo E intdom/. then l[k](XO)
=
I(xo)fork large enough;(jjj) I[k] is Lipschitz-continuous:
I/[k](Xl) - l[k](X2)1 ~ klxl - x21 for all (XI. X2) E JR. x JR.. 0
f(k) (x)
x=11k
Fig. 2.2.1. Moreau-Yosida C1 regularizations
Replacing I by I(k) of (2.2.2) is called Moreau-Yosida regularization. It yields C 1 -smoothness, without essentially changing the set of minimizers; note also that the function to be minimized in (2.2.2) is strictly convex (and even better: so-called strongly convex). It is not too difficult to work out the calculations when I(x)
=
Ix I: the result is the function of (1.1.4), illustrated on Fig. 2.2.1. It has a continuous derivative, as claimed in (iii), but no second derivative at x
=
±l/k. The right part of the picture shows the effect of the same regularization on another function.Fig. 2.2.2. A Lipschitzian regularization
Note the difference between the two regularizations. Basically, I[k] coincides with
I
at those points whereI
has a slope not larger than k. Figure 2.2.2 illustrates the operation (2.2.3), which has the following mechanical interpretation: gr I[k] is a string, which is not allowed slopes larger than k, and which is pulled upwards under the obstacle epiI.
2.3 Behaviour at Infinity
When studying the minimization of a function
f
E Conv JR, the behaviour off
(x) forIxl
-+ 00 is crucial (assuming domf = JR, the case of interest). It turns out that this behaviour is directly linked to that of the slope-function (1.1.3).Indeed, for fixed xo, the increasing slope-function (1.1.3) satisfies lim sex)
=
sup sex)=
sup sex)x_oo x#xo X>Xo
(equalities in JR U {+oo
D.
For x -+ -00, its limit exists as well (in JR U {-ooD
and is likewise its infimum over xi=
xo, or over x < Xo. To embrace the two cases in one, and to eliminate the unpleasant -00, it is convenient to introduce a positive variable t, playing the role ofIx -
XoI:
we fix a number di=
0 and we considerlim f(xo
+
td) - f(xo) = sup f(xo+
td) - f(xo) =: cp(xo, d).t-+oo t t>o t (2.3.1)
When d is positive [resp. negative], cp is the limiting maximal slope to the right [resp. left] of Xo. It is rather obvious that, for a > 0, cp(xo, ad) = acp(xo, d): in other words, cp(xo, .) is positively homogeneous (of degree 1). Hence, only the values cp(xo, d) for d
=
±1 are relevant, the other values being obtained automatically.Theorem 2.3.1 Let f : JR -+ JR be convex. For each Xo E JR (= dom f), the function cp(xo, .) of (2.3.1) is convex and does not depend on Xo.
PROOF. The result will be confirmed in §Iy'3.2, and closedness of epi
f
is needed, which will be proved later; nevertheless, we give the proofbecause it uses an interesting geometric argument. Fix t > 0; the convexity off
implies that, for arbitrary dl , d2and a E ]0, 1[,
f(xo
+
tadl+
t(1 - a)d2) ~ af(xo+
tdl )+
(1 - a)f(xo+
td2).Subtract f(xo) and divide by t > 0 to see that the difference quotient s(xo
+
td) in(2.3.1) is a convex function of d. Moreover, cp(xo, 0) = 0, hence Proposition 2.1.2 establishes the convexity of cp(xo, .).
To show that cp(xo, .) does not depend on Xo is more involved. Let XI
i=
Xo and take (d, r) E epi CP(XI, .); we must show that (d, r) E epi cp(xo, .) (then the proof will be finished, by exchanging the roles of XI and xo).By definition of epi cp(xo, .), what we have to prove is that poet) E epi f (look at Fig. 2.3.1), where t > 0 is arbitrary and poet) has the coordinates Xo
+
td and f(xo)+
tr. By definition of epi CP(XI , .), PI(t) := (XI+
td, f(xd+
tr) is in epif.Taking e E ]0, 1], the key is to write the point Me of the picture as Me
=
ePI (t)+
(1 - e)Po(t)=
ePI (t /e)+
(1 - e)Po(O) .Because (d, r) E epicp(xlo .), the second form above implies that Me E epi f; the first form shows that, when e .j.. 0, Me tends to poet). Admitting that epi f is closed (Theorem 3.1.1 and Proposition 3.2.2 below), poet) E epi f. 0
2 First Properties 15
PoCO) =
(Xo,f(Xo»
Fig.2.3.1. Inscribing a pantograph in a closed convex set
Thus, instead of cp(xo, d), the notation
f~(d):= lim f(xo
+
td) - f(xo)=
sup f(xo+
td) - f(xo)t-++oo t t>O t
is more appropriate; xo is eliminated, the symbols' and 00 suggest that f~ is a sort of
"slope at infinity". This defines a new convex and positively homogeneous function, associated to f, and characterizing its behaviour at infinity in both directions d
= ±
I : Corollary 2.3.2 Forf :
JR --+ JR convex, there holdslim f(x) = +00 ¢=:} f~(l) > 0,
x-++oo (2.3.2)
. f(x) , I
hm - - = +00 {=:::> foo() = +00.
x-++oo x (2.3.3)
PROOF. By definition, f(x)/x --+ f~(l) for x --+ +00, which proves (2.3.3) and the
"¢:=" in (2.3.2). To finish the proof, use
t/~(l)
=
I~(t) ~ I(t) - 1(0) -++00
when t -++00
(remember 0 E domf
=
JR) and observe that tf60(I) --+ +00 certainly impliesf6o(1) > O. 0
Naturally, (2.3.2) and (2.3.3) have symmetric versions, with x --+ -00.
For (2.3.2) to hold, it suffices that
I
be strictly increasing on some interval of positive length. Functions satisfying (2.3.2) [resp. (2.3.3)] in both directions will be called O-coercive [resp. I-coercive]; they are important for some applications.Geometrically, the epigraph of I~ is a convex cone with apex at (0, 0). When this apex is translated to a point (x, I(x» on gr I, the cone becomes included in epi I: in fact, the definition of I~ gives for all y
I(y) = I(x) + I(y) ~ I(x) ~ I(x) + I~(Y - x) .
It is then clear that epi I
+
epi I~=
epi I (see Fig. 2.3.2), i.e. that I=
I t I~.The epigraph of I~ is the largest convex cone K in jR2 (with apex at (0,0» such that epi
I +
K C epiI.
gr
x Fig. 2.3.2. The cones included in epi f