Master MVA Optimization Reminders Part I

(1)

Optimization Reminders Part I

Jean-Christophe Pesquet Center for Visual Computing

Inria, CentraleSup´elec Institut Universitaire de France

jean-christophe@pesquet.eu

(2)

Reference books

◮ D. Bertsekas, Nonlinear programming, Athena Scientic, Belmont, Massachussets, 1996.

◮ Y. Nesterov, Introductory Lectures on Convex Optimization : A Basic Course, Springer, 2004.

◮ S. Boyd and L. Vandenberghe, Convex optimization, Cambridge University Press, 2004.

◮ H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, Springer, New York, 2017.

(3)

A few mathematical background

(4)

Hilbert spaces

A (real) pre-Hilbertian spaceH is a real vector space endowed with a function h· | ·i: H²→R, called inner product such that

➀ (∀(x,y)∈ H²)hx |yi=hy |xi

➁ (∀(x,y,z)∈ H³) hx +y |zi=hx |zi+hy |zi

➂ (∀(x,y)∈ H²) (∀α∈R)hαx |yi=αhy |xi

➃ (∀x ∈ H)hx |xi ≥0 and hx |xi= 0 ⇔ x = 0.

◮ The associated norm is

(∀x ∈ H) kxk=p hx |xi.

(5)

Hilbert spaces

A (real) Hilbert spaceH is a complete pre-Hilbertian space.

◮ Particular case :H=R^N (Euclidean space with dimensionN).

(6)

Hilbert spaces

A (real) Hilbert spaceH is a complete pre-Hilbertian space.

◮ Particular case :H=R^N (Euclidean space with dimensionN).

2^H is the power set of H, i.e. the family of all subsets of H.

(7)

Hilbert spaces

Let HandG be two Hilbert spaces.

A linear operator L:H → G is bounded if kLk= sup

kxkH≤1

kLxk_G <+∞

(8)

Hilbert spaces

kxk≤1

kLxk<+∞

(9)

Hilbert spaces

kxk=1

kLxk<+∞

(10)

Hilbert spaces

x∈H\{0}

kLxk

kxk <+∞

(11)

Hilbert spaces

x∈H\{0}

kLxk

kxk <+∞

◮ A linear operator fromHto G is continuous if and only if it is bounded.

◮ In finite dimension, every linear operator is bounded.

◮ In the following, it will be assumed that all the underlying Hilbert spaces are finite dimensional.

(12)

Hilbert spaces

x∈H\{0}

kLxk

kxk <+∞

◮ A linear operator fromHto G is continuous if and only if it is bounded.

◮ In finite dimension, every linear operator is bounded.

◮ In the following, it will be assumed that all the underlying Hilbert spaces are finite dimensional.

B(H,G) : Banach space of (bounded) linear operators fromH to G.

(13)

Hilbert spaces

Let L∈ B(H,G). Its adjointL^∗ is the operator inB(G,H) defined as (∀(x,y) ∈ H × G) hy |Lxi_G =hL^∗y |xi_H.

(14)

Hilbert spaces

Let L∈ B(H,G). Its adjointL^∗ is the operator inB(G,H) defined as (∀(x,y)∈ H × G) hy |Lxi=hL^∗y |xi.

(15)

Hilbert spaces

Let L∈ B(H,G). Its adjointL^∗ is the operator inB(G,H) defined as (∀(x,y)∈ H × G) hLx |yi=hx |L^∗yi.

(16)

Hilbert spaces

Example :

If L:H → Hⁿ:x 7→(x, . . . ,x) then L^∗:Hⁿ→ H:y = (y₁, . . . ,y_n)7→

Xn i=1

y_i

Proof :

hLx |yi=h(x, . . . ,x)|(y1, . . . ,yn)i= Xn

i=1

hx |yii=

* x|

Xn

i=1

yi

+

(17)

Hilbert spaces

◮ We havekL^∗k=kLk.

◮ If Lis bijective (i.e. an isomorphism ) thenL⁻¹ ∈ B(G,H) and (L⁻¹)^∗ = (L^∗)⁻¹.

◮ If H=R^N andG=R^M then L^∗=L^⊤.

(18)

Infinite values functions

(19)

Functional analysis : definitions

LetS be a nonempty set of a Hilbert space H.

Letf : S →]− ∞, +∞] .

◮ The domain off isdomf ={x ∈S|f(x)<+∞}.

◮ The functionf is proper ifdomf 6=∅.

Domains of the functions ?

x f(x)

δ

(20)

Functional analysis : definitions

Letf : S →]− ∞, +∞] .

x f(x)

domf =R (proper)

x f(x)

δ

(21)

Functional analysis : definitions

Letf : S →]− ∞, +∞] .

x f(x)

domf =R (proper)

x f(x)

δ

domf =]0, δ]

(proper)

(22)

Functional analysis : definitions

Let C ⊂ H.

The indicator function of C is (∀x∈ H) ι_C(x) =

(0 ifx ∈C +∞ otherwise.

Example :C = [δ₁, δ₂]

f(x) =ι[δ1,δ2](x)

δ1 δ2 x

(23)

Limits inf and sup

Let (ξn)_n∈N be a sequence of elements in [−∞,+∞].

Its infimum limit is lim infξn= lim_n→+∞inf ξk

k ≥n ∈[−∞,+∞]

and its supremum limit is lim supξ_n = limn→+∞sup

ξ_k k ≥n ∈ [−∞,+∞].

(24)

Limits inf and sup

Let (ξn)_n∈N be a sequence of elements in [−∞,+∞].

Its infimum limit is lim infξn= lim_n→+∞inf ξk

k ≥n ∈[−∞,+∞]

and its supremum limit is lim supξ_n = limn→+∞sup

ξ_k k ≥n ∈ [−∞,+∞].

◮ lim supξ_n=−lim inf(−ξn)

◮ lim infξ_n≤lim supξ_n

◮ lim_n→+∞ξ_n=ξ∈[−∞,+∞] if and only if lim infξ_n= lim supξ_n=ξ.

(25)

Epigraph

Let f :H → ]−∞,+∞]. The epigraph of f is epif =

(x, ζ)∈domf ×R f(x)≤ζ

(26)

Epigraph

Let f :H → ]−∞,+∞]. The epigraph of f is epif =

(x, ζ)∈domf ×R f(x)≤ζ

x f(x) =|x|

epi_f

x

−δ δ

f(x) =ι[−δ,δ](x) epi_f

(27)

Lower semi-continuity

Let f :H → ]−∞,+∞].

f is a lower semi-continuous (l.s.c.) function at x ∈ H if, for every sequence (x_n)_n∈Nof H,

xn→x ⇒ lim inff(xn)≥f(x).

(28)

Lower semi-continuity

Let f :H → ]−∞,+∞].

f is a lower semi-continuous function on Hif and only if epif is closed

◮ l.s.c. functions ?

x f(x)

(29)

Lower semi-continuity

Let f :H → ]−∞,+∞].

f(x)

x x

f(x)

(30)

Lower semi-continuity

Let f :H → ]−∞,+∞].

x f(x)

(31)

Lower semi-continuity

Let f :H → ]−∞,+∞].

x f(x)

(32)

Lower semi-continuity

Let f :H → ]−∞,+∞].

x f(x)

(33)

Lower semi-continuity

◮ Every continuous function onH is l.s.c.

◮ Every finite sum of l.s.c. functions is l.s.c.

◮ Let (fi)_i∈I be a family of l.s.c functions.

Then, sup_i_∈Ifi is l.s.c.

(34)

Does a minimum exist ?

(35)

Minimizers

Let S be a nonempty set of a Hilbert spaceH.

Letf :S → ]−∞,+∞] be a proper function and let xb∈S.

◮ bx is a local minimizer of f ifxb∈domf and there exists an open neigborhood O of xbsuch that

(∀x∈O∩S) f(x)b ≤f(x).

◮ bx is a (global) minimizer of f if

(∀x∈S) f(x)b ≤f(x).

(36)

Minimizers

Let S be a nonempty set of a Hilbert spaceH.

Letf :S → ]−∞,+∞] be a proper function and let xb∈S.

◮ bx is a strict local minimizer of f if there exists an open neigborhood O ofxbsuch that

(∀x ∈(O∩S)\ {xb}) f(bx)<f(x).

◮ bx is a strict (global and unique) minimizer of f if (∀x∈S\ {x})b f(xb)<f(x).

(37)

Existence of a minimizer

Weierstrass theorem

Let S be a nonempty compact set of a Hilbert spaceH.

Letf :S → ]−∞,+∞] be a proper l.s.c function.

Then, there exists xb∈S such that f(xb) = inf

x∈Sf(x).

(38)

Existence of a minimizer

Let Hbe a Hilbert space. Letf :H → ]−∞,+∞].

f is coercive if lim_kxk→+∞f(x) = +∞.

(39)

Existence of a minimizer

Coercive functions ?

x f(x) +∞

(40)

Existence of a minimizer

Coercive functions ?

x f(x) +∞

(41)

Existence of a minimizer

Theorem

Let Hbe a (finite dimensional) Hilbert space.

Let f :H → ]−∞,+∞] be a proper l.s.c. coercive function.

Then, the set of minimizers of f is a nonempty compact set.

(42)

Existence of a minimizer

Theorem

Let Hbe a (finite dimensional) Hilbert space.

Let f :H → ]−∞,+∞] be a proper l.s.c. coercive function.

Then, the set of minimizers of f is a nonempty compact set.

Proof : Sincef is proper, there exists x₀ ∈ H such thatf(x₀)∈R. The coercivity off implies that there existsη∈ ]0,+∞[ such that, for every x ∈ H satisfyingkx−x₀k> η,f(x)>f(x₀).

LetS =

x ∈ H kx−x₀k ≤η ,S∩domf 6=∅andS is compact.

Then, there existsxb∈S such that f(x) = infb _x∈Sf(x)≤f(x₀). Thus, f(x) = infb _x∈Hf(x).

Argminf ⊂S is bounded. In addition, if (x_n)_n∈N is a sequence of

minimizers converging tobx∈ H. Then,f(x)b ≤lim inff(xn) = inf_x∈Hf(x) and, consequently, bx∈Argminf. Therefore,Argminf is closed.

(43)

Exercise 1

Letf be the Shannon entropy function defined as

f(x) =







 XN

i=1

x⁽ⁱ⁾lnx⁽ⁱ⁾ ifx = (x⁽ⁱ⁾)_1≤i≤N ∈ ]0,+∞[^N +∞ if (∃j ∈ {1, . . . ,N}) x^(j) <0.

1. How can we extend the definition of functionf so that it is l.s.c. on R^N?

2. What can be said about the existence of a minimizer of this function on a nonempty closed subset of the set

C =

(x⁽ⁱ⁾)_1≤i_≤N ∈[0,+∞[^N P^N_i₌₁x⁽ⁱ⁾= 1 ?

(44)

Hints from differential calculus

(45)

Necessary condition for the existence of a minimizer (Euler’s inequality)

Theorem

LetD be an open subset of a Hilbert space Hand letC ⊂D. Let f: D→ ]−∞,+∞] be differentiable at xb∈ C. If bx is a local minimizer off on C then, for every y ∈ H such that [xb,y]⊂C,

h∇f(xb)|y −xbi ≥0.

If xb∈int(C), then the condition reduces to

∇f(xb) = 0.

(46)

Necessary condition for the existence of a minimizer (Euler’s inequality)

Theorem

LetD be an open subset of a Hilbert space Hand letC ⊂D. Let f: D→ ]−∞,+∞] be differentiable at xb∈ C. If bx is a local minimizer off on C then, for every y ∈ H such that [xb,y]⊂C,

h∇f(xb)|y −xbi ≥0.

∇f(xb) = 0.

Remark : A zero of the gradient∇f is called a critical point of f.

(47)

Necessary condition for the existence of a minimizer

Theorem

Let C be an open subset of a Hilbert space H. Let f : C → R be differentiable on C. Let f be twice differentiable at xb ∈ C. If If bx is a local minimizer off on C, then

(i) ∇f(bx) = 0 ;

(ii) the Hessian∇²f(xb) off atxbis positive semi-definite, i.e.

(∀z ∈ H)

z | ∇²f(xb)z

≥0.

(48)

Sufficient conditions for the existence of a minimizer

Theorem

Let C be an open subset of a Hilbert space H. Let f: C →R be differentiable onC.

(i) Iff is twice differentiable atxb∈C, ∇f(bx) = 0 and the Hessian

∇²f(bx) off atbx is positive definite, i.e.

(∀z ∈ H \ {0})

z | ∇²f(xb)z

>0.

thenf has a strict local minimum atbx.

(49)

Sufficient conditions for the existence of a minimizer

Theorem

Let C be an open subset of a Hilbert space H. Let f: C →R be differentiable onC.

(i) Iff is twice differentiable atxb∈C, ∇f(bx) = 0 and the Hessian

∇²f(bx) off atbx is positive definite, i.e.

(∀z ∈ H \ {0})

z | ∇²f(xb)z

>0.

thenf has a strict local minimum atbx.

(ii) Iff is twice differentiable on an open neighborhood D⊂C of bx,

∇f(bx) = 0 and the Hessian off is positive semi-definite on D, i.e.

(∀x ∈D)(∀z ∈ H)

z| ∇²f(x)z

≥0, thenf has a local minimum atxb.

(50)

Magic of convexity

(51)

Convex set : definition

Let Hbe a Hilbert space. C ⊂ His a convex set if

(∀(x,y)∈C²)(∀α∈]0,1[) αx+ (1−α)y ∈C

Convex sets ?

C C C

(52)

Convex set : definition

Let Hbe a Hilbert space. C ⊂ His a convex set if

(∀(x,y)∈C²)(∀α∈]0,1[) αx+ (1−α)y ∈C

Convex sets ?

C C C

(53)

Convex set : properties

◮ ∅is considered as a convex set.

◮ If C is a convex set, then (∀n∈N^∗) (∀(x₁, . . . ,x_n)∈Cⁿ) (∀(α₁, . . . , α_n)∈[0,+∞[ⁿ) with

Xn i=1

α_i = 1, Xn

i=1

α_ix_i ∈C.

◮ Every vector (affine) space is convex.

◮ If C is a convex set, thenint(C) andC are convex sets.

(54)

Convex set : properties

◮ If C is a convex set then, for every α∈R,

αC =

αx x ∈C is a convex set.

◮ If C₁ andC₂ are convex sets, then C₁×C₂

C₁+C₂=

x₁+x₂ (x1,x₂)∈C₁×C₂ are convex sets.

◮ If (Ci)_i∈I is a family of convex sets ofH, then \

i∈I

Ci is convex.

(55)

Convex hull

LetHbe a Hilbert space andC ⊂ H. The convex hull ofC is the smallest convex set includingC. It is denoted by conv(C).

◮ conv(C) is the intersection of all the convex sets including C.

◮ Letx ∈ H.x ∈conv(C) if and only if (∃n∈N^∗) (∃(x₁, . . . ,xn)∈Cⁿ) (∃(α₁, . . . , αn)∈ ]0,+∞[ⁿ with

Xn i=1

αi = 1 such that

x= Xn i=1

α_ix_i.

(56)

Convex function : definitions

f :H → ]−∞,+∞] is a convex function if

∀(x,y)∈ H²

(∀α∈]0,1[)

f(αx + (1−α)y)≤αf(x) + (1−α)f(y)

(57)

Convex function : definitions

∀(x,y)∈(domf)²

(∀α∈]0,1[)

f(αx+ (1−α)y)≤αf(x) + (1−α)f(y)

(58)

Convex function : definitions

∀(x,y)∈(domf)²

(∀α∈]0,1[)

f(αx+ (1−α)y)≤αf(x) + (1−α)f(y)

Convex functions ?

x f(x) =|x|

x f(x) =p

|x|

+∞sinon)

x

−δ δ

f(x) =ι[−δ,δ](x) (0 six ∈[−δ, δ]

(59)

Convex function : definitions

∀(x,y)∈(domf)²

(∀α∈]0,1[)

f(αx+ (1−α)y)≤αf(x) + (1−α)f(y)

Convex functions ?

x f(x) =|x|

x f(x) =p

|x|

+∞sinon)

x

−δ δ

f(x) =ι[−δ,δ](x) (0 six ∈[−δ, δ]

(60)

Convex function : definitions

∀(x,y)∈(domf)²

(∀α∈]0,1[)

f(αx+ (1−α)y)≤αf(x) + (1−α)f(y)

◮ f : H →[−∞,+∞[ is concave if −f is convex.

(61)

Convex functions : definition

f :H → ]−∞,+∞] is convex⇔ its epigraph is a convex set.

(62)

Convex functions : definition

x f(x) =|x|

epi_f

x f(x) =p

|x|

epi_f

x

−δ δ

f(x) =ι[−δ,δ](x) epi_f

(63)

Convex functions : definition

x f(x) =|x|

epi_f

x f(x) =p

|x|

epi_f

x

−δ δ

f(x) =ι[−δ,δ](x epi_f

(64)

Convex functions : properties

◮ If f:H → ]−∞,+∞] is convex, then domf is convex and its lower level set at height η∈R

lev_≤η f =

x∈ H f(x)≤η is a convex set.

◮ f:H → ]−∞,+∞] is convex if and only if (∀(x,y)∈(domf)²) ϕ_x,y: [0,1]→ ]−∞,+∞] :α7→f(αx+ (1−α)y) is convex.

(65)

Convex functions : properties

◮ Every finite sum of convex functions is convex.

◮ Let (f_i)_i∈I be a family of convex functions. Then, sup_i_∈If_i is convex.

◮ Γ₀(H) : class of convex, l.s.c., and proper functions from Hto ]−∞,+∞].

◮ LetC ⊂ H.

ιC ∈Γ₀(H) ⇔ C is a nonempty closed convex set.

Proof : epi_ι_C =C ×[0,+∞[.

(66)

Strictly convex functions

f is strictly convex if

(∀x∈domf)(∀y ∈domf)(∀α∈]0,1[)

x6=y ⇒ f(αx+ (1−α)y)< αf(x) + (1−α)f(y).

(67)

Strictly convex functions

(∀x∈domf)(∀y ∈domf)(∀α∈]0,1[)

x6=y ⇒ f(αx+ (1−α)y)< αf(x) + (1−α)f(y).

Strictly convex functions ?

x f(x)

(68)

Strictly convex functions

(∀x∈domf)(∀y ∈domf)(∀α∈]0,1[)

x6=y ⇒ f(αx+ (1−α)y)< αf(x) + (1−α)f(y).

Strictly convex functions ?

x f(x)

(69)

Minimizers of a convex function

Theorem

Let H be a Hilbert space. Let f: H → ]−∞,+∞] be a proper convex function such that µ= inff >−∞.

◮

x ∈ H f(x) =µ is convex.

◮ Every local minimizer off is a global minimizer.

◮ If f is strictly convex, then there exists at most one minimizer.

(70)

Minimizers of a convex function

Theorem

◮

Proof : Let Ω =

x∈ H f(x) =µ . Let (x,y)∈Ω² and letα∈[0,1].

We have

f(αx+ (1−α)y)≤αf(x) + (1−α)f(y) =µ which shows that αx+ (1−α)y ∈Ω.

(71)

Minimizers of a convex function

Theorem

◮

Proof : Letxbbe a local minimizer off. For everyy ∈ H \ {x}, thereb exists α∈]0,1[ such that

f(x)b ≤f xb+α(y−bx)

≤(1−α)f(bx) +αf(y)

⇒ f(x)b ≤f(y)

If f is strictly convex, the inequality is strict.

(72)

Existence and uniqueness of a minimizer

Theorem

LetHbe a Hilbert space andC a closed convex subset ofH. Letf ∈Γ₀(H) such that domf ∩C 6=∅.

If f is coercive or C is bounded, then there existsxb∈C such that f(bx) = inf

x∈Cf(x).

If, moreover, f is strictly convex, this minimizerxbis unique.

(73)

Exercise 2

LetH be a Hilbert space.

1. Show that the functionx 7→ kxk² is strictly convex.

2. A functionf :H → ]−∞,+∞] is strongly convexwith modulus β∈ ]0,+∞[ if there exists a convex functiong:H → ]−∞,+∞]

such that

f =g +β 2k · k².

Show that that every strongly convex function is strictly convex.

3. Show that a functionf:H → ]−∞,+∞] is strongly convex with modulusβ ∈ ]0,+∞[ if and only if

(∀(x,y)∈ H²)(∀α∈]0,1[) f(αx+ (1−α)y) +α(1−α)β

2kx−yk² ≤αf(x) + (1−α)f(y).

(74)

Convex + smooth

(75)

Characterization of differentiable convex functions

Let f: H → ]−∞,+∞] be differentiable on domf, which is a nonempty open convex set.

Then, f is convex if and only if

(∀(x,y)∈(domf)²) f(y)≥f(x) +h∇f(x)|y −xi.

(76)

Characterization of differentiable strictly convex functions

Let f: H → ]−∞,+∞] be differentiable on domf, which is a nonempty open convex set.

Then, f is strictly convex if and only if, for every (x,y) ∈ (domf)² with x 6=y,

f(y)>f(x) +h∇f(x)|y −xi.

(77)

Characterization of differentiable convex functions

Let f :H → ]−∞,+∞] be differentiable on domf, which is a nonempty open convex set.

Then,f is convex if and only if∇f is monotone on domf, i.e.

(∀(x,y)∈(domf)²) h∇f(y)− ∇f(x)|y−xi ≥0.

(78)

Characterization of strictly differentiable convex functions

Let f :H → ]−∞,+∞] be differentiable on domf, which is a nonempty open convex set.

Then, f is strictly convex if and only if∇f is strictly monotone on domf, i.e. for every (x,y)∈(domf)² withx 6=y,

h∇f(y)− ∇f(x)|y−xi>0.

(79)

Characterization of twice differentiable convex functions

LetHbe a Hilbert space.

Letf: H → ]−∞,+∞] be a twice differentiable function ondomf, which is a nonempty open convex set.

◮ f is convex if and only if, for every x ∈domf, ∇²f(x) is positive semi-definite.

◮ If, for everyx ∈domf, ∇²f(x) is positive definite, thenf is strictly convex.

(80)

Condition for the existence of a minimizer

Theorem

LetHbe Hilbert space.

Let f: H → ]−∞,+∞] be a differentiable convex function on domf, which is an open set. LetC ⊂domf be a nonempty convex set. xb∈C is a (global) minimizer of f on C if and only if

(∀y ∈C) h∇f(bx)|y −xbi ≥0.

∇f(xb) = 0.

(81)

Condition for the existence of a minimizer

Theorem

LetHbe Hilbert space.

Let f: H → ]−∞,+∞] be a differentiable convex function on domf, which is an open set. LetC ⊂domf be a nonempty convex set. xb∈C is a (global) minimizer of f on C if and only if

(∀y ∈C) h∇f(bx)|y −xbi ≥0.

∇f(xb) = 0.

Proof : We have already seen that the inequality is a necessary condition for b

x to be a local minimizer of f on C and that it reduces to the vanishing condition on the gradient ifxb∈int(C).

(82)

Condition for the existence of a minimizer

Theorem

LetH be Hilbert space.

Let f: H → ]−∞,+∞] be a differentiable convex function on domf, which is an open set. Let C ⊂domf be a nonempty convex set.xb∈C is a (global) minimizer off onC if and only if

(∀y ∈C) h∇f(x)b |y−bxi ≥0.

Ifbx ∈int(C), then the condition reduces to

∇f(x) = 0.b

Proof : Conversely, assume that the inequality holds. Lety ∈C. Sincef is convex and Gˆateaux differentiable,

f(y)≥f(bx) +h∇f(xb)|y−xbi ≥f(xb).

Hence, bx is a minimizer off onC.

(83)

Exercice 3

Letf :H →R be differentiable.

Show thatf isβ-strongly convex if and only if

(∀(x,y)∈ H²) f(y)≥f(x) +h∇f(x)|y −xi+β

2ky −xk².

(84)

Exercice 4

Let

f: R^N →R (x⁽ⁱ⁾)_1≤i_≤N 7→ln

XN i=1

exp(x⁽ⁱ⁾)

! .

Show that f is convex. Is it strictly convex ?

(85)

Projections

(86)

Projection onto a closed convex set

Theorem

LetC be a nonempty closed convex set of a Hilbert space H.

(i) For everyx ∈ H, there exists a unique pointxbinC which lies at minimum distance ofx. The applicationP_C: H →C which maps everyx ∈ Hto its associated pointxbis called the projection ontoC . (ii) For everyx ∈ H, bx =P_C(x) if and only ifxb∈C and

(∀y ∈C) hx −xb|y−bxi ≤0.

(87)

Geometrical interpretation

LetC be a nonempty subset of H.

For everyx ∈ H, the normal cone to C atx is defined as N_C(x) =

(u∈ H (∀y∈C) hu|y−xi ≤0 ifx ∈C

∅ otherwise.

C

NC(x)

x u

C

N_C(x)

x u

(88)

Geometrical interpretation

Let C be a nonempty subset ofH.

For everyx ∈ H, the normal cone toC atx is defined as N_C(x) =

(u∈ H (∀y ∈C) hu |y−xi ≤0 if x∈C

∅ otherwise.

◮ If x∈intC, thenN_C(x) ={0}.

◮ If C is a vector space, then for everyx ∈C,N_C(x) =C^⊥.

◮ LetC be nonempty closed convex set of a Hilbert space H. For every x∈ H,

b

x=P_C(x) ⇔ x−xb∈N_C(bx).

(89)

Examples of projections

◮ If C is a (closed) vector space of a Hilbert space H, thenP_C is the (linear) orthogonal projection ontoC. Then, for every x ∈ H,

b

x =P_C(x) ⇔

(xb∈C x−xb∈C^⊥.

(90)

Properties of the projection

◮ LetC be a nonempty closed convex set of a Hilbert space H. The projection onto C is a firmly nonexpansive operator , i.e.

(∀(x,y)∈ H²) kP_C(x)−P_C(y)k² ≤ hx−y|P_C(x)−P_C(y)i.

(91)

Properties of the projection

(92)

Properties of the projection

◮ The projection ontoC is a nonexpansive operator , i.e.

(∀(x,y)∈ H²) kPC(x)−PC(y)k ≤ kx−yk.

◮ The projection ontoC is uniformly continuous.

◮ The distance to C defined as (∀x∈ H) d_C(x) = inf

y∈Ckx−yk=kx−P_C(x)k is continuous.

(93)

Exercise 5

LetH andG be Hilbert spaces.

LetD be a nonempty closed convex set ofG.

LetL∈ B(H,G) be a bijective isometry and let C =

x∈ H Lx ∈D . 1. Show that the projection ontoC is well-defined.

2. Show thatP_C =L^∗◦P_D◦L.

3. ExpressPC when H=G=R^N andD = [0,+∞[^N.