First and second order necessary conditions for stochastic optimal control problems

HAL Id: inria-00537227

https://hal.inria.fr/inria-00537227v2

Submitted on 25 Jun 2011

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

stochastic optimal control problems

Joseph Frédéric Bonnans, Francisco Silva

To cite this version:

Joseph Frédéric Bonnans, Francisco Silva. First and second order necessary conditions for stochastic optimal control problems. Applied Mathematics and Optimization, Springer Verlag (Germany), 2012, 65 (3), pp.403-439. �inria-00537227v2�

a p p o r t

d e r e c h e r c h e

ISSN0249-6399ISRNINRIA/RR--7454--FR+ENG

Thème NUM

First and second order necessary conditions for stochastic optimal control problems

J. Frédéric Bonnans — Francisco J. Silva

N° 7454

Juin 2011

Centre de recherche INRIA Saclay – Île-de-France Parc Orsay Université

4, rue Jacques Monod, 91893 ORSAY Cedex

Téléphone : +33 1 72 92 59 00

stochastic optimal control problems

J. Fr´ed´eric Bonnans^∗ , Francisco J. Silva^†

Th`eme NUM — Syst`emes num´eriques Equipes-Projets Commands´

Rapport de recherche n°7454 — Juin 2011 — 33 pages

Abstract: In this work we consider a stochastic optimal control problem with either convex control constraints or finitely many equality and inequality constraints over the final state. Using the variational approach, we are able to obtain first and second order expansions for the state and cost function, around a local minimum. This fact allows us to prove general first order necessary condition and, under a geometrical assumption over the constraint set, second order necessary conditions are also established. We end by giving second order optimality conditions for problems with constraints on expectations of the final state.

Key-words: Stochastic optimal control, variational approach, first and second order optimality conditions, polyhedric constraints, final state constraints.

∗INRIA-Saclay and CMAP, ´Ecole Polytechnique, 91128 Palaiseau, France, and Laboratoire de Finance des March´es d’ ´Energie, France (Frederic.Bonnans@inria.fr).

† Dipartimento di Matematica Guido Castelnuovo, 00185 Rome, Italy (fsilva@mat.uniroma1.it).

ordre en commande optimale stochastique

R´esum´e : Nous consid`erons un probl`eme de commande optimale stochastique avec soit des contraintes convexes sur la commande soit un nombre fini de contraintes d’´egalit´e et d’in´egalit´e sur l’´etat final. L’approche ditevariationnelle nous permet d’obtenir un d`eveloppement au premier et au second ordre pour l’´etat et la fonction de coˆut, autour d’un minimum local. Avec ces d`eveloppements on peut montrer des conditions g´en´erales d’optimalit´e de premier ordre et, sous une hypoth`ese g´eom´etrique sur l’ensemble des contraintes, des conditions n´ecessaires du second ordre sont aussi ´etablies. On finit l’article en fournissant des conditions d’optimalit´e du second ordre pour de probl`emes avec de contraintes en esp´erance sur l’´etat final

Mots-cl´es : Commande optimale stochastique, approche variationnelle, conditions d’optimalit´e de premier et second ordre, contraintes poly´edriques, contraintes sur l’´etat final.

1 Introduction

Let us consider a controlled Itˆo process satisfying the following stochastic dif- ferential equation (SDE)

dy(t) = f(t, y(t), u(t), ω)dt+σ(t, y(t), u(t), ω)dW(t), on [0, T]×Ω,

y(0) = y₀∈Rⁿ. (1)

In the notation abovey(t)∈Rⁿ denotes thestate function and u(t)∈R^m the control. Associated to a pairstate-control (y, u) we define its costJ(y, u) by

J(y, u) :=E Z T

0

`(t, y(t), u(t))dt+φ(y(T))

!

. (2)

Precise definitions of the suitable spaces for (y, u) and assumptions over the data (f, σ, `, φ), will be provided in the next section. For the dynamics (1) and the cost (2), we will study two types of problems. In the first one, we suppose that we are given a nonempty closed and convex subset U of the space L²_F of adapted square integrable process, and we analyze the following stochastic optimal control problem withcontrol constraints

Min_(y,u) J(y, u) s.t. (1) holds andu∈ U. (SP) As in the case of deterministic optimal control problems, there are two main approaches to study problem (SP) whenU is defined bylocal constraints, i.e.

for a given nonempty closed and convex subsetU ⊆Rⁿ, U:=

u∈L²_F ; u(t, ω)∈U, for almost all (t, ω)∈[0, T]×Ω . (3) The first approach is the global one, based on Bellman’s dynamic programming principle, which yields that the value function of (SP) is the unique viscosity solution of an associated second order Hamilton-Jacobi-Bellman equation. For a complete account of this point of view, widely used in practical computations, we refer the reader to Lions [21, 22] and to the books [11, 26, 29]. The second approach is the variational one, which consists in the local behavior analysis of the value function under small perturbations of a local minimum. Using this technique, Kushner [18, 19, 20], Bensoussan [1, 2], Bismut [3, 4, 5] and Haussmann [14] obtained natural extensions of Pontryagin maximum principle to the stochastic case, that were generalized by Peng [25] to the case where U is not necessarily convex and by Cadenillas and Karatzas [8] to the non- Markovian case. Relations between the global and variational approach are studied in [30, 31].

Nevertheless, to the best of our knowledge, nothing has been said about second order optimality conditions for (SP). Using the variational technique we are able to obtain first and second order expansions for the cost function, which are expressed in terms of the derivatives of the Hamiltonian of problem (SP). The main tool is a kind of generalization of Gronwall’s lemma for the SDEs (proposition 1) obtained by Mou and Yong [24], which allows to expand the cost with respect to directions belonging to a more regular space than the control space. Note that the idea of using a more regular space that the orig- inal one was already used [6] in the context of deterministic state constrained

RR n°7454

optimal control problems. By a density argument, we establish first order op- timality conditions, which in the case of local constraints are a consequence of the maximum principles obtained in the above references. However, we can also deal with constraints sets that are not necessarily local. The main novelty of our work is that under a polyhedricity assumption (see [13, 23]) over the setU, we are also able to provide second order necessary conditions which are new for the stochastic case and are natural extensions of their deterministic counterparts.

In the second type of problem we suppose that we are given functionsgⁱ,h^j withi∈ {1, ..., ng}, j ∈ {1, ..., nh}, and we study the following optimal control problem withfinitely many equality and inequality constraints

Min_(y,u) J(y, u) s.t. (1) holds andE

gⁱ(y(T))

= 0, E

h^j(y(T))

≤0. (SP⁰) First order optimality conditions for (SP⁰), in a maximum principle form, have been obtained in [25, 8]. Under a standard qualification condition over gⁱ, h^j, the techniques employed for (SP) allow us recover particular cases of the results in [25, 8], but in addition we are also able to prove second order necessary conditions for (SP⁰).

The article is organized as follows: After introducing standard notations and assumptions in section 2, we obtain in section 3 first and second order expansions for the state and cost function. The proof of technical lemmas of this section are provided in the Appendix. In section 4, first and second order necessary conditions are proved for problem (SP) with explicit results for the case ofbox constraints over the control. A discussion about a non gap second order sufficient condition is also provided. Finally, in section 5 first and second order necessary conditions are derived for problem (SP⁰).

2 Notations, assumptions and problem state- ment

Let us first fix some standard notation. Forxbelonging to an Euclidean space we will writexⁱfor itsi-th coordinate and|x|for its Euclidean norm. LetT >0 and consider a filtered probability space (Ω,F,F,P), on which ad-dimensional (d∈N^∗) Brownian motion W(·) is defined. We suppose that F ={Ft}_0≤t≤T is the natural filtration, augmented by allP-null sets inF, associated toW(·).

Let (X,k · kX) be a Banach space and forβ∈[1,∞) set L^β(Ω;X) := n

v: Ω→X; v isF -measurable andE

kv(ω)k^β_X

<∞o , L^∞(Ω;X) := {v: Ω→X; visF -measurable and ess sup_ω∈Ωkv(ω)kX <∞}. Forβ, p∈[1,∞] and m∈Nlet us define

L^β,p_F :=

v∈L^β(Ω;L^p([0, T];R^m)) ; (t, ω)→v(t, ω) :=v(ω)(t) isF-adapted . We endow these space with the norms

kvk_β,p:=h E

kv(ω)k^β_Lp([0,T];

R^m)

i_β¹

andkvk_∞,p:= ess sup_ω∈Ωkv(ω)k_Lp([0,T];R^m). For the sake of clarity, when the context is clear, the statement “for a.a. t ∈ [0, T], a.s. ω ∈ Ω (P-a.s.)” will be abbreviated to “for a.a. (t, ω)”. We will

writeL^p_F :=L^p,p_F andk · kp :=k · kp,p. The spacesL^β,p_F endowed with the norms k · kβ,p are Banach spaces and for the specific case p = 2 the space L²_F is a Hilbert space. We will denote byh·,·i2 the obvious scalar product. Evidently, for β ∈ [1,∞] and 1≤p1 ≤p≤p2 ≤ ∞, there exist positive constants cβ,p₁, cβ,p₂,cp₁,β, cp₂,β such that

cβ,p₁kvkβ,p₁ ≤ kvkβ,p≤cβ,p₂kvkβ,p₂, cp₁,βkvkp₁,β ≤ kvkp,β ≤cp₂,βkvkp₂,β. For a function [0, T]×Rⁿ×R^m×Ω3(t, y, u, ω)→ψ(t, y, u, ω)∈Rⁿ such that for a.a. (t, ω) the function (y, u)→ψ(t, y, u, ω) is C², set ψy(t, y, u, ω) :=

Dyψ(t, y, u, ω) and ψu(t, y, u, ω) :=Duψ(t, y, u, ω). As usual, when the context is clear, we will systematically omit the ω argument in the defined functions.

Now, letz∈Rⁿ andv∈R^mbe variations associated withy andurespectively.

The second derivatives of ψare written in the following form

ψyy(t, y, u)z²:=D²_yyψ(t, y, u)(z, z), ψuu(t, y, u)v²:=D_uu² ψ(t, y, u)(v, v), ψ_yu(t, y, u)zv:=D²_yuψ(t, y, u)(z, v),

ψ(y,u)²(t, y, u)(z, v)²:=ψyy(t, y, u)z²+ 2ψyu(t, y, u)zv+ψuu(t, y, u)v². Consider the mapsf, σⁱ: [0, T]×Rⁿ×R^m×Ω→Rⁿ(i= 1, ..., d). These maps will define the dynamics for our problem. Let us assume that:

(H1) [Assumptions for the dynamics] The mapsψ=f, σⁱ satisfy:

(i) The maps areB([0, T]×Rⁿ×R^m)⊗ FT-measurable.

(ii) For all (y, u) ∈ Rⁿ ×R^m the process [0, T] 3 t → ψ(t, y, u) ∈ Rⁿ is F- adapted.

(iii) For almost all (t, ω)∈[0, T]×Ω the mapping (y, u)→ψ(t, y, u, ω) is C³. Moreover, we assume that there exists a constantL1>0 such that for almost all (t, ω)











|ψ(t, y, u, ω)| ≤L1(1 +|y|+|u|),

|ψy(t, y, u, ω)|+|ψu(t, y, u, ω)| ≤L1,

|ψyy(t, y, u, ω)|+|ψyu(t, y, u, ω)|+|ψuu(t, y, u, ω)| ≤L1

|ψ_(y,u)2(t, y, u, ω)−ψ_(y,u)2(t, y⁰, u⁰, ω)| ≤L1(|y−y⁰|+|u−u⁰|).

(4)

Let us define σ(t, y, u) := (σ¹(t, y, u), ..., σ^d(t, y, u)) ∈ R^n×d. For variations z∈Rⁿ andv∈R^m, associated with yandu, set

σy(t, y, u)z := (σ¹_y(t, y, u)z, ..., σ^d_y(t, y, u)z),

σyy(t, y, u)z² := (σ¹_yy(t, y, u)z², ..., σ^d_yy(t, y, u)z²), (5) and σ_u(t, y, u)v, σ_yu(t, y, u)zv, σ_uu(t, y, u)v², σ_(y,u)2(t, y, u)(z, v)² are analo- gously defined.

For everyβ ∈[1,∞), let us define the spaceY^β as Y^β:=

y∈L^β(Ω;C([0, T];Rⁿ)) ; (t, ω)→y(t, ω) :=y(ω)(t) isF-adapted , endowed with the norm k · kβ,∞. Let y₀ ∈ Rⁿ, under (H1) we have that for every u∈L^β,2_F the SDE

dy(t) = f(t, y(t), u(t))dt+σ(t, y(t), u(t))dW(t),

y(0) = y₀, (6)

is well posed. In fact (see [24, Proposition 2.1]):

Proposition 1. Suppose that (H1)holds. Then, there exists C >0 such that for everyu∈L^β,2_F (β ∈[1,∞)) equation (6) admits a unique solution y ∈ Y^β and

kyk^β_β,∞≤C

|y0|^β+kf(·,0, u(·))k^β_β,1+kσ(·,0, u(·))k^β_β,2

. (7)

Remark 1. Note that by the first condition in (4), the right hand side of (7) is finite.

Now, let us consider maps`: [0, T]×Rⁿ×R^m×Ω→Randφ:Rⁿ×Ω→R. These maps will define the cost function of our problem. We assume:

(H2) [Assumptions for cost] (i) The maps`andφare respectivelyB([0, T]×

Rⁿ×R^m)⊗ F_T andB(Rⁿ)⊗ F_T measurable.

(ii) For all (y, u)∈Rⁿ×R^mthe process [0, T]3t→`(t, y, u)∈RisF-adapted.

(iii) For almost all (t, ω) the maps (y, u)→`(t, y, u, ω) and y→φ(y, ω) areC². In addition, there existsL2>0 such that:



















|`(t, y, u, ω)| ≤L₂(1 +|y|+|u|)², |φ(y, ω)| ≤L₂(1 +|y|)²,

|`y(t, y, u, ω)|+|`u(t, y, u, ω)| ≤L₂(1 +|y|+|u|),

|`<sub>yy</sub>(t, y, u, ω)|+|`_yu(t, y, u, ω)|+|`_uu(t, y, u, ω)| ≤L₂,

|`<sub>(y,u)</sub>2(t, y, u, ω)−`_(y,u)2(t, y⁰, u⁰, ω)| ≤L₂(|y−y⁰|+|u−u⁰|),

|φ_y(y, ω)| ≤L₂(1 +|y|)

|φyy(y, ω)| ≤L2, |φyy(y, ω)−φyy(y⁰, ω)| ≤L2(|y−y⁰|).

(8)

Remark 2. The above assumptions include the important case when the cost function is quadratic in(y, u).

In order to provide second order conditions, which will be natural extensions of well known deterministic results, it will be useful to strengthen (H1) and (H2).

[Lipschitz cost] There exists C_`, C_φ > 0 such that for almost all (t, ω) ∈ [0, T]×Ω and for all (y, u), (y⁰, u⁰)∈Rⁿ×R^mwe have

|`(t, y, u, ω)−`(t, y⁰, u⁰, ω)| ≤ C`(|u−u⁰|+|y−y⁰|),

|φ(y, ω)−φ(y⁰, ω)| ≤ Cφ|y−y⁰|. (9) [Affine dynamics] Forψ=f, σⁱand for almost all (t, ω)∈[0, T]×Ω, we have (y, u)∈Rⁿ×R^m→ψ(t, y, u, ω) is affine. (10) In our main results we will assume:

(H3)At least one of the following assumptions hold:

(H3.i)Condition (9) holds andσ_uu≡0. (H3.ii)Condition (10) holds.

For everyu∈L²_F denote by yu ∈ Y² the solution of (6). Let us define the cost functionJ :L²_F→Rby

J(u) =E

"

Z T 0

`(t, yu(t), u(t))dt+φ(yu(T))

#

. (11)

Note that, in view of the first condition in (8) and estimate (7) the functionJ is well defined.

3 Expansions for the state and cost function

From now on we fix ¯u∈L²_F and set ¯y:=yu¯. We also suppose that assumptions (H1)and(H2)hold. Our aim in this section is to obtain first and second order expansions for v ∈ L^∞_F → yu+v¯ ∈ Y² and v ∈ L^∞_F → J(¯u+v) ∈ R around

¯

v= 0. The main tool for obtaining the expansion for the state is the following corollary of proposition 1, whose proof is straightforward.

Corollary 2. Let A1, A2 ∈L^∞_F([0, T];R^n×n), B_i¹ ∈L^β,2_F ([0, T];Rⁿ) and B_i² ∈ L^∞_F([0, T];R^n×d) fori= 1,2. Assume that there exists a constant K >0 such that

kB¹₁kβ,1≤KkB¹₂kβ,2. (12) Then, omitting time from function arguments, for everyw∈L^β,2, the SDE

dz =

A1z+B¹₁+B²₁w dt+

A2z+B₂¹+B₂²w dW(t),

z(0) = 0, (13)

has a unique solution in Y^β and

kzk^β_β,∞=





 O

maxn

kB₂¹k^β_β,2,kwk^β_β,1o

ifB₂²≡0, O

maxn

kB₂¹k^β_β,2,kwk^β_β,2o

otherwise.

Remark 3. Note that the estimates given in corollary 2 are sharp. In fact, suppose that d = 1 and let w ∈ L²([0, T];R) (deterministic). Consider the process z(t)defined by

z(t) :=

Z t 0

w(s)dW(s) for allt∈[0, T].

By definition kzk^β_β,∞ ≥ E(|z(T)|^β) = kwk^β₂E(|Z|^β), where Z is an standard normal random variable. Since, in this specific case, kwk^β_β,2 = kwk^β₂, the conclusion follows.

Regarding the expansion for the cost J, the following lemma, which is a consequence of It’s lemma for multidimensional It process (see e.g. [17, 29]), will be useful. For the reader convenience we provide the short proof.

Lemma 3. Let Z1 andZ2 be Rⁿ-valued continuous process satisfying dZ1(t) = b1(t)dt+σ1(t)dW(t) for all t∈[0, T],

dZ2(t) = b2(t)dt+σ2(t)dW(t) for all t∈[0, T], (14) where b₁, b₂ ∈ L²(Ω, L²([0, T],Rⁿ)) and σ₁, σ₂ ∈ L²(Ω, L²([0, T],R^n×d)) are F-adapted process. Also, let us suppose that P-a.s. we have that Z₁(0) = 0.

Then

E(Z₁(T)·Z₂(T)) =E Z T

0

"

Z₁(t)·b₂(t) +Z₂(t)·b₁(t) +

d

X

i=1

σ₁ⁱ(t)·σ₂ⁱ(t)

# dt

! .

Proof. It’s lemma implies that Z₁(T)·Z₂(T) =

Z T 0

"

Z₁(t)·b₂(t) +Z₂(t)·b₁(t) +

d

X

i=1

σ₁ⁱ(t)·σ₂ⁱ(t)

#

dt+M(T), (15) whereM(t) is a continuous local martingale given by

M(t) :=

d

X

i=1

Z t 0

Z1(s)·σⁱ₂(s) +Z2(s)·σ₁ⁱ(s)

dWⁱ(s).

By theBurkholder-Davis-Gundy inequality (see e.g [17]) we have the existence of a constantK >0 such that for alli= 1, ..., d

E sup

t∈[0,T]

Z t 0

Z1(s)·σⁱ₂(s) +Z2(s)·σ₁ⁱ(s)

dWⁱ(s)

!

≤K

Z1·σ₂ⁱ+Z2·σⁱ_{1 1,2}

.

The Cauchy Schwarz inequality yields that (assuming thatn= 1 for notational convenience)

Z1·σⁱ_{2 1,2}=E



 Z T

0

|Z1(t)|² σⁱ₂(t)

2dt

!^1/2

≤ ||Z1||_2,∞

σ₂ⁱ

₂<+∞, with an analogous estimate for

Z₂·σ₁ⁱ

_1,2. Therefore, by [27, Theorem 51], we have thatM(t) is a martingale with null expectation. The result follows.

Forψ=`, f, σ andt∈[0, T], let us define

ψ_y(t) =ψ_y(t,y(t),¯ u(t));¯ ψ_u(t) =ψ_u(t,y(t),¯ u(t)), ψ¯ _yu(t) =ψ_yu(t,y(t),¯ u(t));¯ ψyy(t) =ψyy(t,y(t),¯ u(t));¯ ψuu(t) =ψuu(t,y(t),¯ u(t)),¯

ψ_(y,u)2(t) =ψ_(y,u)2(t,y(t),¯ u(t)).¯

As usual in optimal control theory, the expansions forJ, with respect to varia- tions of the control variable in the uniform normk · k∞, will be written in terms of an adjoint state and the derivatives of an associated Hamiltonian. Let us define theadjoint state (¯p,q)¯ ∈L²_F([0, T];Rⁿ)×(L²_F([0, T];Rⁿ))^das the unique solution of the following backward stochastic differential equation (BSDE) (see e.g. [1, 5])

dp(t) = −

"

`y(t)^>+fy(t)^>p(t) +

m

X

i=1

σ_yⁱ(t)^>qⁱ(t)

#

dt+q(t)dW(t), p(T) = φy(¯y(T))^>.

(16)

In the notation aboveσⁱ andqⁱ denote respectively the ith column of σandq.

The following estimates hold (see [24, Proposition 3.1]):

Proposition 4. Assume that(H1),(H2)hold and thatu¯∈L^β,2_F (β∈[1,∞)).

Then there existsC⁰>0such that kpk¯ ^β_β,∞+

d

X

i=1

kq¯ⁱk^β_β,2≤C⁰

1 +kuk¯ ^β_β,2 .

TheHamiltonian H : [0, T]×Rⁿ×R^m×Rⁿ×R^n×d×Ω→Ris defined as H(t, y, u, p, q, ω) :=`(t, y, u, ω) +p·f(t, y, u, ω) +

d

X

i=1

qⁱ·σⁱ(t, y, u, ω). (17) For notational convenience, omitting the dependence on ω, we set

H_u(t) :=H_u(t,y(t),¯ u(t),¯ p(t),¯ q(t)),¯

H_(y,u)2(t) :=H_(y,u)2(t,y(t),¯ u(t),¯ p(t),¯ q(t)).¯ (18)

3.1 First order expansions

Let β ∈[1,∞] and v ∈L^β,2_F . We consider thelinearized mapping v ∈L^β,2_F → y1[¯u, v]∈ Y^β, wherey1[¯u, v] is the unique solution of

dy1(t) = [fy(t)y1(t) +fu(t)v(t)]dt+ [σy(t)y1(t) +σu(t)v(t)]dW(t),

y₁(0) = 0. (19)

The second assumption in (4) and proposition 1 yields that y₁[¯u, v] is well de- fined. If the context is clear, for notational convenience we will write y₁ = y₁[¯u, v]. Corollary 2 will be the main tool for establishing the following useful estimates:

Lemma 5. Let v∈L^2β,4_F withβ ∈[1,∞)and set

δy=δy[¯u, v] :=y_u+v¯ −y,¯ d₁=d₁[¯u, v] :=δy−y₁. Then, the following estimates hold:

kδyk^β_β,∞+ky1k^β_β,∞ =

( O(kvk^β_β,1) if σ_u≡0,

O(kvk^β_β,2) otherwise. (20) kd₁k^β_β,∞ =

( O(kvk^2β_2β,2) if σuu≡0,

O(kvk^2β_2β,4) otherwise. (21) Proof. See the appendix.

Now we can prove the following proposition.

Proposition 6. The mapv∈L^∞_F →y(v) :=ˆ yu+v¯ ∈ Y²is differentiable with a Lipschitz derivative given by

Dy(¯ˆv)v=y1[¯u+ ¯v, v] for all ¯v, v∈L^∞_F. (22) Proof. By estimate (21) in lemma 5 we have that kδy−y₁k2,∞ = O(kvk²_∞), implying that (22) holds. Let us prove that Dyˆ is Lipschitz. For notational convenience, we assume that n = m = d = 1. Let v₁, v₂ ∈ L^∞_F and write δv:=v₁−v₂. Forψ=f, σ,i= 1,2 andv⁰ ∈L^∞_F withkv⁰k_∞= 1, by an abuse of notation we set

y⁽ⁱ⁾:=y¯u+v_i, y₁⁽ⁱ⁾:=y1[¯u+vi, v⁰], δy:=y⁽¹⁾−y⁽²⁾, δy1:=y₁⁽¹⁾−y₁⁽²⁾ ψ⁽ⁱ⁾y (t) :=ψy(t, y⁽ⁱ⁾(t),u(t) +¯ vi(t)), ψ⁽ⁱ⁾u (t) :=ψu(t, y⁽ⁱ⁾(t),u(t) +¯ vi(t)), δψy(t) :=ψy⁽¹⁾(t)−ψ⁽²⁾y (t), δψu(t) :=ψu⁽¹⁾(t)−ψ⁽²⁾u (t).

A straightforward calculation shows that

dδy1(t) = [fy⁽²⁾(t)δy1(t) +δfy(t)y₁⁽¹⁾(t) +δfu(t)v⁰(t)]dt +[σy⁽²⁾(t)δy1(t) +δσy(t)y₁⁽¹⁾(t) +δσu(t)v⁰(t)]dW(t), δy1(0) = 0.

(23)

Using(H1)and corollary 2 we obtain that

kδy1k²_2,∞=O kδyk²_2,∞+kδvk²₂ ,

uniformly inv⁰∈L^∞_F withkv⁰k∞= 1. The result follows from (20).

Now we focus our attention on the cost functionJ. Let us define Υ1:L²_F→ Rby

Υ1(v) :=E Z T

0

Hu(t)v(t)dt

!

. (24)

In view of proposition 4, withβ = 2, Υ1 is well defined. Lemma 3 yields the following well known alternative expression for Υ1.

Lemma 7. For every v∈L²_F we have that:

Υ1(v) =E Z T

0

[`y(t)y1(t) +`u(t)v(t)] dt+φy(¯y(T))y1(T)

!

. (25) Proof. Noting that

φ_y(¯y(T))y₁(T) = ¯p(T)·y₁(T)−p(0)¯ ·y₁(0),

lemma 3, applied toZ1=y1andZ2= ¯p, yieldsE(φy(¯y(T))y1(T)) =I1+I2+I3, where

I1 := −E RT

0 y1(t)·h

`y(t)^>+fy(t)^>p(t) +¯ Pd

i=1σⁱ_y(t)^>q¯ⁱ(t)i dt

, I₂ := E

RT

0 p(t)¯ ·[f_y(t)y₁(t) +f_u(t)v(t)] dt , I₃ := Pd

i=1E RT

0 q¯ⁱ(t)·

σ_yⁱ(t)y₁(t) +σ_uⁱ(t)v(t) dt

.

Plugging the expressions of I1, I2 and I3 introduced above into the right hand side of (25) yields the result.

The expression above for Υ1 allows to obtain afirst order expansion of J around ¯u.

Proposition 8. Assume that (H1),(H2) hold and letv∈L⁴_F. Then, J(¯u+v) =J(¯u) + Υ1(v) +r1(v),

with

Υ1(v) =O(kvk2); r1(v) =

O kvk²_4,2

if σ_uu≡0, O kvk²₄

otherwise. (26) If in addition (9) holds, then

Υ₁(v) =

O(kvk1) if σ_u≡0,

O(kvk_1,2) otherwise, ; r₁(v) =

O kvk²₂

ifσ_uu≡0, O kvk²_2,4

otherwise.

(27)