On a class of stochastic optimal control problems related to BSDEs with quadratic growth

HAL Id: hal-00451623

https://hal.archives-ouvertes.fr/hal-00451623

Submitted on 20 Dec 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

to BSDEs with quadratic growth

Marco Fuhrman, Ying Hu, Gianmario Tessitore

To cite this version:

Marco Fuhrman, Ying Hu, Gianmario Tessitore. On a class of stochastic optimal control problems

related to BSDEs with quadratic growth. SIAM Journal on Control and Optimization, Society for

Industrial and Applied Mathematics, 2006, 45 (4), pp.1279-1296. �10.1137/050633548�. �hal-00451623�

Vol. 45, No. 4, pp. 1279–1296

ON A CLASS OF STOCHASTIC OPTIMAL CONTROL PROBLEMS RELATED TO BSDES WITH QUADRATIC GROWTH

MARCO FUHRMAN^†, YING HU^‡, AND GIANMARIO TESSITORE^§

Abstract. In this paper, we study a class of stochastic optimal control problems, where the drift term of the equation has a linear growth on the control variable, the cost functional has a quadratic growth, and the control process takes values in a closed set (not necessarily compact). This problem is related to some backward stochastic differential equations (BSDEs) with quadratic growth and unbounded terminal value. We prove that the optimal feedback control exists, and the optimal cost is given by the initial value of the solution of the related BSDE.

Key words.stochastic optimal control, backward stochastic differential equations, quadratically growing driver, unbounded final condition

AMS subject classifications. 93E20, 60H10 DOI.10.1137/050633548

1. Introduction. In this paper, we consider a controlled equation of the form dX

= b(t, X

) dt + σ(t, X

) [dW

+ r(t, X

, u

) dt], t ∈ [0, T ],

X

= x.

(1.1)

In the equation, W is an R

-valued Wiener process, defined on a complete probability space (Ω, F , P ) with respect to a filtration ( F

)

satisfying the usual conditions;

the unknown process X takes values in R

; x is a given element of R

; and u is the control process, which is assumed to be an ( F

)-adapted process taking values in a given nonempty closed set K ⊂ R

. The control problem consists of minimizing a cost functional of the form

J = E

0

g(t, X

, u

) dt + E φ(X

).

(1.2)

We suppose that r has a linear growth in u, g has quadratic growth in x and u, and φ has quadratic growth in x.

The main novelty of the present paper, in comparison with the existing literature, is that on the one hand we assume that neither K nor r is bounded; on the other hand we consider a degenerate control problem (since nothing is assumed on the image of σ). Moreover, we also allow φ to have quadratic growth.

∗Received by the editors June 14, 2005; accepted for publication (in revised form) March 24, 2006;

published electronically September 26, 2006.

http://www.siam.org/journals/sicon/45-4/63354.html

†Dipartimento di Matematica, Politecnico di Milano, piazza Leonardo da Vinci 32, 20133 Milano, Italy (marco.fuhrman@polimi.it). This author was supported by the European Community’s Human Potential Programme under contracts HPRN-CT-2002-00281 (Evolution Equations) and HPRN-CT- 2002-00279 (QP-Applications).

‡IRMAR, Universit´e de Rennes 1, Campus de Beaulieu, 35042 Rennes Cedex, France (ying.hu@

univ-rennes1.fr).

§Dipartimento di Matematica e Applicazioni, Universit`a di Milano-Bicocca, Via Cozzi 53, 20135, Milano, Italy (gianmario.tessitore@unimib.it). This author was supported by the European Commu- nity’s Human Potential Programme under contract HPRN-CT-2002-00281 (Evolution Equations).

1279

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Nonlinear backward stochastic differential equations (BSDEs) were first intro- duced by Pardoux and Peng [11]. A major application of BSDEs is in stochastic control; see, e.g., [4] and [12]. We also refer the reader to [9], [5], and [7]. From several points of view, the class of control problems addressed in these papers, and in the references therein, is more general than the one considered here, but assumptions implying “bounded control image” (i.e., boundedness of K or r in our notation) are a common feature of all the above papers. The special “unbounded” case correspond- ing to the assumptions K = R

and g =

| u |

+ q(t, x) is treated in [6] by an ad hoc exponential transform. We notice that in [6] φ is allowed to take the value + ∞ . Finally, the same special case (in which the Hamiltonian is exactly the square of the norm of the gradient) was treated in [8] by analytic techniques under nondegeneracy assumptions and in an infinite-dimensional framework.

The difficulty here is that the Hamiltonian corresponding to the control prob- lem has quadratic growth in the gradient and consequently the associated BSDE has quadratic growth in the Z variable. Well-posedness for this class of BSDEs has been proved in [10] in the case of bounded terminal value. Since we allow for unbounded terminal cost, to treat such equations we have to apply the techniques recently intro- duced in [2]. We notice that for such BSDEs no general uniqueness results are known:

we replace uniqueness with the selection of a maximal solution. Moreover, the usual application of the Girsanov technique is not allowed (since the Novikov condition is not guaranteed), and we have to develop specific arguments both to prove the fun- damental relation (see section 4) and to obtain the existence of a (weak) solution to the closed loop equation (see section 5). Our main result is to prove that the optimal feedback control exists and the optimal cost is given by the value Y

of the maximal solution (Y, Z ) of the BSDE with the quadratic growth and unbounded terminal value mentioned above. Moreover, we show that we can construct an optimal feedback in terms of the process Z. Finally, we prove that if we fix a particular optimal feed- back law, then the solution of the corresponding closed loop equation is unique; see Proposition 5.4.

An alternative approach to the control problem may consist of applying the stochastic maximum principle; see, e.g., [12] for a detailed exposition. However, this would require further differentiability conditions on the coefficients of the controlled equation and would not immediately imply existence of an optimal control, since the maximum principle is usually stated as a necessary condition for optimality.

The paper is organized as follows. In the next section, we describe the control problem; in section 3, we study the related BSDE; and in section 4, we establish the fundamental relation between the optimal control problem and BSDE. The last section is devoted to the proof of the existence of optimal feedback control.

2. The control problem. We consider the optimal control problem given by a state equation of the form

dX

= b(t, X

) dt + σ(t, X

) [dW

+ r(t, X

, u

) dt], t ∈ [0, T ], X

= x,

(2.1)

and given by a cost functional of the form J = E

g(t, X

, u

) dt + E φ(X

).

(2.2)

We work under the following assumptions.

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Hypothesis 2.1.

1. The process W is a Wiener process in R

, defined on a complete probability space (Ω, F , P ) with respect to a filtration ( F

) satisfying the usual conditions.

2. The set K is a nonempty closed subset of R

.

3. The functions b : [0, T ] × R

→ R

, σ : [0, T ] × R

→ R

, r : [0, T ] × R

× K → R

, g : [0, T ] × R

× K → R , φ : R

→ R are Borel measurable.

4. For all t ∈ [0, T ], x ∈ R

, r(t, x, · ) and g(t, x, · ) are continuous functions from K to R

and from K to R , respectively.

5. There exists a constant C > 0 such that for every t ∈ [0, T ], x, x

∈ R

, u ∈ K it holds that

| b(t, x) − b(t, x

) | ≤ C | x − x

| , | b(t, x) | ≤ C(1 + | x | ), (2.3)

| σ(t, x) − σ(t, x

) | ≤ C | x − x

| , (2.4)

| σ(t, x) | ≤ C, (2.5)

| r(t, x, u) − r(t, x

, u) | ≤ C(1 + | u | ) | x − x

| , (2.6)

| r(t, x, u) | ≤ C(1 + | u | ), (2.7)

0 ≤ g(t, x, u) ≤ C(1 + | x |

+ | u |

), (2.8)

0 ≤ φ(x) ≤ C(1 + | x |

).

(2.9)

6. There exist R > 0 and c > 0 such that for every t ∈ [0, T ], x ∈ R

, and every u ∈ K satisfying | u | ≥ R,

g(t, x, u) ≥ c | u |

. (2.10)

We will say that an ( F

)-adapted stochastic process { u

, t ∈ [0, T ] } with values in K is an admissible control if it satisfies

E

0

| u

|

dt < ∞ . (2.11)

This square summability requirement is justified by (2.10): a control process which is not square summable would have infinite cost.

Remark 2.2. Some classes of linear quadratic control problems fall within the scope of our result. Consider the controlled system

dX

= A(t)X

dt + b(t) dt + Σ(t) [dW

+ B(t)u

dt], t ∈ [0, T ], X

= x

(2.12)

and the cost functional J = 1

2 E

0

[ Q(t)X

, X

+ R(t)u

, u

] dt + 1

2 E SX

, X

,

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

and assume that A, b, Σ, B, Q, R are bounded Borel measurable functions with values in R

, R

, R

, R

, R

, R

, respectively, and that S ∈ R

; also suppose that Q(t) ≥ 0, S ≥ 0, R(t) ≥ cI for some c > 0 and all t, in the sense of the usual matrix order. Then Hypothesis 2.1 is verified. We are not assuming K = R

in general. If K = R

, then the usual linear quadratic theory applies and gives more general results in the sense that state- and control-dependent noise can be considered, as well as a more general drift coefficient, in (2.12).

Next we show that for every admissible control, the solution to (2.1) exists.

Proposition 2.3. Let u be an admissible control. Then there exists a unique, continuous, ( F

)-adapted process X satisfying E sup

| X

|

< ∞ , and P -a.s., X

= x +

b(s, X

) ds +

0

σ(s, X

) dW

+

0

σ(s, X

) r(s, X

, u

) ds, t ∈ [0, T ].

Proof. The proof of Proposition 2.3 relies on an approximation procedure that will be used again in what follows. We introduce the sequence of stopping times

τ

= inf

t ∈ [0, T ] :

0

| u

|

ds > n

,

with the convention that τ

= T if the indicated set is empty. By (2.11), for P -almost every ω ∈ Ω, there exists an integer N (ω) depending on ω such that

n ≥ N (ω) = ⇒ τ

(ω) = T.

(2.13)

Let us fix u

∈ K, and for every n, let us define u

= u

1

+ u

1

and consider the equation

dX

= b(t, X

) dt + σ(t, X

) [dW

+ r(t, X

, u

) dt], t ∈ [0, T ], X

= x.

(2.14)

We claim that (2.14) has a unique solution X

in the class of ( F

)-adapted pro- cesses X satisfying sup

E | X

|

< ∞ .

To prove the claim we use a classical argument: We first write (2.14) in the form dX

= b(t, X

) dt + b(t, X

) dt + σ(t, X

) dW

,

where b(t, x) = σ(t, x, u

)r(t, x, u

) is a stochastic coefficient which satisfies, by (2.3)–

(2.7),

| b(t, x) − b(t, x

) | = | σ(t, x, u

)r(t, x, u

) − σ(t, x

, u

)r(t, x

, u

) | ≤ C(1 + | u

| ) | x − x

| ,

| b(t, x) | = | σ(t, x, u

)r(t, x, u

) | ≤ C(1 + | u

| ), t ∈ [0, T ], x, x

∈ R

. We define K

(ω) = C(1 + | u

(ω) | ), and we note that

| K

|

dt ≤ C

1 +

0

| u

|

dt

≤ C(1 + n + | u

|

) =: C, P -a.s.,

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

by the definition of τ

. Next we introduce the norm X

:= sup

e

E | X

|

and prove that the mapping Γ defined by

Γ(X)

=

0

b(s, X

) ds +

0

b(s, X

) ds +

0

σ(s, X

) dW

, t ∈ [0, T ], is a contraction with respect to this norm for sufficiently large λ > 0. For a pair of processes X, X

, we have

E

0

[ b(s, X

) − b(s, X

)] ds

≤ E

t 0

K

| X

− X

| ds

≤ E

t

0

| K

|

ds

0

| X

− X

|

ds

≤ C E

0

| X

− X

|

ds

≤ C X − X

0

e

ds, and it follows that

sup

t∈[0,T]

e

E

0

[ b(s, X

) − b(s, X

)] ds

≤ C X − X

0

e

ds

≤ C

2λ X − X

.

The other verifications needed to prove the contraction property are standard. The claim is proved.

It is clear that the solution X

of (2.14) is also continuous. Moreover, we have X

= X

for t ≤ τ

;

therefore there exists a process X such that

X

= X

for t ≤ τ

,

and X is clearly the required solution. The property E sup

| X

|

< ∞ is an immediate consequence of the following lemma, which concludes the proof.

Lemma 2.4. Under the previous assumptions, the family of random variables sup

t∈[0,T]

| X

|

, n = 1, 2, . . . , is uniformly integrable.

Proof. We set M

=

0

σ(s, X

) dW

. By (2.14), we have X

= x +

b(s, X

) ds +

0

σ(s, X

)r(s, X

, u

) ds + M

.

First, we claim that the family { sup

| M

|

; n = 1, 2, . . . } is uniformly integrable;

indeed it is uniformly bounded in L

(Ω, F , P ) for every p ∈ [1, ∞ ), since by the Burkholder–Davis–Gundy inequalities and the boundedness of σ (see (2.5)),

E sup

t∈[0,T]

| M

|

≤ C E

| σ(s, X

) |

ds

≤ C

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

for some constant independent of n. Next we note that, by (2.7),

0

σ(s, X

)r(s, X

, u

) ds

≤ C

(1 + | u

| )

ds ≤ C

1 +

0

| u

|

ds

. Then, setting l

= sup

| X

|

, we obtain by (2.3),

l

≤ C

1 +

0

| u

|

ds + sup

t∈[0,T]

| M

|

+ C

0

| X

|

ds

≤ C

1 +

0

| u

|

ds + sup

t∈[0,T]

| M

|

+ C

0

l

ds.

From the Gronwall lemma we deduce l

≤ C

1 +

| u

|

ds + sup

t∈[0,T]

| M

|

, and the required uniform integrability follows.

The stochastic control problem associated with (2.1)–(2.2) consists of minimizing the cost functional J(x, u) among all the admissible controls.

3. The forward-backward system. We consider again the functions b, σ, g, φ satisfying the assumptions in Hypothesis 2.1. We define the Hamiltonian function

ψ(t, x, z) = inf

u∈K

[g(t, x, u) + z · r(t, x, u)], t ∈ [0, T ], x ∈ R

, z ∈ R

, (3.1)

where · denotes the usual scalar product in R

. We collect some immediate properties of the function ψ.

Lemma 3.1. The map ψ is a Borel measurable function from [0, T ] × R

× R

to R . There exists a constant C > 0 such that

− C(1 + | z |

) ≤ ψ(t, x, z) ≤ g(t, x, u) + C | z | (1 + | u | ) ∀ u ∈ K.

(3.2)

Moreover, the infimum in (3.1) is attained in a ball of radius C(1 + | x | + | z | ), that is, ψ(t, x, z) = min

u∈K,|u|≤C(1+|x|+|z|)

[g(t, x, u) + z · r(t, x, u)], (3.3)

t ∈ [0, T ], x ∈ R

, z ∈ R

, and

ψ(t, x, z) < g(t, x, u) + z · r(t, x, u) if | u | > C (1 + | x | + | z | ).

(3.4)

Finally, for every t ∈ [0, T ] and x ∈ R

, z → ψ(t, x, z) is continuous on R

.

Proof. The measurability of ψ is straightforward since, by the continuity of r and g with respect to u, we have ψ(t, x, z) = inf

[g(t, x, u) + z · r(t, x, u)], t ∈ [0, T ], x ∈ R

, z ∈ R

, where K is any countable dense subset of K.

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Coming now to the proof of (3.2), we notice that, since g is nonnegative and satisfies (2.10), we have g(t, x, u) ≥ c( | u |

− R

) (c and R are the same as in (2.10)).

By (2.7) and (2.8), we have

g(t, x, u) + z · r(t, x, u) ≥ c( | u |

− R

) − C | z | (1 + | u | ), (3.5)

and it follows that ψ(t, x, z) ≥ inf

u∈R^m

[g(t, x, u) + z · r(t, x, u)] ≥ inf

u∈R^m

[c( | u |

− R

) − C | z | (1 + | u | )]

= − C

| z |

− C

,

by direct computation, for suitable constants C

and C

. This proves the left-hand side of (3.2). The right-hand side of (3.2) is immediate, since by (2.7),

ψ(t, x, z) ≤ g(t, x, u) + z · r(t, x, u) ≤ g(t, x, u) + | z | C(1 + | u | ).

We come now to the second assertion. By (3.5) we get g(t, x, u) + z · r(t, x, u) ≥ c | u | | u | − C

c | z |

− cR

− C | z | . (3.6)

On the other hand, if we fix an arbitrary u

∈ K, then

g(t, x, u

) + z · r(t, x, u

) ≤ C(1 + | x |

+ | u

|

) + C | z | (1 + | u

| ) ≤ C

(1 + | x |

+ | z | ).

(3.7)

Hence, there exists a constant C > 0 such that, if | u | > C(1 + | x | + | z | ), then g(t, x, u) + z · r(t, x, u) > g(t, x, u

) + z · r(t, x, u

),

and (3.3) follows from the continuity of g and r with respect to u.

Finally, the continuity of ψ(t, x, · ) can be easily proved, taking into account (3.3).

Next we take an arbitrarily complete probability space (Ω, F , P

) and a Wiener process W

in R

with respect to P

. We denote by ( F

) the associated Brownian filtration, i.e., the filtration generated by W

and augmented by the P

-null sets of F ; ( F

) satisfies the usual conditions.

We introduce the forward equation

dX

= b(t, X

) dt + σ(t, X

) dW

, t ∈ [0, T ], X

= x,

(3.8)

whose solution is a continuous ( F

)-adapted process, which exists and is unique by classical results. Next we consider the associated backward equation

dY

= − ψ(t, X

, Z

) dt + Z

dW

, t ∈ [0, T ], Y

= φ(X

).

(3.9)

The solution of (3.9) exists in the sense specified by the following proposition.

Proposition 3.2. Assume that b, σ, g, φ satisfy Hypothesis 2.1. Then there exist Borel measurable functions

v : [0, T ] × R

→ R , ζ : [0, T ] × R

→ R

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

with the following property: For an arbitrarily chosen complete probability space (Ω, F , P

) and Wiener process W

in R

, denoting by X the solution of (3.8), the processes Y, Z defined by

Y

= v(t, X

), Z

= ζ(t, X

) satisfy

E

sup

t∈[0,T]

| Y

|

< ∞ , E

0

| Z

|

dt < ∞ ;

moreover, Y is continuous and nonnegative, and P

-a.s., Y

+

Z

dW

= φ(X

) +

t

ψ(s, X

, Z

) ds, t ∈ [0, T ].

Finally, this solution is the maximal solution among all the solutions (Y

, Z

) of (3.9) satisfying

E

sup

t∈[0,T]

| Y

|

< + ∞ .

Proof. From Lemma 3.1, there exists a constant C > 0 such that

− C(1 + | z |

) ≤ ψ(t, x, z) ≤ g(t, x, u

) + C(1 + | u

| ) | z | . Let us first note that

E

sup

t∈[0,T]

| X

|

< ∞ ∀ p ≥ 2.

Next, we adopt the same strategy as that in [2] to construct a maximal solution to (3.9); i.e., for each n ≥ C, we define the globally Lipschitz continuous function,

ψ

(t, x, z) = sup { ψ(t, x, q) − n | q − z | : q ∈ Q

} ,

which is decreasing and converges to ψ; then by (Y

, Z

) we denote the unique solution to the BSDE with Lipschitz coefficient ψ

,

dY

= − ψ

(t, X

, Z

) dt + Z

dW

, t ∈ [0, T ], Y

= φ(X

),

(3.10)

and by (Y

, Z

) the unique solution to the BSDE,

dY

= − [g(t, X

, u

) + C(1 + | u

| ) | Z

| ] dt + Z

dW

, t ∈ [0, T ], Y

= φ(X

),

(3.11)

where C is the same as in (3.2). We notice that, since ψ

(t, x, 0) ≥ ψ(t, x, 0) ≥ 0, then by an application of the comparison theorem (see [4]),

0 ≤ Y

≤ Y

.

Then let us introduce the following stopping time: For k ≥ 1, τ

= inf { t ∈ [0, T ] : max( | X

| , Y

) > k } , with the convention that τ

= T if the indicated set is empty.

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Then (Y

, Z

) := (Y

k

, Z

1

) satisfies the following BSDE:

Y

(t) = ξ

+

t

1

ψ

(s, X

, Z

(s))ds −

t

Z

(s)dW

, where of course ξ

= Y

(T ) = Y

.

For fixed k, Y

is decreasing in n and remains bounded by k. It follows from Lemma 3 in [2] (which is a slight generalization of Proposition 2.4 in [10]) that there exists a process (Y

, Z

) such that Y

is a continuous process, E

0

| Z

(s) |

ds < + ∞ , lim

sup

t∈[0,T]

| Y

(t) − Y

(t) | = 0, lim

n

E

0

| Z

(t) − Z

(t) |

dt = 0,

and (Y

, Z

) solves the BSDE Y

(t) = ξ

+

1

ψ(s, X

, Z

(s))ds −

t

Z

(s)dW

, (3.12)

where ξ

= inf

Y

.

On the other hand, τ

≤ τ

, and, from the definition of (Y

, Z

), we have Y

(t ∧ τ

) = Y

(t), Z

(t)1

= Z

(t).

Sending n to infinity, we get

Y

(t ∧ τ

) = Y

(t), Z

(t)1

= Z

(t).

Now we define Y and Z on [0, T ] by setting

Y

= Y

(t), Z

= Z

(t) if t ∈ [0, τ

].

For P

-a.s. ω, there exists an integer K(ω) such that for k ≥ K(ω), τ

(ω) = T.

Thus Y is a continuous process, Y

= φ(X

), and

0

| Z

|

ds < ∞ , P

-a.s.

From (3.12), (Y, Z ) satisfies Y

= Y

+

ψ(s, X

, Z(s))ds −

t∧τ_k

Z (s)dW

. (3.13)

By sending k to infinity, we deduce that (Y, Z ) is a solution of (3.9) and lim

sup

t∈[0,T]

| Y

− Y

| = 0, lim

n

| Z

− Z

|

dt = 0, P

-a.s.

Thus | Z

− Z | converges to zero in measure d P

⊗ dt, and passing, if needed, to a subsequence (that by abuse of language we still denote Z

), we can assume that

| Z

− Z | → 0, d P

⊗ dt almost everywhere.

Now, as ψ

is globally Lipschitz continuous, from [4] (see also [6]) there exist Borel measurable functions

v

: [0, T ] × R

→ R , ζ

: [0, T ] × R

→ R

such that

Y

= v

(t, X

), Z

= ζ

(t, X

).

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

It suffices to define

v(t, x) = lim inf

n→∞

v

(t, x), and ζ(t, x) = lim inf

n→∞

ζ

(t, x) (where the second lim inf is intended coordinatewise) to get

Y

= v(t, X

), Z

= ζ(t, X

), which implies that (v, ζ) is the Borel function we look for.

Finally, 0 ≤ Y

≤ Y

implies that E sup

t∈[0,T]

| Y

|

< ∞ , and from the equation

| Y

|

+

t

| Z

|

ds = 2

t

Y

ψ(s, X

, Z

)ds − 2

t

Y

Z

dW

, taking into consideration that

Y

ψ(s, X

, Z

) ≤ Y

(g(s, X

, u

) + C(1 + | u

| ) | Z

| ) ≤ Y

(g(s, X

, u

) + C(1 + | u

| ) | Z

| ), we deduce, by standard arguments, that

E

0

| Z

|

dt < ∞ .

Moreover, this solution is the maximal solution among all the solutions (Y

, Z

) satisfying

E

sup

t∈[0,T]

| Y

|

< + ∞ ,

and it suffices to apply Proposition 5 of [2] to deduce that Y

≥ Y

and then Y ≥ Y

. 4. The fundamental relation. In this section we revert to the notation intro- duced in the first section and still assume that Hypothesis 2.1 holds.

Proposition 4.1. Let v, ζ denote the functions in the statement of Proposi- tion 3.2. Then for every admissible control u and for the corresponding trajectory X starting at x, we have

J(u) = v(0, x) + E

0

[ − ψ(t, X

, ζ(t, X

)) + ζ(t, X

) · r(t, X

, u

) + g(t, X

, u

)] dt.

Proof. We introduce stopping times τ

and control processes u

as in the proof of Proposition 2.3, and we denote by X

the solution to (2.14). Let us define

W

= W

+

0

r(s, X

, u

) ds.

From the definition of τ

and from (2.7), it follows that

0

| r(s, X

, u

) |

ds ≤ C

0

(1 + | u

| )

ds ≤ C

0

(1 + | u

| )

ds + C ≤ C + Cn.

(4.1)

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Therefore defining ρ

= exp

− r(s, X

, u

) dW

− 1 2

| r(s, X

, u

) |

ds

,

the Novikov condition implies that E ρ

= 1. Setting d P

= ρ

d P , by the Girsanov theorem W

is a Wiener process under P

. Let us denote by ( F

) its natural aug- mented filtration. Since

dX

= b(t, X

) dt + σ(t, X

) dW

, t ∈ [0, T ], X

= x

has a strong solution by classical results, the process X

is also ( F

) adapted. Let us define

Y

= v(t, X

), Z

= ζ(t, X

).

Then by Proposition 3.2, we have

dY

= Z

dW

− ψ(t, X

, Z

) dt, t ∈ [0, T ], Y

= φ(X

)

(4.2) and E

0

| Z

|

dt < ∞ , where E

denotes expectation with respect to P

. It follows that

Y

n

= φ(X

) +

τ_n

ψ(t, X

, Z

) dt −

τ_n

Z

dW

−

τ_n

Z

· r(t, X

, u

) dt.

(4.3)

We note that for every p ∈ [1, ∞ ) we have ρ

= exp

p

r(s, X

, u

) dW

− p

2

| r(s, X

, u

) |

ds

· exp

p

− p 2

| r(s, X

, u

) |

ds

.

By (4.1) the second exponential is bounded by a constant depending on n and p, while the first one has P

-expectation, equal to 1. So we conclude that E

ρ

< ∞ . It follows that

E

0

| Z

|

dt

= E

ρ

0

| Z

|

dt

≤

E

ρ

E

| Z

|

dt

< ∞ . We conclude that the stochastic integral in (4.3) has zero expectation, and we obtain

E Y

n

= E φ(X

) + E

τn

[ψ(t, X

, Z

) − Z

· r(t, X

, u

)] dt.

Since by definition, ψ(t, x, z) − z · r(t, x, u) − g(t, x, u) ≤ 0, we have E Y

≤ E φ(X

) + E

g(t, X

, u

) dt.

(4.4)

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Now we let n → ∞ . By the definition of u

and (2.8), E

g(t, X

, u

) dt = E

0

1

g(t, X

, u

) dt

≤ C E

0

1

(1 + | X

|

+ | u

|

)dt ≤ C E

(T − τ

)

1 + sup

t∈[0,T]

| X

|

, and the right-hand side tends to 0 by Lemma 2.4 and (2.13). Next we note that, again by (2.13), for n ≥ N(ω) we have τ

(ω) = T and

φ(X

) = φ(X

) = φ(X

) = φ(X

).

Moreover, by (2.9),

| φ(X

) | ≤ C(1 + | X

|

) ≤ C

1 + sup

t∈[0,T]

| X

|

,

and by Lemma 2.4 the right-hand side is uniformly integrable. We deduce that E φ(X

) → E φ(X

), and from (4.4) we conclude that lim sup

E Y

n

≤ E φ(X

).

On the other hand, for n ≥ N (ω) we have τ

(ω) = T and Y

= Y

= φ(X

) = φ(X

).

Since Y

is positive, by the Fatou lemma, E φ(X

) ≤ lim inf

E Y

. We have thus proved that

lim

n→∞

E Y

n

= E φ(X

).

(4.5)

Now we return to (4.2) and write Y

= Y

+

− ψ(t, X

, Z

) dt +

0

Z

dW

+

0

Z

· r(t, X

, u

) dt.

Arguing as before, we conclude that the stochastic integral has zero P -expectation.

Moreover, we have Y

= v(0, x), and, for t ≤ τ

, also have u

= u

, X

= X

, and Z

= ζ(t, X

). Thus we obtain

E Y

= v(0, x) + E

0

[ − ψ(t, X

, ζ (t, X

)) + ζ(t, X

) · r(t, X

, u

)] dt and

E

0

g(t, X

, u

) dt + E Y

= v(0, x) + E

0

[ − ψ(t, X

, ζ(t, X

)) + ζ(t, X

) · r(t, X

, u

) + g(t, X

, u

)] dt.

Noting that − ψ(t, x, z) + z · r(t, x, u) + g(t, x, u) ≥ 0 and recalling that g(t, x, u) ≥ 0, by (4.5) and the monotone convergence theorem, we obtain for n → ∞ ,

E

0

g(t, X

, u

) dt + E φ(X

) = v(0, x) + E

[ − ψ(t, X

, ζ(t, X

)) + ζ(t, X

) · r(t, X

, u

) + g(t, X

, u

)] dt, which gives the required conclusion.

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Corollary 4.2. For every admissible control u and any initial datum x, we have J (u) ≥ v(0, x), and the equality holds if and only if the following feedback law holds P -a.s. for almost every t ∈ [0, T ]:

ψ(t, X

, ζ (t, X

)) = ζ(t, X

) · r(t, X

, u

) + g(t, X

, u

), where X is the trajectory starting at x and corresponding to control u.

5. Existence of optimal controls: The closed loop equation. Let us con- sider again the functions b, σ, g, φ satisfying the assumptions in Hypothesis 2.1. We recall the definition of the Hamiltonian function:

ψ(t, x, z) = inf

u∈K

[g(t, x, u) + z · r(t, x, u)], t ∈ [0, T ], x ∈ R

, z ∈ R

. (5.1)

Lemma 5.1. There exists a Borel measurable function γ : [0, T ] × R

× R

→ K such that

ψ(t, x, z) = g(t, x, γ(t, x, z)) + z · r(t, x, γ(t, x, z)), t ∈ [0, T ], x ∈ R

, z ∈ R

. (5.2)

Moreover, there exists a constant C > 0 such that

| γ(t, x, z) | ≤ C(1 + | x | + | z | ).

(5.3)

Proof. Consider the function F (t, x, z, u) = g(t, x, u) + z · r(t, x, u), t ∈ [0, T ], x ∈ R

, z ∈ R

. Clearly F is a Carath´ eodory map (that is, F (t, x, z, · ) is continuous for all t ∈ [0, T ], x ∈ R

, and z ∈ R

, and F ( · , · , · , u) is Borel measurable for all u ∈ K;

see [1, p. 311]). By (3.3) we have

ψ(t, x, z) ∈ { F(t, x, z, u) : u ∈ K } ∀ t ∈ [0, T ], x ∈ R

, z ∈ R

.

Thus by the Filippov theorem (see, e.g., [1, Thm. 8.2.10, p. 316]) there exists a Borel measurable map γ : [0, T ] × R

× R

→ K such that F(t, x, z, γ(t, x, z)) = ψ(t, x, z);

see [3] as well. The fact that | γ(t, x, z) | ≤ C(1 + | x | + | z | ) is an immediate consequence of (3.4).

Next we address the problem of finding a weak solution to the so-called closed loop equation. We define

u(t, x) = γ(t, x, ζ(t, x)), t ∈ [0, T ], x ∈ R

, where ζ is defined in Proposition 3.2. The closed loop equation is

dX

= b(t, X

) dt + σ(t, X

) [dW

+ r(t, X

, u(t, X

)) dt], t ∈ [0, T ], X

= x.

(5.4)

By a weak solution we mean a complete probability space (Ω, F , P ) with a filtration ( F

) satisfying the usual conditions, a Wiener process W in R

with respect to P and ( F

), and a continuous ( F

)-adapted process X with values in R

satisfying, P -a.s.,

| u(t, X

) |

dt < ∞ , (5.5)

and such that (5.4) holds. We note that by (2.7) it also follows that

0

| r(t, X

, u(t, X

)) | dt < ∞ , P -a.s., so that (5.4) makes sense.

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Proposition 5.2. Assume that b, σ, g, φ satisfy Hypothesis 2.1. Then there exists a weak solution of the closed loop equation, satisfying in addition

E

0

| u(t, X

) |

dt < ∞ . (5.6)

Proof. Let us take an arbitrary complete probability space (Ω, F , P

) and a Wiener process W

in R

with respect to P

. Let ( F

) be the associated Brownian filtration.

We define the process X as the solution of the equation

dX

= b(t, X

) dt + σ(t, X

) dW

, t ∈ [0, T ], X

= x.

(5.7)

The solution is a continuous ( F

)-adapted process, which exists and is unique by classical results. Moreover, it satisfies E

sup

| X

|

< ∞ for every p ∈ [1, ∞ ).

By Proposition 3.2, setting

Y

= v(t, X

), Z

= ζ(t, X

), the following backward equation holds:

dY

= − ψ(t, X

, Z

) dt + Z

dW

, t ∈ [0, T ], Y

= φ(X

),

and we have

E

0

| Z

|

dt < ∞ . (5.8)

By (2.7) we have | r(t, X

, u(t, X

)) | ≤ C(1 + | u(t, X

) | ), and by (5.3),

| u(t, X

) | = | γ(t, X

, ζ(t, X

)) | ≤ C(1 + | X

| + | ζ(t, X

) | ) = C(1 + | X

| + | Z

| ).

(5.9)

Now let us define stopping times τ

= inf

t ∈ [0, T ] :

0

| u(s, X

) |

ds > n

,

with the convention that τ

= T if the indicated set is empty. By (5.8) and (5.9), for P

-a.s. ω ∈ Ω, there exists an integer N(ω) depending on ω such that τ

(ω) = T for n ≥ N (ω). Let us fix u

∈ K, and for every n let us define

u

= u(t, X

) 1

+ u

1

, M

= exp

t 0

r(s, X

, u

) dW

− 1 2

| r(s, X

, u

) |

ds

,

M

= exp

t 0

r(s, X

, u(s, X

)) dW

− 1 2

| r(s, X

, u(s, X

)) |

ds

,

W

= W

−

0

r(s, X

, u

) ds, W

= W

−

r(s, X

, u(s, X

)) ds.

Downloaded 12/20/17 to 129.20.36.117. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php