Infinite Horizon Stochastic Optimal Control Problems with Running Maximum Cost

HAL Id: hal-01585766

https://hal.archives-ouvertes.fr/hal-01585766v3

Submitted on 25 Sep 2018

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Infinite Horizon Stochastic Optimal Control Problems with Running Maximum Cost

Axel Kröner, Athena Picarelli, Hasnaa Zidani

To cite this version:

Axel Kröner, Athena Picarelli, Hasnaa Zidani. Infinite Horizon Stochastic Optimal Control Problems with Running Maximum Cost. SIAM Journal on Control and Optimization, Society for Industrial and Applied Mathematics, 2018, 56 (5), pp.3296-3319. �10.1137/17M115253X�. �hal-01585766v3�

INFINITE HORIZON STOCHASTIC OPTIMAL CONTROL PROBLEMS WITH RUNNING MAXIMUM COST

AXEL KR ¨ONER, ATHENA PICARELLI, AND HASNAA ZIDANI

Abstract. An infinite horizon stochastic optimal control problem with running maximum cost is considered. The value function is character- ized as the viscosity solution of a second-order Hamilton-Jacobi-Bellman (HJB) equation with mixed boundary condition. A general numerical scheme is proposed and convergence is established under the assump- tions of consistency, monotonicity and stability of the scheme. These properties are verified for a specific semi-Lagrangian scheme.

1. Introduction

In this paper we consider infinite horizon stochastic optimal control prob- lems with cost in a maximum form of the following type:

(1.1)











u∈Uinf E

s∈[0,∞)max e^−λsg(X_x^u(s))

, subject to

dX_x^u(s) =b(X_x^u(s), u(s))ds+σ(X_x^u(s), u(s))dB(s), s∈[0,∞), X_x^u(0) =x∈R^d

with p-dimensional Brownian motion B(·). The control u belongs to a set U of progressively measurable processes with values in a given compact set U ⊂R^m. The functionsg:R^d→R,b:R^d×U →R^dandσ:R^d×U →R^d×p, and the discount factorλ >0 are supposed to be known. Control problems of this type can be used for the characterization of viable and invariant sets and arise in the study of some path-dependent options in finance (lookback, Russian options). The study of this problem is also motivated by some engineering applications, see for instance [29, 3].

In the deteministic case, problems with supremum costs and finite time horizon have been extensively studied in the literature, we refer for instance to [8, 9, 2, 10] where the value function is characterized as unique solution of a Hamilton-Jacobi-Bellman (HJB) equation with an obstacle. The value function is also analysed within the viability framework in [34, 35], and linearization techniques for such problems are presented in [23].

The case of deterministic control problems with supremum costs and in- finite horizon has been also considered in [8, 20, 21]. Note that in this case it is shown that the value function satisfies a stationary HJB equation with an obstacle. However, unlike the finite horizon case, the HJB equation

The first author acknowledges the hospitality and support of the COMMANDS team at ENSTA ParisTech (Palaiseau, France) where, during a research visit, this project started.

1

may have several viscosity solutions. The only characterization of the value function is obtained by the optimality principles derived from the dynamic programming principle (DPP). In [1, 21], a control problem without dis- count factor is considered and the value function of such a problem is shown to be the limit of value functions associated to some control problems with maximum running cost in finite horizon.

In the stochastic setting with finite time horizon, the characterization of the value function is studied in [4, 5, 7, 8, 11]. We refer also to [26] where some special cases of control problems with running costs are solved by using the DPP. We point out that the stochastic framework presents specific difficulties coming from the non commutativity between the expectation and the maximum operator. In [4, 5, 7, 8] the characterization of the value function is obtained by considering an L^p approximation technique where the maximum cost is approximated by a sequence ofL^p costs using the fact that for anya, b≥0

max(a, b)'(a^p+b^p)^1p (forp→ ∞).

Note that this technique can be used only when the cost function g is po- sitive. In [11], the DPP and the HJB equation are derived directly for the maximum running cost problem without using the L^p approximation. In addition, a numerical scheme based on a Semi-Lagrangian (SL) approach is analysed for the computation of the value function.

In the present work, we consider the case of stochastic control problem with supremum running cost in infinite horizon. The cost function involves also a discount factorλthat is positive. We are interested in the character- ization of the value function and its numerical approximation. By using the same viscosity arguments as in [11], we show that the value function satisfies a HJB equation with a boundary condition involving an oblique derivative.

Unlike the finite time horizon case (see [4, 7, 11]), this HJB equation fails to be a good characterization of the value function as it may admit several trivial solutions.

To garantee the uniqueness, we complete the HJB system by a Dirich- let boundary condition. While the oblique derivative boundary condition is understood in the viscosity sense [28], the Dirichlet condition is consid- ered pointwise. A similar idea has been also used in [25], however in that paper, the uniqueness result is proved by using some arguments of non- smooth analysis that are valid only under some strong stability assumptions on the differential process. Here, we use only PDE arguments and prove the uniqueness result without assuming additional assumptions on the state.

The second part of the paper is devoted to the numerical approximation.

Thereby we follow some ideas developed in [11] for the corresponding finite horizon problem. The extension is not straightforward since we are now dealing with a stationary equation with mixed boundary conditions.

First, by using the framework of Barles-Souganidis [6], we obtain a conver- gence result for a general class of numerical schemes satisfying some adequate monotonicity, stability and consistency properties. Then, we introduce a SL scheme and prove its convergence. Recall that SL schemes have been intro- duced in [16] for first order Hamilton-Jacobi equations and then extended to the second order case in [15, 18, 19, 32, 33]. For time-dependent equa- tions with an oblique boundary condition (that appears in the case of finite horizon control problems), the SL method has been investigated in [11].

For the stationary equations as considered in the present work, the SL schemes are formulated as fixed-point problems. Let us point out that the presence of mixed boundary conditions arises some new difficulties in the analysis of this fixed point problem. To deal with these difficulties, the numerical scheme we propose couples the classical SL scheme with additional projection steps on the boundary taking into account the overstepping of the domain which is typical in such a wide stencil scheme. We prove that our scheme is stable, consistent and monotone. We analyse also the fixed-point operator in an adequate space where the fixed-point algorithm converges.

The paper is organized as follows: Section 2 introduces the problem and the main assumptions. Sections 3 and 4 are devoted to the characteriza- tion of the value function by the appropriate HJB equation: the DPP is established, the HJB equation is derived and uniqueness proven by a strong comparison principle. In Section 5 the numerical approximation is discussed and a general convergence result is provided. The SL scheme is presented in Section 5.2 and its convergence properties are investigated. A numerical test in two dimensions is presented in Section 6.

2. Formulation of the problem

Let (Ω,F,P) be a probability space,{ F_t, t≥0}a filtration onFandB(·) a {F_t}_t≥0-Brownian motion in R^p, p ≥ 1. Let U be a set of progressively measurable processes with values in a compact set U ⊂R^m (with m ≥1).

For every control inputu∈ U, and everyx∈R^d, we consider the stochastic differential equation:

(2.1)

(dX_x^u(s) =b(X_x^u(s), u(s))ds+σ(X_x^u(s), u(s))dB(s), s∈[0,∞), X_x^u(0) =x.

Throughout the paper we make the following assumptions on the coefficients in (2.1):

(H1) b: R^d×U →R^d and σ:R^d×U → R^d×p are continuous functions.

There existsC₀ ≥0 such that for anyx, y∈R^dandu∈U, we have:

|b(x, u)−b(y, u)|+|σ(x, u)−σ(y, u)| ≤C₀|x−y|.

Proposition 2.1. Let assumption (H1) hold. Then, for any x ∈ R^d and u∈ U there exists a unique strong solution X_x^u(·) of (2.1). Moreover, there

exists C≥0 such that E

θ∈[0,Tmax]

|X_x^u(θ)−X_x^u0(θ)|

≤Ce^CT|x−x⁰| (2.2)

for any u∈ U, T >0 and x, x⁰ ∈R^d.

The proof of the above result can be found in [37, p. 42, Thm. 6.3] and [36, p.14]. Now, consider a cost functiongsatisfying the following condition:

(H2) g:R^d → R is Lipschitz continuous and bounded, i.e., there exist M_g, L_g ≥0 such that

|g(x)−g(y)| ≤Lg|x−y|, |g(x)| ≤Mg, ∀x, y∈R^d.

For given discount factor λ > 0 and initial position x ∈ R^d the infinite horizon optimal control problem reads as

u∈Uinf J(x, u) :=E

max

s∈[0,∞)e^−λsg(X_x^u(s))

. (2.3)

We will denote by v:R^d→Rthe associated value function, i.e.

v(x) := inf

u∈UJ(x, u).

(2.4)

3. Dynamic Programming Principle

In the next sections, we aim to find a characterization of the value func- tion v as solution of a PDE. The presence of the maximum operator inside the expectation makes the cost J in (2.3) non-Markovian and it is well- known that this prevents to establish a DPP, which is the first fundamental result towards the HJB characterisation. A classical strategy to overcome this difficulty consists in adding an auxiliary variableythat, roughly speak- ing, gets rid of the non-Markovian component of the cost. Let us define the auxiliary value function ϑ:R^d×R→R

ϑ(x, y) := inf

u∈UE

s∈[0,∞)max e^−λsg(X_x^u(s))∨y

, which satisfies, for any x∈R^d,

ϑ(x, g(x)) =v(x),

where v is defined as in (2.4). Consequently, if the value function ϑ of the auxiliary problem is known, one can immediately recover the original value functionv. Therefore, we consider onlyϑin the sequel of the paper.

We start by proving a continuity result for the auxiliary value function.

Proposition 3.1. Let assumptions (H1)-(H2) be satisfied. Then the value functionϑ is uniformly continuous in R^d×R. Moreover,

ϑ(x, Mg) =Mg and |ϑ(x, y)| ≤Mg

for any (x, y)∈R^d×R with|y| ≤M_g.

Proof. The last statement follows directly by the definition of ϑ and the bound ong given by assumption (H2).

By the very definition ofϑwe have that for any (x, y),(ξ, η)∈R^d×R

|ϑ(x, y)−ϑ(ξ, η)|

≤sup

u∈UE

max

s∈[0,∞)e^−λsg(X_x^u(s))∨y− max

s∈[0,∞)e^−λsg(X_ξ^u(s))∨η

≤sup

u∈UE

s∈[0,∞)max

e^−λsg(X_x^u(s))−e^−λsg(X_ξ^u(s)) ∨

y−η

≤sup

u∈UE

s∈[0,∞)max

e^−λsg(X_x^u(s))−e^−λsg(X_ξ^u(s))

+ y−η

For anyT >0 we have that (recalling thatg is bounded)

E

s∈[0,∞)max e^−λs

g(X_x^u(s))−g(X_ξ^u(s))

≤E

s∈[0,Tmax]e^−λs

g(X_x^u(s))−g(X_ξ^u(s))

+ max

s∈[T ,∞)e^−λs

g(X_x^u(s))−g(X_ξ^u(s))

≤E

s∈[0,Tmax]e^−λs

g(X_x^u(s))−g(X_ξ^u(s))

+E

s∈[T,∞)max 2M_ge^−λs

=E

s∈[0,Tmax]e^−λs

g(X_x^u(s))−g(X_ξ^u(s))

+ 2Mge^−λT.

For the first term we have the classical estimates (see Theorem 2.1) E

h

s∈[0,Tmax]e^−λs

g(X_x^u(s))−g(X_ξ^u(s)) i

≤LgCe^CT|x−ξ|.

So putting everything together we get that for everyT

|ϑ(x, y)−ϑ(ξ, η)| ≤L_gCe^CT|x−ξ|+ 2M_ge^−λT +|y−η|.

Now, for any ε > 0 we can fix T ≡ T(ε) such that 2M_ge^−λT < ε/3. It follows that we can choose a δ =δ(ε, T(ε)) such that if |(x, y)−(ξ, η)| ≤δ then

|ϑ(x, y)−ϑ(ξ, η)|< ε.

Remark 3.2. In many cases, under assumptions (H1)-(H2), the value func- tion is expected to be H¨older continuous. This holds true in the case of finite horizon problems for both integral and maximum running costs (see [37, Proposition 3.1] and [11, Proposition 2.2], respectively). In infinite time horizon, such a property can be recovered only for a discount factor suffi- ciently big ensuring a dissipative property of the process e^−λtX_x^u(t) (see for instance [31, Theorem 3.4] for the case of integral running cost). However, the proof of Proposition 3.1 shows thatϑis Lipschitz continuous with respect to the auxiliary variabley. This will turn out to be a fundamental property to

be preserved when defining a numerical scheme in order to get convergence (see assumption (H3.a) in Section 5).

Thanks to the presence of the auxiliary variableywe are able to state the DPP forϑ. Let us define fort≥0

Y_x,y^u (t) := max

s∈[0,t]e^−λsg(X_x^u(s))∨y.

Following the arguments in [14] and thanks to the uniform continuity of ϑone has:

Theorem 3.3(DPP). Let assumptions (H1)-(H2) be satisfied. For (x, y)∈ R^d×R and any finite stopping timeθ≥0 there holds

ϑ(x, y) = inf

u∈UE h

e^−λθϑ(X_x^u(θ), e^λθY_x,y^u (θ)) i

.

Proof. The proof of the DPP is based on the fact that for every u∈ U, we have:

s∈[0,∞)max e^−λsg(X_x^u(s))∨y= max

s∈[θ,∞)e^−λsg(X_x^u(s))_

s∈[0,θ]max e^−λsg(X_x^u(s))∨y

| {z }

=:Yx,y(θ)

=e^−λθ max

s∈[0,∞)e^−λsg(X_X^uu

x(θ)(s))_

Yx,y(θ)

=e^−λθ

s∈[0,∞)max e^−λsg(X_X^uu

x(θ)(s))_

e^λθY_x,y(θ) . By the tower property of the expectation, we conclude the proof. We refer

to [14] and [36] for more detailed arguments.

4. The Hamilton-Jacobi-Bellman equation

The value function ϑ can be characterized in terms of a solution of a second-order HJB equation.

Theorem 4.1 (Second-order HJB equation). Under assumptions (H1)- (H2), the value function ϑis a continuous viscosity solution of

(4.1) λϑ+H(x, y, D_xϑ, ∂_yϑ, D_x²ϑ) = 0 for x∈R^d, y > g(x),

−∂_yϑ= 0 for x∈R^d, y=g(x)

with λ >0 and Hamiltonian H:R^d×R×R^d×R×R^d×d→Rdefined by H(x, y, p, q, P) = sup

u∈U

−b(x, u)·p−1

2T r[(σσ^T)(x, u)P]

−λyq.

(4.2)

We recall the notion of viscosity solution for second-order HJB equation.

Definition 4.2. [17, Definition 7.4]Let O be a locally compact set in R^d+1. A USC function ϑ (resp. LSC function ϑ) on O is a viscosity sub-solution (resp. super-solution) of (4.1), if for every function ϕ ∈ C²(O) at every

maximum (resp. minimum) point x of ϑ−ϕ (resp. ϑ−ϕ) the following inequalities hold

( λϕ+H(x, y, D_xϕ, ∂_yϕ, D_x²ϕ)≤0 in O, min(λϕ+H(x, y, D_xϕ, ∂_yϕ, D²_xϕ),−∂_yϕ)≤0 on ∂O, resp.

( λϕ+H(x, y, Dxϕ, ∂yϕ, D_x²ϕ)≥0 in O, max(λϕ+H(x, y, D_xϕ, ∂_yϕ, D²_xϕ),−∂_yϕ)≥0 on∂O.

A continuous function ϑis called a viscosity solution of (4.1) if it is both a sub- and super-solution.

Proof of Theorem 4.1. Let

O:={(x, y)∈R^d+1 :y≥g(x)}.

We first check that ϑ is a viscosity sub-solution. Let ϕ∈ C²(O) such that ϑ−ϕ attains a maximum at point (¯x,y)¯ ∈ O. Without loss of generality we can assume that (¯x,y) is a strict local maximum point (let us say in a¯ ball of radius r > 0 centered in (¯x,y)) and¯ ϑ(¯x,y) =¯ ϕ(¯x,y). Thanks to¯ Theorem 3.3, for any u ∈ U and for any sufficiently small stopping time θ=θ^u, we have:

ϕ(¯x,y) =¯ ϑ(¯x,y)¯ ≤E

e^−λθϑ X_x^u_¯(θ), e^λθY_x,¯¯^u_y(θ)

≤E

e^−λθϕ X_¯x^u(θ), e^λθY_¯x,¯^u_y(θ)

. (4.3)

Two cases will be considered depending on if the point (¯x,y) belongs to the¯ boundary ofO or not.

— Case 1: g(¯x) <y. Consider a constant control¯ u(s) ≡u ∈U. From the continuity of g and the a.s. continuity of the sample paths it follows that for a.e. ω ∈ Ω there exists ¯s(ω) > 0 such that g(X_x¯^u(s))e^−λs < y¯ if s∈[0,¯s(ω)). Given h >0, let ¯θ be the following stopping time:

(4.4)

θ¯:= inf

s >0 : (X_x^u_¯(s), e^λsY_¯x,¯^u_y(s))∈/ B((¯x,y), r)¯ ∧ h

∧inf

s >0 :e^−λsg(X_x^u_¯(s))≥y¯

(where B((¯x,y), r) denotes the ball of radius¯ r > 0 centered at (¯x,y) and¯ a∧b= min(a, b)). One can easily observe that a.s. ¯θ >0 and Y_x,¯¯^u_y(¯θ) = ¯y, then by (4.3)

ϕ(¯x,y)¯ ≤E

e^−λθ¯ϕ X_x¯^u(¯θ), e^λθ¯y¯

∀u∈U.

By applying Ito’s formula [37, Theorem 5.5], and thanks to the smoothness of ϕ, we get:

0≥E Z θ^¯

0

−d

e^−λsϕ(X_x^u_¯(s), e^λsy)¯

=E Z θ^¯

0

n

λϕ(X_¯x^u(s), e^λsy)¯ −b(X_x^u_¯(s), u)Dxϕ(X_x¯^u(s), e^λsy)¯

−1

2T r[σσ^T(X_¯x^u(s), u)D²_xϕ(X_x^u_¯(s), e^λsy)]¯ −λ¯ye^λs∂yϕ(X_x^u_¯(s), e^λsy)¯o e^−λsds

. Observing that the stopping times

infn

s >0 : (X_x^u_¯(s), e^λsY_x,¯¯^u_y(s))∈/ B_r(¯x,y)¯ o

and infn

s >0 :e^−λsg(X_x^u_¯(s))≥y¯o are a.s. strictly greater than 0, for a.e. fixed ω one obtains ¯θ = h for a sufficiently small h in (4.4). Dividing byh >0, one gets

E 1

h Z θ^¯

0

n

λϕ(X_x¯^u(s), e^λsy)¯ −b(X_¯x^u(s), u)Dxϕ(X_x^u_¯(s), e^λsy)¯

−1

2T r[σσ^T(X_x^u_¯(s), u)D_x²ϕ(X_¯x^u(s), e^λsy)]¯ −λ¯ye^λs∂yϕ(X_¯x^u(s), e^λsy)¯ o e^−λsds

≤0.

By the dominate convergence theorem, taking the limit for h going to 0 inside the expectation and applying the mean value theorem, it follows

λϕ(¯x,y)¯ −b(¯x, u)Dxϕ(¯x,y)¯ −1

2T r[σσ^T(¯x, u)D²_xϕ(¯x,y)]¯ −λ¯y∂yϕ(¯x,y)¯ ≤0.

Finally, thanks to the arbitrariness of u∈U: λϕ(¯x,y) + sup¯

u∈U

−b(¯x, u)D_xϕ(¯x,y)¯ −1

2T r[σσ^T(¯x, u)D²_xϕ(¯x,y)]¯

−λ¯y∂_yϕ(¯x,y)¯ ≤0.

— Case 2: g(¯x) = ¯y. Let us assume that −∂_yϕ(¯x,y)¯ >0, otherwise the conclusion is straightforward.

As in the previous case, we consider a constant control u(s) ≡ u ∈ U. Thanks to the continuity of the sample paths and the smoothness of ϕ, for a.e. ω there is a time ¯s(ω)>0 and η >0 such that:

ϕ(X_¯x^u(s), e^λsy)≤ϕ(X_¯x^u(s), e^λsy)¯ ∀s∈[0,s], y¯ ∈[¯y,y¯+η).

Let ¯θ be the stopping time given by:

θ¯:= inf

s >0 : (X_¯x^u(s), e^λsY_x,¯¯^u_y(s))∈/B((¯x,y), r)¯ ∧inf{s >0 :∂yϕ(X_x^u_¯(s),y)¯ ≥0}

∧inf

s >0 : e^−λsg(X_x^u_¯(s))∈/ [¯y,y¯+η) ∧ h.

By (4.3) we have ϕ(¯x,y)¯ ≤ E

e^−λθ¯ϕ X_x^u_¯(¯θ), e^λθ¯y¯

, which implies (as in Case 1):

λϕ(¯x,y) + sup¯

u∈U

n−b(¯x, u)Dxϕ(¯x,y)¯ −1

2T r[σσ^T(¯x, u)D_x²ϕ(¯x,y)]¯ o

−λ¯y∂yϕ(¯x,y)¯ ≤0.

In conclusion at (¯x,y)¯ ∈∂O we have

min λϕ+H(¯x,y, D¯ _xϕ, ∂_yϕ, D_x²ϕ),−∂_yϕ

≤0, and ϑis a viscosity sub-solution of equation (4.1).

It remains to prove that ϑ is a viscosity super-solution of (4.1). Let ϕ ∈ C²(O) be such that ϑ−ϕ attains a minimum at point (¯x,y)¯ ∈ O.

Without loss of generality we can always assume that (¯x,y) is a strict local¯ minimum point in a ball B((¯x,y), r) and¯ ϑ(¯x,y) =¯ ϕ(¯x,y). We consider¯ again the two cases:

— Case 1: g(¯x)<y. We assume by contradiction that¯ λϕ+H(¯x,y, D¯ _xϕ, ∂_yϕ, D²_xϕ)<0.

By using continuity arguments we can also state that (4.5) λϕ+H(·,·, D_xϕ, ∂_yϕ, D²_xϕ)≤0

in a neighborhood B((¯x,y), r¯ ₁) of (¯x,y) for some¯ r₁ >0. Moreover, thanks to the continuity ofg, if ¯y−g(¯x) =:ρ >0 we can findr2(ρ)>0 andT(ρ)>0 such that

t∈[0,Tmax], x∈B(¯x,r2)

e^−λtg(x)−g(¯x)≤ ρ 2 and we have

t∈[0,Tmax], x∈B(¯x,r2)

e^−λtg(x)∨y¯= ¯y.

For anyu∈ Uwe define the stopping timeθ^uas the first exit time of the pro- cess (X_x¯^u(s), e^λsY_¯x,¯^u_y(s)) from the ballB((¯x,y), R) for¯ R:= min(r, r₁, r₂)>0, i.e.

θ^u:= infn

s >0 : (X_x^u_¯(s), e^λsY_x,¯¯^u_y(s))∈/B((¯x,y), R)¯ o

∧T.

Applying Ito’s formula and taking the expectation we get ϕ(¯x,y)¯ −E

e^−λθuϕ(X_x^u_¯(θ^u), e^λθuY_¯x,¯^u_y(θ^u))

=ϕ(¯x,y)¯ −E

e^−λθuϕ(X_x^u_¯(θ^u), e^λθuy)¯

=E Z θ^u

0

n

λϕ(X_¯x^u(s), e^λsy)¯ −b(X_x¯^u(s), u)D_xϕ(X_x^u_¯(s), e^λsy)¯

−1

2T r[σσ^T(X_x^u_¯(s), u)D_x²ϕ(X_x^u_¯(s), e^λsy)]¯ −λ¯ye^λs∂yϕ(X_¯x^u(s), e^λsy)¯ o

e^−λsds

≤0, that leads to

ϑ(¯x,y) =¯ ϕ(¯x,y)¯ ≤E

e^−λθuϕ(X_x^u_¯(θ^u), e^λθuY_¯x,¯^u_y(θ^u))

. (4.6)

The continuity of the sample paths implies that either θ^u=T or

(X_x^u_¯(θ^u), e^λθuY_¯x,¯^u_y(θ^u))∈∂B((¯x,y), R) a.s.¯ . Since (¯x,y) is a strict minimum point we have¯

minn

(ϑ−ϕ) : (x, y)∈∂B((¯x,y), R)¯ [

B((¯x,y), R)¯ ∩y=e^λTy¯o

=:η >0, and hence

ϑ(¯x,y)¯ ≤ϕ(X_¯x^u(θ^u), e^λθuY_x,¯¯^u_y(θ^u))≤ϑ(X_x^u_¯(θ^u), e^λθuY_¯x,¯^u_y(θ^u))−η.

Substituting in (4.6), for anyu∈ U one has ϑ(¯x,y)¯ ≤E

e^−λθuϑ(X_¯x^u(θ^u), e^λθuY_x,¯¯^u_y(θ^u))

−ηe^−λT.

Since η and T do not depend onu, a minimisation over u contradicts the DPP.

— Case 2: g(¯x) = ¯y. Assume by contradiction that

−∂_yϕ(¯x,y)¯ <0 and λϕ+H(¯x,y, D¯ xϕ, ∂yϕ, D²_xϕ)<0.

We can again define r₁ > 0 such that (4.5) is satisfied in B((¯x,y), r¯ ₁).

Moreover there exists ˜r2 >0 such that

ϕ(ξ, ζ)≤ϕ(ξ, ζ⁰)

for any (ξ, ζ),(ξ, ζ⁰)∈B((¯x,y),¯ r˜₂) such thatζ≤ζ⁰. For anyu∈ Uwe define the stopping timeθ^u as the first exit time of the process (X_x^u_¯(s), e^λsY_x,¯¯^u_y(s)) and (X_x^u_¯(s), e^λsy) from the ball¯ B((¯x,y),¯ R) for ˜˜ R:= min(r, r1,r˜2) >0. As for Case 1, we can still say that a.s.

ϑ(X_x¯^u(θ^u), Y_x,¯¯^u_y(θ^u))≥ϕ(X_x¯^u(θ^u), Y_x,¯¯^u_y(θ^u)) +η⁰. for someη⁰ >0 not depending onu. Therefore, observing that

e^λsY_x,¯¯^u_y(s)≥e^λsy¯ for any s≥0 and using Ito’s formula, we get

ϑ(¯x,y) =¯ ϕ(¯x,y)¯ ≤E h

ϕ(X_x¯^u(θ^u), e^λθuy)¯ i

≤E h

ϕ(X_x^u_¯(θ^u), e^λθuY_¯x,¯^u_y(θ^u))i

≤E h

ϑ(X_x¯^u(θ^u), e^λθuY_x,¯¯^u_y(θ^u) i

−η⁰,

which yields again to a contradiction of the DPP.

Remark 4.3. Unlike the case of finite time horizon (see [4, 11]), the HJB equation (4.1) obtained by the DPP does not admit a unique solution. In- deed, one can observe that the function identically equal to zero is always a solution of (4.1). To overcome this problem we will restrict our domain to

D:={(x, y)∈R^d+1 :y > g(x), y ∈(−M_g, Mg)}.

and add a Dirichlet boundary condition for y=M_g.

Let

Γ₁ :={(x, y)∈D:y=M_g}; Γ₂:={(x, y)∈D:y=g(x)}. (4.7)

Then the value functionϑis a viscosity solution of the following problem λϑ+H(x, y, Dxϑ, ∂yϑ, D²_xϑ) = 0 inD,

(4.8a)

−∂_yϑ= 0 on Γ2

(4.8b)

and satisfies additional

ϑ=Mg on Γ1

(4.8c)

in a strong sense. We point out that in our case the strong constant Dirich- let boundary condition onΓ₁ is compatible with the homogeneus derivative condition onΓ₂. This prevents possible problems related with mixed bound- ary conditions at the junctions where different components of the boundary cross.

Now, we can establish the following comparison result:

Theorem 4.4. Let assumptions (H1)-(H2) be satisfied and letϑ∈U SC(D), ϑ∈LSC(D)respectively a bounded sub- and super-solution to equation(4.8) in D such that

ϑ≤Mg and ϑ≥Mg on Γ1. Then ϑ≤ϑ in D.

The proof can be obtained by a modification of the arguments in [22, Theorem 2.1] which shows how to deal with the derivative conditions; see also [24]. We report here the main steps.

Proof. We set

˜b(x, y, u) :=

b(x, u) λy

∈R^d+1 and ˜σ(x, y, u) :=

σ(x, u) 0. . .0

∈R^(d+1)×p. Further, let x denote the variable in the augmented state space R^M with M :=d+ 1, i.e. x≡(x, y)∈R^M and x_M =y. Then the HamiltonianH in (4.2) is given by

H(x, p, P) := sup

u∈U

−˜b(x, u)·p−1

2T r[˜σσ˜^T(x, u)P]

.

Thanks to the Lipschitz continuity of the function g that defines the boundary Γ2, we can observe that there exists µ > 0 such that for any z∈Γ₂ we have

(4.9) [

0≤ξ≤µ

B(z−ξ, ξµ)⊂D^C,

whereD^C denotes the complementary of the setD; this corresponds to [22, condition (2.9)]. Let us define for positiveα and β the functions

ϑ_α,β(x) :=ϑ(x)−α(M_g−x_M)−β, ϑ_α,β(x) :=ϑ(x) +α(M_g−x_M) +β.

Thanks to the non-negativity of (Mg−x_M) for y∈D we have ϑ_α,β ≤ϑ and ϑ_α,β ≥ϑ on D and in particular

ϑ_α,β ≤M_g and ϑ_α,β ≥M_g on Γ₁.

With an abuse of notation, we denote byDϑ, Dϑ, D²ϑ, D²ϑthe correspond- ing elements of the semijets ofϑand ϑ, see [17, Section 2]. We have

λϑ_α,β+H(x, Dϑ_α,β, D²ϑ_α,β)−λϑ−H(x, Dϑ, D²ϑ)≤ −λβ+λαMg

and

λϑ_α,β+H(x, Dϑ_α,β, D²ϑ_α,β)−λϑ−H(x, Dϑ, D²ϑ)≥λβ−λαM_g. Moreover, we have

−∂_xMϑ_α,β=−∂_xMϑ−α, −∂_xMϑ_α,β=−∂_xMϑ+α

Therefore for anyβ >0 andα=α(β)>0 small enough,ϑ_α,β (resp. ϑα,β) is a sub-solution (resp. super-solution) to (4.8) with the following modified boundary condition onΓ₂:

(4.10) −∂_xMϑ+α≤0 (resp. −∂_xMϑ−α≥0).

In the sequel we prove a comparison principle between the sub-solutionϑ_α,β and super-solution ϑ_α,β of the modified problem. The comparison result between ϑ and ϑ follows from taking the limit of α, β → 0. In order to simplify the notation we denote the sub- and super-solution of the modified problem also byϑand ϑ.

Let

Φ_γ(x) :=ϑ(x)−ϑ(x)−2γlog(1 +|(x₁. . . x_M−1)|²).

In order to simplify the notation, from now on we will denote |x|_M−1 =

|(x₁. . . x_M−1)|. Thanks to the boundedness and the upper semicontinuity of ϑ−ϑ, the function Φ_γ admits a maximum point ˆx_γ= ˆxin D.

Let us assume that there exists a sequence {γ_k}_k≥0 such that γ_k → 0 and the points ˆx_γk approach the boundary Γ₁. In this case for any x ∈ D we have

(4.11) ϑ(x)−ϑ(x) = lim sup_k→∞ϑ(x)−ϑ(x)−2γ_klog(1 +|x|²

M−1)

≤lim sup_k→∞ϑ(ˆx)−ϑ(ˆx)≤0.

the last inequality follows by the fact that any convergent subsequence of points{ˆxγ_k}converges to a point ofΓ1 where one hasϑ−ϑ≤0 by assump- tion. We obtain the same result if Φ_γk(ˆx)≤0 for allk.

Therefore, in the sequel, we assume that there exists a ¯γ small enough such thatd(ˆx, Γ₁) =ρ >0 and ϑ(ˆx)−ϑ(ˆx)≥Φ_γ(ˆx)> ρfor any γ ≤γ¯ and some ρ >0 (withd(·, Γ₁) we have denoted the Euclidean distance function toΓ1).

Moreover, by classical doubling variable techniques it is standard to obtain a contradiction if ˆxbelongs to the interior ofD(see for instance [27, Theorem

7.3]). Such techniques have to be modified if ˆxbelongs toΓ2. Thanks to the property (4.9) of the domain, the existence of a family ofC² test functions {w_ε}_ε>0 can be proven as in [22, Theorem 4.1]. Among the other properties, {w_ε}:R^M ×R^M →R satisfies:

wε(x, x)≤ε (4.12)

w_ε(x, y)≥C|x−y|² (4.13) ε

−∂x_Mwε(x, y)≥ −C|x−y|²

ε ifx∈Γ2∩B(ˆx, η), y∈B(ˆx, η) (4.14)

−∂y_Mwε(x, y)≥0 ify ∈Γ2∩B(ˆx, η), x∈B(ˆx, η) (4.15)

forε >0 and someη >0 small enough.

Applying the doubling variables procedure we define

Φε(x, y) :=ϑ(x)−ϑ(y)−γlog(1 +|x|²_M−1)−γlog(1 +|y|²_M−1)−wε(x, y)− |x−x|ˆ⁴. and we denote by (xε, yε) its maximum point. Thanks to properties (4.12)

and (4.13), we have for ε→0 that

(4.16) xε, yε→xˆ and |x_ε−yε|²

ε →0.

Thus for εsmall enough we can assume thatxε, yε∈/ Γ1, however since the case ˆx∈Γ₂ is taken into account, the sequence (x_ε, y_ε) may consider points onΓ2. Takingεsmall enough we can also say thatxε, yε∈B(ˆx, η) and then we can make use of properties (4.14) and (4.15).

Ifxε∈Γ2, using (4.16), we have that forεsmall enough

−∂_x

M

w_ε(x_ε, y_ε) +γlog(1 +|x_ε|²

M−1) +|x_ε−x|ˆ⁴

≥ −C|x_ε−yε|²

ε −4|x_ε−x|ˆ³>−α.

Similarly if y_ε∈Γ₂ one has

−∂_y

M

−wε(xε, yε)−γlog(1 +|y_ε|²_M−1)

≤0< α forεsmall enough.

This means that, considering wε(·, y_ε) + γlog(1 +| · |²

M−1) + | · −x|ˆ⁴ and

−w_ε(x_ε,·)−γlog(1 +| · |²

M−1) as test functions for ϑandϑrespectively, for sufficiently small values of ε the derivative boundary conditions in x_ε and yε can be neglected and one can only consider in Γ2∪D

λϑ+H(x_ε, Dϑ, D²ϑ)≤0 and λϑ+H(y_ε, Dϑ, D²ϑ)≥0

in the viscosity sense. Thanks to the properties of H and w_ε, this leads to a contradiction using the arguments in [22, 27].

Corollary 4.5. The value functionϑis the unique bounded continuous vis- cosity solution to equation (4.8) in D.

Starting by its HJB characterization inD,ϑcan be extended in a unique way to the full space R^d×Rby

(4.17) ϑ(x, y) =y for any y≥M_g

ϑ(x, y) =ϑ(x, g(x)) for any y≤g(x).

However, we can observe that, in order to characterize the original value function v, the way we extend ϑfor y > M_g does not matter and only the values ofϑ fory=g(x)≤M_g are relevant.

5. Numerical approximation

In this section, convergence is proven for a general class of numerical schemes. Then, we show that a fully-discrete semi-Lagrangian method ap- plied to the problem under consideration belongs to this class.

Let BC denote the space of bounded and continuous functions in R^d+1 equipped with the L^∞-norm. Further, we introduce the discretization pa- rameters

∆x= (∆x₁, . . . , ∆x_d)∈(R^>0)^d, ∆y >0, ρ= (∆x, ∆y) and denote the corresponding mesh by

G^ρ:={(xi, yj) = (i∆x, j∆y), (i, j)∈Z^d×Z}, wherei∆x= (i₁∆x₁, . . . , i_N∆x_N), i∈Z^d. We set

j_x:= min{j∈Z|j∆y≥g(x)}, (5.1)

jM := min{j∈Z|j∆y≥Mg} (5.2)

and introduce a projection operator (5.3) Π^Gρ(φ)(x_i, y_j) :=







y_j ifj≥j_M φ(xi, yj) ifjM > j ≥jxi

φ(x_i, y_jxi) ifj < j_xi

for functions φ ∈ BC. We aim to define a general approximation of the value functionϑ. We start considering a general scheme

S^ρ(x_i, y_j, φ_i,j, φ) = 0 that approximates the equation

λφ+H(x, y, D_xφ, ∂_yφ, D_x²φ) = 0

at node (x_i, y_j), withH defined by (4.2). Hereφ_i,j =φ(x_i, y_j) andφdenote the values of φ at nodes different from (x_i, y_j). S^ρ may represent a finite difference operator (see [13, 12, 30]), or a semi-Lagrangian (SL) scheme ([32, 15, 19]). The main idea of the numerical method described here is to mix the use of a standard scheme for (4.8a), together with a projection step on∂Din order to get the desired boundary conditions. Let us point out that a similar method was introduced for treating oblique derivative boundary conditions, i.e. the condition we have in Γ₂, in [5] for the case g(x) ≡ |x|

and in [11] for a general Lipschitz continuous function g. In addition, here

the numerical solution has also to satisfy the Dirichlet condition on Γ1. We defineV on G^ρsuch that

(5.4)







Vi,j =yj ifyj ≥Mg,

S^ρ(x_i, y_j, V_ij, Π^Gρ(V)) = 0 ifg(x_i)≤y_j < M_g, Vi,j =Vi,j_xi ifyj < g(xi)

and we denote byV^ρits continuous extension by bilinear interpolation. We assume the grid G^ρ aligned with the domain [−M_g, Mg], so that we have V^ρ=M_g on Γ₁.

5.1. General convergence result. In order to prove the convergence of V^ρ to the unique viscosity solution ϑ of (4.8) we will make use of the ar- guments introduced by Barles and Souganidis in [6]. These make use of the properties of stability, consistency and monotonicity of the scheme. We assume that the scheme defined by (5.4) satisfies the following assumptions:

(H3.a) Stability: for anyρthe scheme (5.4) admits a solutionV^ρ∈BC(D).

Moreover, there existM, L≥0 such that

V^ρ(x, y)≤M and |V^ρ(x, y)−V^ρ(x, y⁰)| ≤L|y−y⁰| for any (x, y),(x, y⁰)∈Dand ρ >0.

(H3.b) Consistency: the schemeS^ρis consistent with respect to (4.1) inD, i.e. for all (x, y)∈D and everyφ∈C²(D) there holds

ρ→0,lim

D3(ξ,γ)→(x,y), ζ→0

S^ρ(ξ, γ, φ(ξ, γ) +ζ, φ+ζ) =λφ+H(x, Dxφ, D_x²φ).

(H3.c) Monotonicity: for everyρ,r ∈ R, (x, y)∈D, S^ρ(x, y, r, φ) depends only on the values of φin a neighborhood B_η(ρ)(x, y) of (x, y) with η(ρ)≥0 such thatη(ρ)→0 for ρ→0. For all functionφ1, φ2:R^d× R→Rwith φ1 ≥φ2 onBρ(x, y), there holds

S^ρ(x, y, r, φ₁)≤S^ρ(t, x, y, r, φ₂).

(5.5)

Remark 5.1. Stability, consistency and monotonicity are the classical re- quirements to prove convergence of numerical schemes in the framework of viscosity solutions. However, in our case some slight modifications with respect to the original assumptions considerend in[6] are necessary. In par- ticular, the additional regularity assumption with respect to the variable y turns out to be fundamental to deal with Dirichlet boundary conditions, while the reason for our formulation of the monotonicity condition is the fact that the numerical scheme S^ρ defined on Dmay use some values of the function V^ρ outside the domain D.

Theorem 5.2. Let assumptions (H1)-(H2) be satisfied and let the scheme (5.4)satisfy assumption (H3). Then for ρ→0the solution V^ρ converges to the unique viscosity solution to (4.1).

Proof. Let us define for (x, y)∈D

V(x, y) := lim sup

D3(ξ,γ)→(x,y) ρ→0

V^ρ(ξ, γ), V(x, y) := lim inf

D3(ξ,γ)→(x,y) ρ→0

V^ρ(ξ, γ)

Observe that the semi-limits are well-defined thanks to the boundedness of V^ρ uniformly with respect to ρ. We start by proving that V is a viscosity sub-solution to equation (4.1).

Thanks to the Lipschitz continuity of V^ρ with respect to y (uniform with respect toρ) and the fact that, by definition,V^ρ(x, y) = M_g onΓ₁ one has for any (ξ, γ)∈D

|V^ρ(ξ, γ)−M_g|=|V^ρ(ξ, γ)−V^ρ(ξ, M_g)| ≤L|γ−M_g|, so that

lim

D3(ξ,γ)→(x,y) ρ→0

V^ρ(ξ, γ) =Mg.

Hence,V =Mg onΓ1 and the Dirichlet condition is satisfied.

Let ϕ ∈ C²(D) and let (¯x,y) be a local maximum point for¯ V −ϕ on D.

Without loss of generality we can assume that (¯x,y) is a strict local maxi-¯ mum inB_r(¯x,y)¯ ∩D for a certainr >0 andϕ≥2 sup_ρkW^ρk_∞ outside the ball B_r(¯x,y). We claim that¯

(λu+H(¯x,y, D¯ _xϕ, ∂_yϕ, D²_x)ϕ≤0 if (¯x,y)¯ ∈D, min(λu+H(¯x,y, D¯ xϕ, ∂yϕ, D²_xϕ),−∂_yϕ)≤0 if (¯x,y)¯ ∈Γ2. We follow the argument in Barles and Souganidis [6]. There exists a sequence ρ_k, (x_k, y_k) such thatρ_k→0, (x_k, y_k)→(¯x,y) for¯ k→ ∞, and

(V^ρk −ϕ)(xk, yk) = max

D

(V^ρk−ϕ) =δk→0, ask→ ∞ (5.6)

and

V^ρk(x_k, y_k)→V(¯x,y),¯ ask→ ∞.

–Case 1: (¯x,y)¯ ∈ D. For k large enough (xk, yk) ∈ D. Since g is continuous, for ρ_k small enough we can assume that y > g(x) for any (x, y) ∈ B_η(ρk)(xk, yk) (where B_η(ρk)(xk, yk) is the neighborhood that ap- pears in assumption (H3.c)). Consequently, for kbig enough

Π^Gρk(V^ρk) =V^ρk and V^ρk < ϕ+δ_k

in B_η(ρk)(x_k, y_k). By the monotonicity of the scheme (assumption (H3.c)) we further deduce

0 =S^ρk(xk, yk, V^ρk(xk, yk), Π^Gρk(V^ρk))

≥S^ρk(x_k, y_k, ϕ(x_k, y_k) +δ_k, ϕ+δ_k))

and by the consistency assumption (H3.b) we obtain that ask→ ∞ λϕ+H(¯x,y, D¯ _xϕ, ∂_yϕ, D²_xϕ)≤0.

–Case 2: (¯x,y)¯ ∈Γ₂. If−∂_yϕ(¯x,y)¯ ≤0 the sub-solution property on Γ₂ is automatically satisfied. Let us assume that −∂_yϕ(¯x,y)¯ >0.

We point out that if ¯y = g(¯x), (x_k, y_k) can also be on Γ2 and the scheme may involve valuesVm,n on some point (xm, yn) which is not in D.

If−∂_yϕ(¯x,y)¯ >0, there exists a neighbourhoodV of (¯x,y) where¯ ∂_yϕis well defined and−∂_yϕ >0. Therefore,

y≤y⁰⇒ϕ(x, y)≥ϕ(x, y⁰) ∀(x, y),(x, y⁰)∈ V. (5.7)

and, taking k large enough, B_η(ρk)(xk, yk) ⊂ V. Let (x, y)∈ B_η(ρk)(xk, yk).

Ify ≥g(x), we have

Π^Gρ(V^ρk(x, y)) =V^ρk(x, y)≤ϕ(x, y) +δk. Ify < g(x),Π^Gρ(V^ρk(x, y)) =V^ρk(x, yjx) and we have

(5.8)

Π^Gρ(V^ρk(x, y)) =V^ρk(x, y_jx)

≤ϕ(x, yjx) +δk using (5.6)

≤ϕ(x, y) +δ_k using (5.7)

For the last inequality one also need to observe that if (x, y) ∈ V and y < g(x), thanks to the continuity of g one can choose k big enough so that also (x, yjx)∈ V.

Consequently, we haveΠ^Gρ(V^ρk)≤ϕ+δ_konB_η(ρk)(x_k, y_k). Thus by mono- tonicity we have

0 =S^ρk(x_k, y_k, V^ρk(x_k, y_k), Π^Gρ(V^ρk))

≥S^ρk(xk, yk, ϕ(xk, yk) +δk, ϕ+δk)) and using consistency whenk→ ∞ we have

λϕ+H(¯x,y, D¯ _xϕ, ∂_yϕ, D²_xϕ)≤0.

(5.9)

This proves that V is a viscosity subsolution to (4.1). Analogously one can show thatV is a viscosity sub-solution. Applying the comparison principle (Theorem 4.4), it follows that V ≥ V on D. Since V ≤ V is always true by definition, it is possible to conclude thatV =V on D, which proves the

assertion.

5.2. Semi-Lagrangian scheme. In this section we introduce a semi-Lagran- gian scheme and verify that it satisfies assumptions (H3).

5.2.1. Semi-discretization. To simplify the presentation, we consider first a semi-discrete version of the scheme. Let h > 0 be the time discretization parameter of the dynamics. We define the mapping T0:BC →BC with

T0(φ)(x, y) :=

minu∈U

((1−λh) 2p

2p

X

k=1

φ

x+hb(x, u) +p

hp(−1)^kσ_bk+1

2 c(x, u), y 1−λh

)

where b·c denotes the integer part. Furthermore, let X(Mg) be the subset of BC given by

X(Mg) :=

φ∈BC ∩ Lipy φ(x, y) =y for all (x, y)∈R^d×R withMg ≤y≤2Mg(1−λh)

, where Lip_y denotes the set of functions that are Lipschitz continuous with respect toy. We introduce the operatorT0:X(Mg)→X(Mg) defined by (5.10) T₀(φ)(x, y) :=T0(Π(φ))(x, y)

whereΠ is the the continuous version of the projection Π^Gρ, i.e. : Π(φ)(x, y) :=







y∧2M_g ify≥M_g,

φ(x, y) ifg(x)≤y < Mg, φ(x, g(x)) ify≤g(x).

Remark 5.3. Compared with (5.3) here we have modified the value of the operatorΠ for y≥M_g in order to guarantee the boundedness of the opera- tor T. It is possible to verify that this does not affect the proof of the main convergence result which only concerns neighborhoords of points in D.

Observe that for functions φinX(Mg) the operatorΠ simplies to (5.11) Π(φ)(x, y) =φ(x, g(x)∨(y∧2M_g)).

Lemma 5.4. The operator T0 is well-defined.

Proof. For every φ ∈ X(M_g) the image T₀(φ) is bounded and continuous.

For L ≥ 0 let Lipy(L) ⊂ Lipy be the subset of functions with Lipschitz constantL. Then we have for φ∈Lip_y(L)

|T₀(φ)(x, y)−T0(φ)(x, y⁰)|

≤max

u∈U

(1−λh) 2p

2p

X

k=1

Π(φ)

X_x^u,k,p(h), Y_y^u(h)

−Π(φ)

X_x^u,k,p(h), Y_y^u0(h) where we use the notation

X_x^u,k,p(h) :=x+hb(x, u) +p

hp(−1)^kσ_bk+1

2 c(x, u), Y_y^u(h) := y

1−λh