On a continuous time game with incomplete information

HAL Id: hal-00326093

https://hal.archives-ouvertes.fr/hal-00326093

Submitted on 1 Oct 2008

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires

On a continuous time game with incomplete information

Pierre Cardaliaguet, Catherine Rainer

To cite this version:

Pierre Cardaliaguet, Catherine Rainer. On a continuous time game with incomplete information.

Mathematics of Operations Research, INFORMS, 2009, 34 (4), pp.769-794. �hal-00326093�

On a continuous time game with incomplete information

Pierre Cardaliaguet^∗and Catherine Rainer^† October 1, 2008

Abstract

For zero-sum two-player continuous-time games with integral payoff and incomplete information on one side, one shows that the optimal strategy of the informed player can be computed through an auxiliary optimization problem over some martingale measures.

One also characterizes the optimal martingale measures and compute it explicitely in several examples.

Key words: Continuous-time game, incomplete information, Hamilton-Jacobi equations.

MSC Subject classifications: 91A05, 49N70, 60G44, 60G42

In this paper we investigate a two-player zero-sum continuous time game in which the players have an asymmetric information on the running payoff. The description of the game involves

(i) an initial time t₀≥0 and a terminal timeT > t₀,

(ii) I integral payoffs (where I ≥2): ℓi : [0, T]×U ×V →IR fori= 1, . . . I whereU and V are compact subsets of some finite dimensional spaces,

(iii) a probability p= (p_i)_i=1,...,I belonging to the set ∆(I) of probabilities on{1, . . . , I}. The game is played in two steps: at timet₀, the indexiis chosen at random among{1, . . . , I} according to the probabilityp ; the choice ofiis communicated to Player 1 only.

∗Laboratoire de Math´ematiques, U.M.R CNRS 6205 Universit´e de Brest, 6, avenue Victor-le-Gorgeu, B.P. 809, 29285 Brest cedex, France e-mail : Pierre.Cardaliaguet@univ-brest.fr

†Laboratoire de Math´ematiques, U.M.R CNRS 6205 Universit´e de Brest, 6, avenue Victor-le-Gorgeu, B.P. 809, 29285 Brest cedex, France e-mail : Catherine.Rainer@univ-brest.fr

Then the players choose their respective controls in order, for Player 1, to minimize the integral payoffR_T

t0 ℓ_i(s, u(s), v(s))ds, and for Player 2 to maximize it. We assume that both players observe their opponent’s control. Note however that Player 2 does not know which payoff he/she is actually maximizing.

Our game is a continuous time version of the famous repeated game with lack of infor- mation on one side studied by Aumann and Maschler (see [1, 13]). The existence of a value for our game has been investigated in [5] (in a more general framework): if Isaacs’ condition holds:

H(t, p) = inf

u∈Usup

v∈V I

X

i=1

p_iℓ_i(t, u, v) = sup

v∈V u∈Uinf

I

X

i=1

p_iℓ_i(t, u, v) ∀(t, p)∈[0, T]×∆(I), (0.1) then the game has a valueV=V(t₀, p) given by

V(t₀, p) = inf

(αi)∈(Ar(t0))^I sup

β∈Br(t0) I

X

i=1

p_iE_αiβ Z T

t0

ℓ_i(s, α_i(s), β(s))ds

(0.2)

= sup

β∈Br(t0)

(αi)∈(Ainfr(t0))^I I

X

i=1

piE_αiβ Z T

t0

ℓi(s, αi(s), β(s))ds

,

for any (t₀, p)∈[0, T]×∆(I), where theα_i ∈ Ar(t₀) (fori= 1, . . . , I) areIrandom strategies for Player 1,β∈ Br(t₀) is a random strategy for Player 2 andE_αiβ

R_T

t0 ℓ_i(s, α_i(s), β(s))ds is the payoff associated with the pair of strategies (αi, β): these notions are explained in the next section. In [5] we also show that the value functionV can be characterized in terms of dual solutions of some Hamilton-Jacobi equations, which, following [4], is equivalent to saying thatV is the unique viscosity solution of the following HJ equation:

min

w_t+H(t, p) ;λ_min ∂²w

∂p²

= 0 in [0, T]×∆(I). (0.3) In the above equation, λ_min(A) denotes the minimal eigenvalue of A, for any symmetric matrixA. Note in particular that this equation says thatV is convex with respect to p.

This paper is mainly devoted to the construction and the analysis of the optimal strategy for the informed player (Player 1). In particular we want to understand how he/she has to quantify the amount information he/she reveals at each time. Our key step towards this aim is the following equality:

V(t₀, p₀) = min

P∈M(t0,p0)EP

Z _T

t0

H(s,p(s))ds

∀(t₀, p₀)∈[0, T]×∆(I), (0.4)

whereM(t₀, p₀) is the set of martingale measuresPon the set D(t₀) of c`adl`ag processesp living in ∆(I) and such that p(t⁻₀) =p₀ and p(T) belong to the extremal points of ∆(I).

Equality (0.4) is reminiscent of a result of [7] in the discrete-time framework. It is directly related with the construction of the optimal strategy for the informed player. Indeed, let P¯ be an optimal martingale measure in (0.4), p(t) = (p₁(t), . . . ,p_I(t)) the coordonnate mapping on D(t₀) and {e₁, . . . , e_I} the canonical basis of IR^I. Then the informed player, knowing that the indexi has been chosen by nature, has just to play the random control t→ argmin_uP

jpj(t)ℓj(t, u, v) with probability ¯Pi, where ¯Pi is the restriction of the law P¯ to the event{p(T) =e_i}.

We give two proofs of (0.4). The first one relies on a time discretization of the value func- tion taken from [14] and the explicit construction of an approximated martingale measure in the discrete game. The second one is based on a dynamic programming for the right- hand side of (0.4) and on the relation between this dynamic programming and the notion of viscosity solution of (0.3). We shall see in particular that the obstacle termλ_min

∂²w

∂p²

≥0 is directly related with the minimization over the martingale measuresP.

Since the optimal martingale in (0.4) plays a central role in our game, an analysis of this martingale is now in order. This is the aim of section 3. We show that, under such a martingale measure ¯P, the canonical process p(t) has to live in the set H(t) ⊂ ∆(I) in which, heuristically, the Hamilton-Jacobi equation

∂V

∂t +H(t, p) = 0

holds. Moreover the processp(t) can only jump on the faces of the graph ofV(t,·). Namely, for any t∈[t0, T], there is a measurable selection ξ of ∂V(t,p(t⁻)), such that

V(t,p(t))−V(t,p(t⁻))− hξ,p(t)−p(t⁻)i= 0 P¯ −a.s.

Conversely, under suitable regularity assumptions on V and on H, a martingale measure satisfying these two conditions turns out to be optimal in (0.4).

In section 4 we compute the optimal martingale measures for several examples of games.

These examples show that, under the optimal measure, the processpcan have very different behavior. When I = 2 and under suitable regularity conditions, the optimal measure is purely discontinuous. For instance we describe a game in which this optimal measure is unique and has to be the law of an Az´ema martingale. In higher dimension, uniqueness is lost in general. Moreover we give a class of examples in which, although there are optimal measures which are purely discontinuous, they are also optimal measures under whichp is

a continuous process. For instance we show a game in whichplives on an expending convex body moving with (reversed time) mean curvature motion. In this case case the law of p takes the form

dp_t= (I−a_t⊗a_t)dB_t,

where (Bt) is a Brownian motion and the processat lives in the unit sphere.

We complete the paper in section 5 by a list of open questions.

1 Notations and general results

In this part we introduce the main notations and assumptions needed in the paper. We also recall the main results of [3, 5]: existence of a value for the game with lack of information on one side, as well as the characterization of the value.

Notations : Throughout the paper, x.y and hx, yi denote the scalar product in the space of vectorsx, y∈IR^K (for someK≥1), and|·|the euclidean norm. The closed ball of centerxand radiusr is denoted byB_r(x). The set ∆(I) is the set of probabilities measures on{1, . . . , I}, always identified with the simplex ofR^I:

p= (p1, . . . , pI)∈∆(I) ⇔

I

X

i=1

pi = 1 andpi≥0 fori= 1, . . . I .

Ifp∈∆(I), we denote byT_∆(I)(p) the tangent cone to ∆(I) at p:

T_∆(I)(p) = [

λ>0

(∆(I)−p)/λ .

We denote by {e_i, i = 1, . . . , I} the canonical basis of IR^I. For any map φ : ∆(I) → IR, Vexφis the convex hull ofφ. We also denote bySI the set of symetric matrices of sizeI×I.

Throughout the paper we assume the following conditions on the data:

( i) U and V are compact subsets of some finite dimensional spaces,

ii) For i= 1, . . . , I, the payoff functionsℓ_i :U×V →IR are continuous. (1.5) We also always assume that Isaac’s condition holds and define the Hamiltonian of the game as:

H(t, p) := min

u∈Umax

v∈V I

X

i=1

p_iℓ_i(t, u, v) = max

v∈V min

u∈U I

X

i=1

p_iℓ_i(t, u, v) (1.6)

for any (t, p)∈[0, T]×∆(I).

For anyt₀ < T, the set of open-loop controls for Player I is defined by U(t₀) ={u: [t₀, T]→U Lebesgue measurable}.

Open-loop controls for Player II are defined symmetrically and denotedV(t₀).

Apurestrategy for Player I at timet₀is a mapα:V(t₀)→ U(t₀) which is nonanticipative with delay, i.e., there is a partition 0 ≤ t₀ < t₁ < · · · < t_k = T such that, for any v₁, v₂ ∈ V(t₀), if v₁ =v₂ a.e. on [t₀, t_i] for some i∈ {0, . . . , k−1}, then α(v₁) =α(v₂) a.e.

on [t₀, t_i+1].

Let us fix E a set of probability spaces which is non trivial and stable by product. A random control for Player I at timet₀ is a pair ((Ω_u,Fu,P_u), u) where the probability space (Ωu,F^u,Pu) belongs to E and whereu : Ωu → U(t0) is Borel measurable from (Ωu,F^u) to U(t₀) endowed with the L¹−distance. We denote by Ur(t₀) the set of random controls for Player I at timet₀ and abbreviate the notation ((Ω_u,Fu,P_u), u) into simply u.

In the same way, a random strategy for Player I is a pair ((Ω_α,Fα,P_α), α), where (Ω_α,Fα,P_α) is a probability space inE andα : Ω_α× V(t₀)→ U(t₀) satisfies

(i) α is measurable from Ωα× V(t0) toU(t0), with Ωα endowed with theσ−fieldF^α and U(t₀) andV(t₀) with the Borel σ−field associated with theL¹−distance,

(ii) there is a partition t₀ < t₁ <· · ·< t_k=T such that, for anyv₁, v₂ ∈ V(t₀), if v₁≡v₂ a.e. on [t₀, t_i] for somei∈ {0, . . . , k−1}, thenα(ω, v₁)≡α(ω, v₂) a.e. on [t₀, t_i+1] for any ω ∈Ω_α.

We denote byA(t₀) the set of pure strategies and by Ar(t₀) the set of random strategies for Player I. By abuse of notations, an element of Ar(t₀) is simply noted α, instead of ((Ω_α,Fα,P_α), α), the underlying probability space being always denoted by (Ω_α,Fα,P_α).

Note thatUr(t₀)⊂ Ar(t₀).

In order to take into account the fact that Player I knows the index i of the terminal payoff, an admissible strategy for Player I is actually aI−uple ˆα= (α₁, . . . , α_I)∈(Ar(t₀))^I. Pure and random controls and strategies for Player II are defined symmetrically;V^r(t0) denotes the set of random controls for Player II, while B(t₀) (resp. Br(t₀)) denotes the set of pure strategies (resp. random strategies). Generic elements ofBr(t₀) are denoted by β, with associated probability space (Ω_β,Fβ,P_β).

Let us recall the

Lemma 1.1 (Lemma 2.2 of [3]) For any pair (α, β) ∈ Ar(t₀)× Br(t₀) and any ω :=

(ω₁, ω₂)∈Ω_α×Ω_β, there is a unique pair (u_ω, v_ω)∈ U(t₀)× V(t₀), such that

α(ω₁, v_ω) =u_ω andβ(ω₂, u_ω) =v_ω . (1.7) Furthermore the mapω→(u_ω, v_ω) is measurable fromΩ_α×Ω_β endowed withFα⊗ Fβ into U(t₀)× V(t₀) endowed with the Borel σ−field associated with the L¹−distance.

Notations : Given any pair (α, β) ∈ Ar(t₀)×Br(t₀), the expectationE_αβis the integral over Ω_α×Ω_βagainst the probability measureP_α⊗P_β. In particular, ifφ: [0, T]×U×V →IR is some bounded continuous map andt∈(t₀, T], we have

E_αβ Z _T

t

φ(s, α, β)ds

:=

Z

Ωα×Ωβ

Z _T

t

φ(s, u_ω(s), v_ω(s)ds

dP_α⊗P_β(ω), (1.8) where (uω, vω) is defined by (1.7). If one of the strategies is deterministic, we simply drop its subscript in the expectation.

As a particular case of Theorem 4.2 of [5] we have:

Theorem 1.2 (Existence of the value [5]) Assume that conditions (1.5) and (1.6) are satisfied. Then equality (0.2) holds and we denote by V(t₀, p) the common value.

In order to give the characterization ofV, we have to recall that the Fenchel conjugate w^∗ of a map w: [0, T]×∆(I)→IR is defined by

w^∗(t,p) = maxˆ

p∈∆(I)p.ˆp−w(t, p) ∀(t,p)ˆ ∈[0, T]×IR^I.

In particularV^∗denotes the conjugate ofV. If nowwis defined on the dual space [0, T]×IR^I, we also denote byw^∗ its conjugate with respect to ˆp given by

w^∗(t, p) = max

ˆ

p∈R^Ip.ˆp−w(t,p)ˆ ∀(t, p)∈[0, T]×∆(I).

Proposition 1.3 (Characterization of the value, [5]) Under the assumptions of The- orem 1.2, the value functionV is the unique function defined on [0, T]×∆(I) such that

(i) V is Lipschitz continuous in all its variables, convex with respect top and vanishes at t=T,

(ii) for any p∈∆(I), t→V(t, p) is a viscosity subsolution of the Hamilton-Jacobi equa- tion

w_t+H(t, p) = 0 in (0, T) (1.9)

where H is defined by (1.6),

(iii) For any smooth test function φ = φ(ˆp) and any pˆ∈ IR^I such that t → V^∗(t,p)ˆ −φ has a local maximum at some tand such that the derivative p:= ^∂V_∂pˆ^∗(t,p)ˆ exists, one has:

φ_t(t,p)ˆ −H(t, p)≥0. (1.10) We say that V is the unique dual solution of the HJ equation (1.9) with terminal condition V(T, p) = 0.

Remark 1.4 In particular V does not depend on the class of probability space E chosen to define the random strategies. In view of the construction of the next chapter, we note that, given a family of probability measures, one can always built a set E containing this family and such thatE is stable by product.

In [4] we prove that the following characterization ofV also holds:

Proposition 1.5 (Equivalent characterization [4]) Vis the unique Lipschitz continu- ous viscosity solution of the following obstacle problem

min

w_t+H(t, p) ;λ_min ∂²w

∂p²

= 0 in (0, T)×∆(I) (1.11) which satisfiesV(T, p) = 0 in ∆(I).

In the above proposition, we say thatwis a subsolution of the terminal time Hamilton- Jacobi equation (1.11) if, for any smooth test function φ : (0, T)×∆(I) → IR such that w−φhas a local maximum at some point (t, p)∈(0, T)×Int(∆(I)), one has

max

φ_t(t, p) +H(t, p) ;λ_min

p,∂²φ

∂p²(t, p)

≥ 0 where, for any (p, A)∈∆(I)× SI,

λ_min(p, A) = min

z∈T_∆(I)(p)\{0}hAz, zi/|z|² .

We say that w is a supersolution of (1.11) if, for any test function φ: (0, T)×∆(I) →IR such thatw−φhas a local minimum at some point (t, p)∈(0, T)×∆(I), one has

max

φ_t(t, p) +H(t, p) ;λ_min

p,∂²φ

∂p²(t, p)

≤ 0. Finally a solution of (1.11) is a sub- and a super-solution of (1.11).

2 Representation of the solution

Let us denote by D(t₀) the set of c`adl`ag functions from IR→ ∆(I) which are constant on (−∞, t₀) and on [T,+∞), by t 7→p(t) the coordinate mapping on D(t₀) and by G = (Gt) the filtration generated byt7→p(t).

Given p0 ∈ ∆(I), we denote by M(t0, p0) the set of probability measures P on D(t0) such that, underP, (p(t), t∈[0, T]) is a martingale and satisfies :

fort < t₀, p(t) =p₀ and, for t≥T, p(t)∈ {e_i, i= 1, . . . , I} P −a.s. .

Finally for any measurePonD(t₀), we denote byE_P[. . .] the expectation with respect to P.

Our main result is the following equality:

Theorem 2.1

V(t₀, p₀) = inf

P∈M(t0,p0)E_P Z T

t0

H(s,p(s))ds

∀(t₀, p₀)∈[0, T]×∆(I). (2.12) We shall give two proofs of the result. The first one is based on a discretization procedure forV, while the second one uses a more direct approach of dynamic programming. Before this we show how to use the above theorem to get optimal strategies for the first player.

2.1 Construction of an optimal strategy

We explain here how to use Theorem 2.1 to built an optimal strategy for the informed Player. The construction of an optimal strategy for the non-informed Player, which uses completely different arguments (the so-called approchability procedure) is described in [15].

Lemma 2.2 For any (t₀, p₀) there is at least one optimal martingale measure for problem (2.12).

Proof: Let be a sequence of measures (P_n)_n∈IN⊂M(t₀, p₀) satisfying V(t0, p0) = lim

n→+∞EPn

Z T t0

H(s,p(s))ds

. (2.13)

Since, under all P ∈ M(t0, p0), the coordinate process p is a martingale with support in the same compact space ∆(I), (P_n) converges weakly (up to some subsequence) to some measure ¯P that still belongs toM(t0, p0) (see Meyer-Zheng [11]). SinceH is bounded and continuous, passing to the limit in (2.13) gives

V(t₀, p₀) =EP¯

Z _T

t0

H(s,p(s))ds

.

Hence ¯P is optimal.

Let (t₀, p₀) ∈ [0, T)×∆(I) be fixed, ¯P be optimal in the problem (2.12). Let us set E_i ={p(T) =e_i} and define the probability measure ¯P_i by: ∀A∈ G, ¯P_i(A) := ¯P[A|E_i] =

P(A∩E¯ i)

pi , ifp_i>0, and ¯P_i(A) =P(A) for an arbitrary probability measureP ∈M(t₀, p₀) if p_i= 0.

We also set

¯

u(t) =u^∗(t,p(t)) ∀t∈IR

and denote by ¯u_i the random control ¯u_i = ((D(t₀),G,P¯_i),u)¯ ∈ Ur(t₀).

Theorem 2.3 The strategy consisting in playing the random control(¯u_i)_i=1,...,I ∈(Ur(t₀))^I is optimal forV(t₀, p₀). Namely

V(t₀, p₀) = sup

β∈Br(t0) I

X

i=1

p_iE_u¯i Z T

t0

ℓ_i(s,u¯_i(s), β(¯u_i)(s))ds

Proof: Since sup

β∈Br(t0) I

X

i=1

p_iE_u¯i,β Z T

t0

ℓ_i(s,u¯_i(s), β(¯u_i)(s))ds

= sup

β∈B(t0) I

X

i=1

p_iE_u¯i Z T

t0

ℓ_i(s,u¯_i(s), β(¯u_i)(s))ds

,

it is enough to prove the equality V(t₀, p₀) = sup

β∈B(t0) I

X

i=1

p_iE_u¯i Z _T

t0

ℓ_i(s,u¯_i(s), β(¯u_i)(s))ds

. (2.14)

Let us note that, sincep(T)∈ {e₁, . . . , e_I}P¯ a.s., we have

I

X

i=1

1Eiei=p(T) P¯ a.s.. (2.15)

Let us fix a strategyβ ∈ B(t₀) and setv=β(¯u). Then

I

X

i=1

p_iEu_¯i

Z _T

t0

ℓ_i(s,u¯_i(s), β(¯u_i)(s))ds

=

I

X

i=1

EP¯

Z _T

t0

ℓ_i(s,u(s),¯ v(s))ds |E_i

P(E¯ _i)

=EP¯

" _I X

i=1

1_Ei Z _T

t0

ℓ_i(s,u(s),¯ v(s))ds

#

=EP¯

hp(T),

Z _T

t0

ℓ(s,u(s),¯ v(s))dsi

(from (2.15) and whereℓ= (ℓ₁, . . . , ℓ_I) )

=EP¯

Z _T

t0

hp(s), ℓ(s,u(s),¯ v(s))ids

(by Itˆo’s formula, sincep(s) is a martingale under ¯Pand the processℓ(s,u(s),¯ v(s)) is (G^s) adapted)

≤EP¯

Z T t0

maxv∈V hp(s), ℓ(s, u^∗(s,p(s)), v)ids

=EP¯

Z T t0

H(s,p(s))ds

=V(t₀, p₀). So we have proved that (2.14) holds.

2.2 Proof of Theorem 2.1 by discretization

For simplicity of notations we shall prove Theorem 2.1 for t₀ = 0, the proof in the general case being similar.

Let us start with an approximation of the value function. This approximation is a particular case of [14]. Letn∈IN^∗,τ = 1/n be the time-step and let us sett_k=kT /n for k= 0, . . . , n. We define by backward induction

V^τ(T, p) = 0 ∀p∈∆(I) and, ifV^τ(t_k+1,·) is defined, then

V^τ(t_k, p) = Vex (V^τ(t_k+1,·) +τ H(t_k,·)) (p)

where Vex(φ)(p) stands for the convex hull of the mapφ=φ(p) with respect top.

Lemma 2.4 ([14]) V^τ uniformly converges to V as τ →0 in the following sense:

lim

τ→0⁺, tk→t, p^′→pV^τ(t_k, p^′) =V(t, p) ∀(t, p)∈[0, T]×∆(I).

For any k= 0, . . . , n and p∈∆(I) there are λ^k = (λ^k_l) ∈∆(I) and π^k = (π^k,l) ∈∆(I) such that

(i)

I

X

l=1

λ^k_lπ^k,l=p and (ii)V^τ(t_k, p) =

I

X

l=1

λ^k_l

V^τ(t_k+1, π^k,l) +τ H(t_k, π^k,l) (2.16) Without loss of generality we can choose the maps p 7→ λ^k(p) and p 7→ π^k(p) Borel mea- surable. For any i ∈ {1, . . . , I} we now define a process (pⁱ_k, k ∈ {0, . . . , n+ 1}) on some arbitrary, big enough probability space (Ω,F, P) with values in ∆(I):

we start withpⁱ₀=p₀ and, if pⁱ_k is defined for k≤n−1,

• if thei-th coordinate ofpⁱ_k satisfies (pⁱ_k)_i>0, then the variablepⁱ_k+1 takes its values in {π^k,l pⁱ_k

, l∈ {1, . . . , I} }with

∀l∈ {1, . . . , I}, P[pⁱ_k+1=π^k,l pⁱ_k

|p^j₀, . . . ,p^j_k, j= 1, . . . , I] = λ^k_l(pⁱ_k)π^k,l_i (pⁱ_k) (pⁱ_k)_i ,

• if (pⁱ_k)_i = 0, then we setpⁱ_k+1=pⁱ_k. Fork=n+ 1, we simply set pⁱ_n+1=e_i .

Finally we setp_k=pⁱ_kwhere iis the index chosen at random by nature (i.e. iis a random variable that is independent from the processes (pⁱ_k), i ∈ {1, . . . , I} and takes the values 1, . . . , I with probability p1, . . . , pI respectively). The following Lemma is classical in the framework of repeated game theory with lack of information on one side (see [1, 13]).

Lemma 2.5 If we denote by (Fk, k= 0, . . . , n+ 1) the filtration generated by (p_k), then P[i=i|Fk] = (p_k)_i ∀k∈ {0, . . . , n+ 1}, ∀i∈ {1, . . . , I}.

In particular, the process(p_k, k= 0, . . . , n+ 1) is a martingale.

Proof: The result is obvious for k = n+ 1. Let us prove by induction on k that, for all i∈ {1, . . . , I},

P[i=i|Fk] = (p_k)_i , (2.17)

fork = 0, . . . , n. For k = 0, equality (2.17) is obvious. We now assume that (2.17) holds true up to some k and check that P[i=i| Fk+1] = (p_k+1)_i. Since the variables p_k take their values in a finite set, we can explicitely write

P[i=i| Fk+1] = X

A∈A

P[i=i|A]1_A, (2.18) where the set A is also finite and contains only sets of the form A = {p₁ =α₁, . . . ,p_k = α_k,p_k+1 =π^k,l(p_k)} with α₁, . . . , α_k ∈∆(I) and l∈ {1, . . . , I} with P[A >0]. For such a A∈ A, let us write

P[i=i|A] =P[{i=i} ∩A]/(

I

X

j=1

P[{i=j} ∩A]). (2.19) and, forj such that (α_k)_j >0, using the independence between iand the processes (p^j_k),

P[{i=j} ∩A] = P[i=j,p^j₁ =α₁, . . . ,p^j_k=α_k,p^j_k+1 =π^k,l(p^j_k)]

= P[i=j]P[p^j_k+1 =π^k,l(p^j_k)|p^j₁=α₁, . . . ,p^j_k=α_k]P[p^j₁ =α₁, . . . ,p^j_k =α_k]

= P[i=j,p^j₁ =α₁, . . . ,p^j_k=α_k]^λ

k

l(αk)π_j^k,l(αk) (αk)j .

(2.20)

Now, by the induction assumption,

P[i=j,p^j₁ =α₁, . . . ,p^j_k=α_k] = P[i=j,p₁=α₁, . . . ,p_k=α_k]

= (α_k)_jP[p₁ =α₁, . . . ,p_k =α_k]. (2.21) Thus, putting together (2.19),(2.20) and (2.21), we find out that

P[i=i|A] =π_i^k,l(α_k).

But, onA,α_k=p_k, therefore, comming back to (2.18), we get P[i=i| Fk+1] = P

A∈Aπ^k,l_i (p_k)1_A

= P_I

l=1π^k,l_i (p_k)1_{p

k+1=π_i^k,l(pk)}

= (p_k+1)_i.

Lemma 2.6

V^τ(0, p₀) =E

"

τ

n−1

X

r=0

H(t_r,p_r+1)

# .

Proof: Let us show by (backward) induction onkthat E

"

τ

n−1

X

r=0

H(t_r,p_r+1)

#

=E

"

V^τ(t_k,p_k) +τ

k−1

X

r=0

H(t_r,p_r+1)

# .

Note that settingk= 0 gives the Lemma.

Fork=n, the result is obvious sinceV^τ(T, p) = 0. Let us assume that the result holds true fork+ 1 and show that it still holds true fork. By the induction assumption we have

E

"

τ

n−1

X

r=0

H(t_r,p_r+1)

#

=E

"

V^τ(t_k+1,p_k+1) +τ

k

X

r=0

H(t_r,p_r+1)

# .

We note that, for all suitable functionf, E[f(p_k+1)|σ{i,p_k}] = P

i1_{i=i}P

l

λ^k_l(pⁱ_k)π^k,l_i (pⁱ_k)

(pⁱ_k)i f(π^k,l(pⁱ_k))

= P

i1_{i=i}P

l

λ^k_l(pk)π^k,l_i (pk)

(pk)i f(π^k,l(p_k)) Thus, by Lemma 2.5,

E[f(p_k+1)|σ{p_k}] =X

i

(p_k)_iX

l

λ^k_l(p_k)π_i^k,l(p_k)

(p_k)_i f(π^k,l(p_k)) =X

l

λ^k_l(p_k)f(π^k,l(p_k)).

In particular, by the definition ofλ^k and π^k in (2.16), we deduce that E[V^τ(t_k+1,p_k+1) +τ H(t_k,p_k+1)|p_k] =V^τ(t_k,p_k).

So

E

"

τ

n−1

X

r=0

H(tr,pr+1)

#

=E

"

V^τ(t_k,p_k) +τ

k−1

X

r=0

H(tr,pr+1)

# , which completes the proof.

We are now ready to show the inequality V(0, p₀)≥ inf

P∈M(0,p0)E_P Z T

0

H(s,p(s))ds

. (2.22)

Let W(0, p₀) denote the right-hand side of this inequality. Let us fix ǫ > 0 and let τ = 1/n >0 sufficiently small so that

|V(0, p₀)−V^τ(0, p₀)| ≤ǫ and

|H(t, p)−H(s, p)| ≤ǫ ∀|s−t| ≤τ, ∀p∈∆(I).

Let (p_k) be the martingale defined above. We built with this discrete time martingale a continuous one by setting: ˜p(t) =p0 ift <0, ˜p(t) =p_k+1ift∈[t_k, t_k+1) fork= 0, . . . , n−1 and ˜p(t) =p_n+1 if t≥ T. The law of (˜p(t)) defines a martingale measureP ∈ M(0, p₀).

Then

V(0, p₀) ≥ V^τ(0, p₀)−ǫ

≥ Eh

τP_n−1

r=0H(t_r,p_r+1)i

−ǫ

≥ Eh RT

0 H(s,p(s))ds˜ i

−2ǫ

≥ EP

hR_T

0 H(s,p(s))dsi

−2ǫ

≥ W(0, p₀)−2ǫ This proves (2.22).

Next we show

V(0, p₀)≤ inf

P∈M(0,p0)EP

Z _T

0

H(s,p(s))ds

. (2.23)

For this let us fix a martingale measureP∈M(0, p0). For n∈IN^∗ large, we discretize the canonical processpin the usual way: pⁿ(t) =p(t_k) ift∈[t_k, t_k+1) fork= 0, . . . , n−1 and t_k=kT /n. Then pⁿ(t) converges top(t) P⊗ L¹−a.s. Therefore

n→+∞lim E_P

"

1 n

n−1

X

r=0

H(t_r,pⁿ(t_r+1))

#

=E_P Z T

0

H(s,p(s))ds

.

To complete the proof of (2.23) it is enough to show that EP

"

1 n

n−1

X

r=0

H(t_r,pⁿ(t_r+1))

#

≥V^τ(0, p₀) (2.24)

whereτ = 1/n. As for Lemma 2.6, the proof of (2.24) is achieved by showing by induction onk∈ {0, . . . , n} that

E_P

"

1 n

n−1

X

r=0

H(t_r,pⁿ(t_r+1))

#

≥E_P

"

V^τ(t_k,p(t_k)) +τ

k−1

X

r=0

H(t_r,pⁿ(t_r+1))

#

(2.25) Fork=nthe result is obvious sinceV^τ(T,·) = 0. Assume that the result holds for k+ 1.

We note that, since

V^τ(t_k+1, p) +τ H(t_k, p)≥V^τ(t_k, p) ∀p∈∆(I)

from the construction ofV^τ(t_k,·) and sinceV^τ(t_k,·) is convex andpⁿis a martingale under P, we have

EP[V^τ(t_k+1,pⁿ(t_k+1)) +τ H(t_k,p(t_k+1)) | Gk] ≥ EP[V^τ(t_k,pⁿ(t_k+1)) | Gk]

≥ V^τ(t_k,E_P[pⁿ(t_k+1) | Gk]) = V^τ(t_k,pⁿ(t_k)) So

EP

h1 n

P_n−1

r=0 H(t_r,pⁿ(t_r+1))i

≥ EP

h

V^τ(t_k+1,p(t_k+1)) +τP_k

r=0H(t_r,pⁿ(t_r+1))i

≥ EP

h

V^τ(t_k,p(t_k)) +τPk−1

r=0H(tr,pⁿ(tr+1))i , which completes the proof of (2.25).

2.3 Proof of Theorem 2.1 by dynamic programming principle

The idea is to show that the map W(t₀, p₀) = inf

P∈M(t0,p0)EP

Z _T

t0

H(s,p(s))ds

∀(t₀, p₀)∈[0, T]×∆(I) (2.26) is a solution of equation (1.11) such thatW(T, p) = 0. Since, in view of Proposition 1.5 this equation has a unique solution andV is also solution, we get the desired result: W =V.

Arguing as in the proof of Lemma 2.2, we first note that there is at least an optimal martingale measure in the minimization problem (2.26).

Lemma 2.7 W is convex with respect to p₀ and Lipschitz continuous in all variables.

Proof: Let p₀, p₁ ∈ ∆(I), P₀ ∈ M(t₀, p₀) and P₁ ∈ M(t₀, p₁) be optimal martingale measures for the game starting from (t₀, p₀) and (t₀, p₁):

EPj

Z _T

t0

H(s,p(s))ds

= W(t₀, p_j)

forj = 0,1. For any λ∈ [0,1], let p_λ = (1−λ)p₀+λp₁ and p_λ be the process on D(t₀) defined by

p_λ(t) =

( p_λ, ift < t0

p(t), ift≥t₀.

We define finally a probability measure P_λ ∈ M(t0, p0) by : For all measurable function Φ :D(t₀)→IR₊,

E_Pλ[Φ(p)] = (1−λ)E_P0[Φ(p_λ)] +λE_P1[Φ(p_λ)].

ThenP_λ clearly belongs toM(t₀, p_λ) and W(t₀, p_λ)≤E_Pλ

Z _T

t0

H(t,p(t))ds

= (1−λ)W(t₀, p₀) +λW(t₀, p₁). This proves thatW is convex with respect to p₀.

We now prove that W is Lipschitz continuous. Since W is convex with respect to p0, we need only to prove that W is Lipschitz continuous at the extremal points (t₀, e_i), t₀ ∈ [0, T], i∈ {1, . . . , I}: namely we have to show that there is some K≥0 such that

|W(t^′₀, p₀)−W(t₀, e_i)| ≤K(|p₀−e_i|+|t^′₀−t₀|) ∀t^′₀, t₀ ∈[0, T], p₀∈∆(I), i∈ {1, . . . , I}. Lett^′₀, t0∈[0, T],p0 ∈∆(I) andi∈ {1, . . . , I}. One easily checks thatM(t0, ei) consists in the single probability measure under which (p(s), t ≤s≤ T) is constant and equal to e_i. Consequently

W(t₀, e_i) = Z _T

t

H(s, e_i)ds .

Let P be the optimal probability measure for problem (2.26) with starting point (t^′₀, p₀).

We can write

|W(t^′₀, p0)−W(t0, ei)|= |EP

RT

t^′₀ H(s,p(s))ds−RT

t0 H(s, ei)ds|

≤ C|t0−t^′₀|+EP

RT

t0 |H(s,p(s))−H(s, ei)|ds , whereC is an upper bound of the map (t, p)7→ |H(t, p)|on [0, T]×∆(I).

Now letκ be a Lipschitz constant for H. Then E_PR_T

t0 |H(s,p(s))−H(s, e_i)|ds≤ κE_PR_T

t0 |p(s)−e_i|ds

≤ P_I

j=1EP

RT

t0 |pj(s)−δij|ds

= P_I

j6=iE_PR_T

t0 p_j(s)ds+E_PR_T

t0(1−p_i(s))ds

= (T−t0)PI

j=1|p^j₀−δij|,

where, for the last line, we used the fact that, under P and for all j ∈ {1, . . . , I}, p_j is a martingale. Finally

I

X

j=1

|p^j₀−δ_ij| ≤√

I|p₀−e_i|,

which completes the proof.

Lemma 2.8 The following dynamic programming holds: for any G-stopping time θ taking its values in [t₀, T],

W(t₀, p₀) = inf

P∈M(t0,p0)EP

Z _θ

t0

H(s,p(s))ds+W(θ,p(θ))

. (2.27)

Proof: Let us introduce the subset M^f(t₀, p₀) of M(t₀, p₀) consisting in the martingale measuresPonD(t₀) starting fromp₀ at timet₀ and for which there is a finite setS⊂∆(I) such that anyp ∈Spt(P) satisfiesp(t)∈S fort∈[t₀, T]P-a.s. It is known thatM^f(t₀, p₀) is dense inM(t₀, p₀) for the weak* convergence of measures. In particular it holds that

W(t0, p0) = inf

P∈M^f(t0,p0)EP

Z T t0

H(s,p(s))ds

∀(t0, p0)∈[0, T]×∆(I), and, since the map P → E_PR_θ

t0H(s,p(s))ds+W(θ,p(θ))

is continuous for the weak*

topology, the lemma is proved as soon we have shown that W(t₀, p₀) = inf

P∈M^f(t0,p0)E_P Z θ

t0

H(s,p(s))ds+W(θ,p(θ))

. (2.28)

We shall prove (2.28) for stopping times taking a finite number of values, then we generalize the result to all stopping times by passing to the limit.

Letθbe aG-stopping time of the form θ=

L

X

l=1

1_Alτ_l=

L

X

l=1

1_{θ=τl}τ_l, (2.29)

with t₀ ≤ τ₁ < . . . < τ_L ≤T and, for all l ∈ {1, . . . , L}, A_l ∈ Gτl. Let P∈ M^f(t₀, p₀) be ǫ−optimal forW(t₀, p₀) andS ={p¹, . . . , p^K}be such that P[p(θ)∈S] = 1. We have EP

Z _T

θ

H(s,p(s))ds

=EP



 X

j,l

EP

Z _T

τl

H(s,p(s))ds

A_l∩ {p(τ_l) =p^j}

1_{Al∩{p(τl)=p}^j_}



 . (2.30) But, sinceP|Al∩{p(τl)=p^j} ∈M(τ_l, p^j), we have, for all land j,

EP

Z T τl

H(s,p(s))ds

A_l∩ {p(τ_l) =p^j}

≥W(τ_l, p^j). (2.31)

Therefore

W(t₀, p₀) +ǫ≥ E_Ph RT

t0 H(s,p(s))dsi

≥ EP

hR_θ

t0H(s,p(s))ds+P

l,jW(τ_l, p^j)1_{Al∩{p(τl)=p}^j_}i

= E_Ph Rθ

t0H(s,p(s))ds+W(θ,p(θ))i .

(2.32)

To prove the converse inequality, let P₀ ∈ M^f(t₀, p₀) be ǫ−optimal in the right-hand side of (2.28) and let S ={p¹, . . . , p^K} be such thatP0[p(θ) ∈S] = 1. For any τ_l ∈[t0, T] andp^j ∈S, let P_l,j beǫ−optimal forW(τ_l, p^j). We define a measure ¯P∈M(t₀, p₀) in the following way : For any functionf : IR→∆(I) andl∈ {1, . . . , I}, we define a process

p_f,l(t) =

( f(t), ift < τ_l p(t), ift≥τ_l. Then we set, for all measurable function Φ :D(t₀)→IR₊,

EP¯[Φ(p)] =EP0



 X

l,j

1_{Al∩{p(τl)=p}^j_}EPj,l[Φ(p_f,l)]_f=p



.

Then ¯P∈M(t₀, p₀) and we have W(t₀, p₀) ≤ EP¯

hR_T

t0 H(s,p(s))dsi

= E_P0h Rθ

t0H(s,p(s))ds+P

l,j1_{Al∩{p(τl)=p}^j_}E_Pl,j[RT

τl H(s,p(s))ds]i

≤ EP0

hR_θ

t0H(s,p(s))ds+P

l,j1_{Al∩{p(τl)=p}^j_}(W(τ_l, p^j) +ǫ)i

= E_P0h R_θ

t0H(s,p(s))ds+W(θ,p(θ)) +ǫi

≤ inf_P∈M^f_(t0,p0)EP

hR_θ

t0H(s,p(s))ds+W(θ,p(θ))i + 2ǫ .

(2.33)

This allows us to conclude that (2.28) holds for stopping times taking a finite number of values, and it remains now to show that (2.28) holds for all stopping times. But this last part of the proof is standard: we just have to notice that, if θ stands now for a general G-stopping time with θ ∈[t0, T], we can always find a sequence (θn)n≥0 of stopping times of the form (2.29) such thatθ_nցθ asn→ ∞ and that, for all P∈M(t₀, p₀), we have

EP

Z _θn

t0

H(s,p(s))ds+W(θ_n,p(θ_n))

→n→∞ EP

Z _θ

t0

H(s,p(s))ds+W(θ,p(θ))

.

Lemma 2.9 W is a solution of (1.11).

Remark 2.10 The proof of Lemma 2.9 is interesting because it shows that the martingale minimization problem gives rise to the penalization termλ_min

p,^∂_∂p^2φ2

in (1.11).

Proof: Let us first show thatW is a supersolution of (1.11). Let φ=φ(t, p) be a smooth function such that φ ≤W with an equality at (t0, p0) ∈[0, T)×∆(I). We want to prove that

min

φ_t(t₀, p₀) +H(t₀, p₀) ;λ_min

p₀,∂²φ

∂p²(t₀, p₀)

≤ 0. For this we assume thatλ_min

p₀,^∂_∂p^2φ2(t₀, p₀)

>0 and it remains to show that

φ_t(t₀, p₀) +H(t₀, p₀) ≤ 0. (2.34) We claim that there arer, δ >0 such that

W(t, p)≥φ(t, p₀) +h∂φ

∂p(t, p₀), p−p₀i+δ|p−p₀|² ∀t∈[t₀, t₀+r], ∀p∈∆(I). (2.35) Proof of (2.35) : From our assumption, there are η >0 and δ >0 such that

h∂²φ

∂p²(t, p)z, zi ≥4δ|z|² ∀z∈T_∆(I)(p₀), ∀(t, p)∈B_η(t₀, p₀), whereT_∆(I)(p₀) is the tangent cone to ∆(I) atp₀:

T_∆(I)(p₀) = [

h>0

(∆(I)−p₀)/h Hence for (t, p)∈Bη(t0, p0) we have

W(t, p)≥φ(t, p)≥φ(t, p₀) +h∂φ

∂p(t, p₀), p−p₀i+ 2δ|p−p₀|² . (2.36) We also note that, for anyp∈∆(I)\Int(Bη(p0)), we have

W(t₀, p)≥φ(t₀, p₀) +h∂φ

∂p(t₀, p₀), p−p₀i+ 2δη² (2.37) because, if we setp₁=p₀+ _|p−p^p−p0

0|η and if ˆp₁∈∂_p⁻W(t₀, p₁), we have W(t₀, p) ≥ W(t₀, p₁) +hpˆ₁, p−p₁i

≥ φ(t₀, p₀) +h^∂φ_∂p(t₀, p₀), p₁−p₀i+ 2δη²+hpˆ₁, p−p₁i

≥ φ(t₀, p₀) +h^∂φ_∂p(t₀, p₀), p−p₀i+ 2δη²+hpˆ₁−^∂φ_∂p(t₀, p₀), p−p₁i where

hpˆ₁−∂φ

∂p(t₀, p₀), p−p₁i ≥0