1.1. Introduction
In this chapter we study optimal control problems for a class of semi-Markov processes using a suitable class of backward stochastic differential equations, driven by the random measure associated to the semi-Markov process itself.
Let us briefly describe our framework. Our starting point is a semi-Markov pure jump process X on a general state space E. It is constructed starting from a jump rate functionλ(x, ϑ) and a jump measureA7→Q(x, ϑ, A) onE, depending onx∈E and ϑ≥0. Our approach is to consider a semi-Markov pure jump process as a two dimensional time-homogeneous and strong Markov process{(Xs, θs), s≥0}with its natural filtrationFand a family of probabilitiesPx,ϑ forx∈E,ϑ∈[0,∞) such that Px,ϑ(X0 =x, θ0 = ϑ) = 1. If the process starts from (x, ϑ) at time t = 0 then the distribution of its first jump timeT1 underPx,ϑ is described by the formula
Px,ϑ(T1 > s) = exp
− Z ϑ+s
ϑ
λ(x, r)dr
, (1.1)
and the conditional probability that the process is inAimmediately after a jump at timeT1=sis
Px,ϑ(XT1 ∈A|T1=s) =Q(x, s, A).
Xs is called the state of the process at times, andθs is the duration period in this state up to moment s:
θs=
θ0+s ifXp =Xs ∀06p6s, p, s∈R,
s−sup{p: 06p6s, Xp 6=Xs} otherwise.
We note that X alone is not a Markov process. We limit ourselves to the case of a semi-Markov process X such that the survivor function of T1 under Px,0 is 33
absolutely continuous and admits a hazard rate functionλas in (1.1). The holding times of the process are not necessarily exponentially distributed and can be infinite with positive probability. Our main restriction is that the jump rate function λ is uniformly bounded, which implies that the processXis non explosive. Denoting by Tn the jump times of X, we consider the marked point process (Tn, XTn) and the associated random measurep(dt dy) =P
nδ(Tn,XTn) on (0,∞)×E, whereδ denotes the Dirac measure. The dual predictable projection ˜pofp(shortly, the compensator) has the following explicit expression
˜
p(ds dy) =λ(Xs−, θs−)Q(Xs−, θs−, dy)ds.
In Section 1.3 we address an optimal intensity-control problem for the semi-Markov process. This is formulated in a classical way by means of a change of probability measure, see e.g. El Karoui [49], Elliott [57] and Br´emaud [18]. We define a classA of admissible control processes (us)s∈[0, T]; for every fixedt∈[0, T] and (x, ϑ) ∈ E ×[0,∞), the cost to be minimized and the corresponding value function are
J(t, x, ϑ, u(·)) = Ex,ϑu,t
Z T−t 0
l(t+s, Xs, θs, us)ds+g(XT−t, θT−t)
, v(t, x, ϑ) = inf
u(·)∈AJ(t, x, ϑ, u(·)),
where g, l are given real functions. Here Ex,ϑu,t denotes the expectation with respect to another probability Px,ϑu,t, depending on t and on the control process u and con-structed in such a way that the compensator underPx,ϑu,t equalsr(t+s, Xs−, θs−, y, us) λ(Xs−, θs−) Q(Xs−, θs−, dy)ds, for some function r given in advance as another da-tum of the control problem. Since the process (Xs, θs)s≥0 we want to control is time-homogeneous and starts from (x, ϑ) at time s = 0, we introduce a temporal translation which allows to define the cost functional for all t ∈ [0, T]. For more details see Remark 1.3.2.
Our approach to this control problem consists in introducing a family of BSDEs parametrized by (t, x, ϑ)∈[0, T]×E×[0,∞):
Ys,tx,ϑ+ Z T−t
s
Z
E
Zσ,tx,ϑ(y)q(dσ dy)
=g(XT−t, θT−t) + Z T−t
s
f
t+σ, Xσ, θσ, Zσ,tx,ϑ(·)
dσ, (1.2)
s∈ [0, T −t], where the generator is given by the Hamiltonian function f defined for everys∈[0, T], (x, ϑ)∈E×[0,+∞),z∈L2(E,E, λ(x, ϑ)Q(x, ϑ, dy)), as
f(s, x, ϑ, z(·)) = inf
u∈U
n
l(s, x, ϑ, u) + Z
E
z(y)(r(s, x, ϑ, y, u)−1)λ(x, ϑ)Q(x, ϑ, dy) o
. (1.3) Under appropriate assumptions we prove that the optimal control problem has a solution and that the value function and the optimal control can be represented by means of the solution to the BSDE (1.2).
1.1. Introduction 35
Backward equations driven by random measures have been studied in many papers, within Tang and Li [128], Barles, Buckdahn and Pardoux [10], Royer [114], Kharroubi, Ma, Pham and Zhang [87], Xia [131], and more recently Becherer [12], Cr´epey and Matoussi [33], Kazi-Tani, Possama¨ı and Zhou [84], [83], Confortola and Fuhrman [27], [28]. In many of them, among which [128], [10], [114] and [87], the stochastic equations are driven by a Wiener process and a Poisson process. A more general result on BSDEs driven by random measures is given by [131], but in this case the generatorf depends on the processZin a specific way and this condition prevents a direct application to optimal control problems. In [12], [33], [84], [83], the authors deal with BSDEs with jumps with a random compensator more general than the compensator of a Poisson random measure; here are involved random compensators which are absolutely continuous with respect to a deterministic measure, that can be reduced to a Poisson measure by a Girsanov change of probability. Finally, in [27]
have been recently studied BSDEs driven by a random measure related to a pure jump process, and in [28] the pure jump Markov case is considered.
Our backward equation (1.2) is driven by a random measure associated to a two dimensional Markov process (X, θ), and his compensator is a stochastic random measure with a non-dominated intensity as in [28]. Even if the associated process is not pure jump, the existence, uniqueness and continuous dependence on the data for the BSDE (1.2) can be deduced extending in a straightforward way the results in [28].
Concerning the optimal control of semi-Markov processes, the case of a finite number of states has been studied in Chitopekar [24], Howard [74], Jewell [81], Osaki [95], while the case of arbitrary state space is considered in Ross [112], Gihman and Skorohod [69], and Stone [125]. As in [24] and in [125], in our formulation we admit control actions that can depend not only on the state process but also on the length of time the process has remained in that state. The approach based on BSDEs is classical in the diffusive context and is also present in the literature in the case of BSDEs with jumps, see as instance Lim and Quenez [92]. However, it seems to us be pursued here for the first time in the case of the semi-Markov processes. It allows to treat in a unified way a large class of control problems, where the state space is general and the running and final cost are not necessarily bounded.
We remark that, comparing with [125], the controlled processes we deal with have laws absolutely continuous with respect to a given, uncontrolled process; see also a more detailed comment in Remark 1.3.3 below. Moreover, in [125] optimal control problems for semi-Markov processes are studied in the case of infinite time horizon.
In Section 1.4 we solve a nonlinear variant of the Kolmogorov equation for the process (X, θ), with the BSDEs approach. The process (X, θ) is time-homogeneous and Markov, but is not a pure jump process. In particular it has the integro-differential infinitesimal generator
L˜ψ(x, ϑ) :=∂ϑψ(x, ϑ)+
Z
E
[ψ(y,0)−ψ(x, ϑ)]λ(x, ϑ)Q(x, ϑ, dy), (x, ϑ)∈E×[0,∞).
The additional differential term∂ϑdoes not allow to study the associated nonlinear Kolmogorov equation proceeding as in the pure jump Markov processes framework
(see [28]). On the other hand, the two dimensional Markov process (Xs, θs)s>0 belongs to the larger class of piecewise-deterministic Markov processes (PDMPs) introduced by Davis in [35], and studied in the optimal control framework by several authors, within Davis and Farid [36], Vermes [129], Dempster [40], Lenhart and Yamada [91]. Moreover, we deal with a very specific PDMP: taking into account the particular structure of semi-Markov processes, we present a reformulation of the Kolmogorov equation which allows us to consider solutions in a classical sense. In particular, we notice that the second component of the process (Xs, θs)s>0 is linear ins. This fact suggests to introduce the formal directional derivative operator
(Dv)(t, x, ϑ) := lim
h↓0
v(t+h, x, ϑ+h)−v(t, x, ϑ)
h ,
and to consider the following nonlinear Kolmogorov equation
Dv(t, x, ϑ) +Lv(t, x, ϑ) +f(t, x, ϑ, v(t, x, ϑ), v(t,·,0)−v(t, x, ϑ)) = 0, t∈[0, T], x∈E, ϑ∈[0,∞), v(T, x, ϑ) =g(x, ϑ),
(1.4) where
Lψ(x, ϑ) :=
Z
E
[ψ(y,0)−ψ(x, ϑ)]λ(x, ϑ)Q(x, ϑ, dy), (x, ϑ)∈E×[0,∞).
Then we look for a solution v such that the map t 7→ v(t, x, t+c) is absolutely continuous on [0, T], for all constants c ∈ [−T, +∞). The functions f, g in (1.4) are given. While it is easy to prove well-posedness of (1.4) under boundedness assumptions, we achieve the purpose of finding a unique solution under much weaker conditions related to the distribution of the process (X, θ): see Theorem 1.4.7. To this end we need to define a formula of Itˆo type, involving the directional derivative operatorD, for the composition of the process (Xs, θs)s>0 with functions v smooth enough (see Lemma 1.4.2 below).
We construct the solutionvby means of a family of BSDEs of the form (1.2). By the results above there exists a unique solution (Ys,tx,ϑ, Zs,tx,ϑ)s∈[0, T−t] and the estimates on the BSDEs are used to prove well-posedness of (1.4). As a by-product we also obtain the representation formulae
v(t, x, ϑ) =Y0,tx,ϑ, Ys,tx,ϑ =v(t+s, Xs, θs), Zs,tx,ϑ(y) =v(t+s, y,0)−v(t+s, Xs−, θs−), which are sometimes called, at least in the diffusive case, non linear Feynman-Kac formulae.
Finally we can go back to the original control problem and observe that the associated Hamilton-Jacobi-Bellman equation has the form (1.4) where f is the Hamiltonian function (1.3). By previous results we are able to identify the HJB solutionv(t, x, ϑ), constructed probabilistically via BSDEs, with the value function.
1.2. Notation, preliminaries and basic assumptions
1.2.1. Semi-Markov Jump Processes. We recall the definition of a semi-Markov process, as given, for instance, in [69]. More precisely we will deal with a semi-Markov process with infinite lifetime (i.e. non explosive). Suppose we are given
1.2. Notation, preliminaries and basic assumptions 37
a measurable space (E,E), a set Ω and two functions X : Ω ×[0,∞) → E, θ : Ω×[0,∞)→[0,∞). For everyt≥0, we denote by Ft theσ-algebra σ((Xs, θs), s∈ [0, t]). We suppose that for every x∈E and ϑ∈[0,∞), a probability Px,ϑ is given on (Ω,F[0,∞)) and the following conditions hold.
(1) E contains all one-point sets. ∆ denotes a point not included inE.
(2) Px,ϑ(X0 =x, θ0 =ϑ) = 1 for everyx∈E,ϑ∈[0,∞).
(3) For everys, p>0 andA∈Ethe function (x, ϑ)7→Px,ϑ(Xs∈A, θs 6p) is E⊗B+-measurable.
(4) For every 0≤t≤s,p>0, and A∈Ewe havePx,ϑ(Xs ∈A, θs 6p|Ft) = PXt,θt(Xs∈A, θs 6p),Px,ϑ-a.s.
(5) All the trajectories of the process X have right limits whenE is given the discrete topology (the one where all subsets are open). This is equivalent to require that for every ω ∈ Ω and t ≥ 0 there exists δ > 0 such that Xs(ω) =Xt(ω) for s∈[t, t+δ].
(6) All the trajectories of the processaare continuous from the right piecewise linear functions. For every ω ∈ Ω, if [α, β) is the interval of linearity of θ·(ω) thenθs(ω) =θα(ω) +s−αandXα(ω) =Xs(ω); ifβ is a discontinuity point of θ·(ω) thenθβ+(ω) = 0 and Xβ(ω)6=Xβ−(ω).
(7) For everyω ∈Ω the number of jumps of the trajectory t7→ Xt(ω) is finite on every bounded interval.
Xs is called the state of the process at times,θs is theduration period in this state up to moments. Also we callXsthephase andθs theage or thetime component of a semi-Markov process. X is a non explosive process because of condition (7). We note, moreover, that the two-dimensional process (X, θ) is a strong Markov process with time-homogeneous transition probabilities because of conditions (2), (3), and (4). It has right-continuous sample paths because of conditions (1), (5) and (6), and it is not a pure jump Markov process, but only a PDMP.
The class of semi-Markov processes we consider in the chapter will be described by means of a special form of joint lawR under Px,ϑ of the first jump timeT1, and the corresponding positionXT1. To proceed formally, we fix X0 =x∈E and define the first jump time
T1 = inf{p >0 : Xp 6=x},
with the convention that T1 = +∞ if the indicated set is empty.
We introduceS :=E×[0,+∞) an we denote bySthe smallestσ-algebra containing all sets of E⊗B([0,+∞)). (Here and in the following B(Λ) denotes the Borel σ-algebra of a topological space Λ). Take an extra point ∆∈/ E and defineX∞(ω) = ∆ for all ω ∈ Ω, so that XT1 : Ω → E ∪ {∆} is well defined. Then on the extended spaceS∪ {(∆,∞)} we consider the smallest σ-algebra, denoted by Senl, containing {(∆,∞)}and all sets ofE⊗B([0,+∞)). Then (XT1, T1) is a random variable with values in (S∪ {(∆,∞)},Senl). Its law under Px,ϑ will be denoted by R(x, ϑ,·).
We will assume thatR is constructed from two given functions denoted byλand Q. More precisely we assume the following.
Hypothesis 1.2.1. There exist two functions
λ:S →[0,∞) and Q:S×E→[0,1]
such that
(i) (x, ϑ)7→λ(x, ϑ) isS-measurable;
(ii) sup(x,ϑ)∈Sλ(x, ϑ)6C∈R+;
(iii) (x, ϑ)7→Q(x, ϑ, A) isS-measurable∀A∈E;
(iv) A7→Q(x, ϑ, A) is a probability measure onE for all (x, ϑ)∈S.
We define a functionH on E×[0,∞] by
H(x, s) := 1−e−R0sλ(x,r)dr. (1.5) Given λ and Q, we will require that for the semi-Markov process X we have, for every (x, ϑ)∈S and forA∈E, 0≤c < d≤ ∞,
R(x, ϑ, A×(c, d)) = 1 1−H(x, ϑ)
Z d c
Q(x, s, A) d
d sH(x, ϑ+s)ds
= Z d
c
Q(x, s, A)λ(x, ϑ+s) exp
− Z ϑ+s
ϑ
λ(x, r)dr
ds, (1.6) where R was described above as the law of (XT1, T1) under Px,ϑ. The existence of a semi-Markov process satisfying (1.6) is a well known fact, see for instance [125]
Theorem 2.1, where it is proved thatXis in addition a strong Markov process. The nonexplosive character ofX is made possible by Hypothesis 1.2.1-(ii).
We note that our data only consist initially in a measurable space (E,E) (E contains all singleton subsets ofE), and in two functionsλ,Qsatisfying Hypothesis 1.2.1. The semi-Markov processX can be constructed in an arbitrary way provided (1.6) holds.
Remark 1.2.2.
(1) Note that (1.6) completely specifies the probability measure R(x, ϑ,·) on (S ∪ {(∆,∞)},Senl): indeed simple computations show that, for s≥0,
Px,ϑ(T1∈(s,∞]) = 1−R(x, ϑ, E×(0, s])
= exp
− Z ϑ+s
ϑ
λ(x, r)dr
, (1.7)
and we clearly have
Px,ϑ(T1 =∞) = R(x, ϑ,{(∆,∞)}) = exp −R∞
ϑ λ(x, r)dr .
Moreover, the kernelRis well defined, becauseH(x, ϑ)<1 for all (x, ϑ)∈S by Hypothesis 1.2.1-(ii).
1.2. Notation, preliminaries and basic assumptions 39
(2) The dataλandQ have themselves a probabilistic interpretation. In fact if in (1.7) we set ϑ= 0 we obtain
Px,0(T1 > s) = exp
− Z s
0
λ(x, r)dr
= 1−H(x, s). (1.8) This means that under Px,0 the law of T1 is described by the distribution functionH, and
λ(x, ϑ) =
∂H
∂ϑ(x, ϑ) 1−H(x, ϑ).
Then λ(x, ϑ) is the jump rate of the process X given that it has been in statex for a timeϑ.
Moreover, the probability Q(x, s,·) can be interpreted as the conditional probability thatXT1 is inA∈Egiven thatT1=s; more precisely,
Px,ϑ(XT1 ∈A, T1 <∞ |T1) =Q(x, T1, A) 1T1<∞, Px,ϑ−a.s.
(3) In [69] the following observation is made: starting from T0 = t define inductively Tn+1 = inf{s > Tn : Xs 6= XTn}, with the convention that Tn+1 = ∞ if the indicated set is empty; then, under the probability Px,ϑ, the sequence of the successive states of the semi-Markov X is a Markov chain, as in the case of Markov processes. However, while for the latter the duration period in the state depends only on this state and it is necessarily exponentially distributed, in the case of a semi Markov process the duration period depends also on the state into which the process moves and the distribution of the duration period may be arbitrary.
(4) In [69] is also proved that the sequence (XTn, Tn)n≥0 is a discrete-time Markov process in (S∪ {(∆,∞)},Senl) with transition kernel R, provided we extend the definition of R making the state (∆,∞) absorbing, i.e. we define
R(∆,∞, S) = 0, R(∆,∞,{(∆,∞)}) = 1.
Note that (XTn, Tn)n≥0 is time-homogeneous.
This fact allows for a simple description of the process X. Suppose one starts with a discrete-time Markov process (τn, ξn)n≥0 in S with transition probability kernel R and a given starting point (x, ϑ) ∈ S (conceptually, trajectories of such a process are easy to simulate). One can then define a process Y in E setting Yt =PN
n=0ξn1[τn,τn+1)(t), where N = sup{n≥0 : τn6∞}. Then Y has the same law as the process X underPx,ϑ.
(5) We stress that (1.5) limits ourselves to deal with a class of semi-Markov processes for which the survivor function T1 under Px,0 admits a hazard rate function λ.
1.2.2. BSDEs driven by a Semi-Markov Process. Let be given a measurable space (E,E), a transition measureQonEand a given positive functionλ, satisfying Hypothesis 1.2.1. Let X be the associated semi-Markov process constructed out of them as described in Section 1.2.1. We fix a deterministic terminal timeT >0 and a pair (x, ϑ)∈S, and we look at all processes under the probabilityPx,ϑ. We denote by F the natural filtration (Ft)t∈[0,∞) of X. Conditions 1, 5 and 6 above imply that the filtration F is right continuous (see [18], Appendix A2, Theorem T26).
The predictableσ-algebra (respectively, the progressive σ-algebra) on Ω×[0,∞) is denoted byP(respectively, byP rog). The same symbols also denote the restriction to Ω×[0, T].
We define a sequence (Tn)n>1 of random variables with values in [0,∞], setting T0(ω) = 0, Tn+1(ω) = inf{s>Tn(ω) : Xs(ω)6=XTn(ω)}, (1.9) with the convention that Tn+1(ω) = ∞ if the indicated set is empty. Being X a jump process we haveTn(ω)6Tn+1(ω) if Tn+1(ω)<∞, while the non explosion of X means thatTn+1(ω)→ ∞. We stress the fact that (Tn)n>1 coincide by definition with the time jumps of the two dimensional process (X, θ).
Forω∈Ω we define a random measure on ([0,∞)×E,B[0,∞)⊗E) setting p(ω, C) =X
n>1
1{(Tn(ω), XTn(ω))∈C}, C ∈B[0,∞)⊗E. (1.10) The random measure λ(Xs−, θs−)Q(Xs−, θs−, dy)ds is called the compensator, or the dual predictable projection, of p(ds, dy). We are interested in the following family of backward equations driven by the compensated random measureq(ds dy) = p(ds dy)−λ(Xs−, θs−)Q(Xs−, θs−, dy)dsand parametrized by (x, ϑ): Px,ϑ-a.s., Ys+
Z T s
Z
E
Zr(y)q(dr dy) =g(XT, θT)+
Z T s
f
r, Xr, θr, Yr, Zr(·)
dr, s∈[0, T].
(1.11) We consider the following assumptions on the dataf and g.
Hypothesis 1.2.3.
(1) The final conditiong:S →RisS-measurable andEx,a h
|g(XT, θT)|2i
<∞.
(2) The generator f is such that
(i) for everys∈[0, T], (x, ϑ)∈S,r ∈R,f is a mapping f(s, x, ϑ, r,·) :L2(E,E, λ(x, ϑ)Q(x, ϑ, dy))→R;
(ii) for every bounded and E-measurablez:E →R the mapping
(s, x, ϑ, r)7→f(s, x, ϑ, r, z(·)) (1.12) is B([0, T])⊗S⊗B(R)-measurable.
(iii) There exist L > 0, L0 >0 such that for every s ∈ [0, T], (x, ϑ) ∈ S, r, r0 ∈R, z, z0 ∈L2(E,E, λ(x, ϑ)Q(x, ϑ, dy)) we have
f(s, x, ϑ, r, z(·))−f(s, x, ϑ, r0, z0(·)) 6L0
r−r0 +L
Z
E
z(y)−z0(y)
2λ(x, ϑ)Q(x, ϑ, dy) 1/2
. (1.13)
1.2. Notation, preliminaries and basic assumptions 41
(iv) We have Ex,ϑ
Z T 0
|f(s, Xs, θs,0,0)|2ds
<∞. (1.14)
Remark 1.2.4. Assumptions (i), (ii), and (iii) imply the following measurability properties off(s, Xs, θs, Ys, Zs(·)):
• ifZ ∈L2(p), then the mapping
(ω, s, y)7→f(s, Xs−(ω), θs−(ω), y, Zs(ω,·)) is P⊗B(R)-measurable;
• if, in addition, Y is aP rog-measurable process, then (ω, s)7→f(s, Xs−(ω), θs−(ω), Ys(ω), Zs(ω,·)) is P rog-measurable.
We introduce the space Mx,ϑ of the processes (Y, Z) on [0, T] such that Y is real-valued andP rog-measurable,Z : Ω×E →RisP⊗E-measurable, and
||(Y, Z)||2
Mx,ϑ :=Ex,ϑ Z T
0
|Ys|2ds+ Z T
0
Z
E
|Zs(y)|2λ(Xs, θs)Q(Xs, θs, dy)ds
<∞.
The space Mx,ϑ endowed with this norm is a Banach space, provided we identify pairs of processes whose difference has norm zero.
Theorem 1.2.5. Suppose that Hypothesis 1.2.3 holds for some (x, ϑ)∈S.
Then there exists a unique pair (Y, Z) in Mx,ϑ which solves the BSDE (1.11). Let moreover (Y0, Z0) be another solution in Mx,ϑ to the BSDE (1.11) associated with the driver f0 and final datum g0. Then
sup
s∈[0, T]
Ex,ϑ
|Ys−Ys0|2 +Ex,ϑ
Z T 0
|Ys−Ys0|2ds
+Ex,ϑ Z T
0
Z
E
|Zs(y)−Zs0(y)|2λ(Xs, θs)Q(Xs, θs, dy)ds
6CEx,ϑ
|g(XT)−g0(XT)|2 +CEx,ϑ
Z T 0
|f(s, Xs, θs, Ys0, Zs0(·))−f0(s, Xs, θs, Ys0, Zs0(·))|2ds
, (1.15) where C is a constant depending onT, L, L0.
Remark 1.2.6. The construction of a solution to the BSDE (1.11) is based on the integral representation theorem of marked point process martingales (see, e.g., [35]), and on a fixed-point argument. Similar results of well-posedness for BSDEs driven by random measures can be found in literature, see, in particular, the theorems given in [28], Section 3, and in [12]. Notice that these results can not be a priori straight applied to our framework: in [12] are involved random compensators which are
absolutely continuous with respect to a deterministic measure, instead in our case the compensator is a stochastic random measure with a non-dominated intensity;
[28] apply to BSDEs driven by a random measure associated to a pure jump Markov process, while the two dimensional process (X, θ) is Markov but not pure jump.
Nevertheless, under Hypothesis 1.2.3, Theorem 3.4 and Proposition 3.5 in [28] can be extended to our framework without additional difficulties. The proofs turn out to be very similar to those of the mentioned results, and we do not report them here
to alleviate the presentation.
1.3. Optimal control of semi-Markov processes
1.3.1. Formulation of the problem. In this section we consider again a mea-surable space (E,E), a transition measure Qand a functionλsatisfying Hypothesis 1.2.1. The data specifying the optimal control problem we will address to are an action (or decision) space U, a running cost function l, a terminal cost function g, a (deterministic, finite) time horizon T > 0 and another function r specifying the effect of the control process. We define an admissible control process, or simply a control, as a predictable process (us)s∈[0, T] with values in U. The set of admissible control processes is denoted by A. We will make the following assumptions:
Hypothesis 1.3.1.
(1) (U,U) is a measurable space.
(2) The functionr: [0, T]×S×E×U →RisB([0, T])⊗S⊗E⊗U-measurable and there exists a constant Cr >1 such that,
06r(t, x, ϑ, y, u)6Cr, t∈[0, T],(x, ϑ)∈S, y∈E, u∈U. (1.16) (3) The function g:S →R isS-measurable, and for all fixed t∈[0, T],
Ex,ϑ
h|g(XT−t, θT−t)|2i
<∞, ∀(x, ϑ)∈S. (1.17) (4) The function l : [0, T]×S×U → R is B([0 T])⊗S⊗U-measurable and there exists α >1 such that, for every fixed t∈[0, T], for every (x, ϑ)∈S and u(·)∈A,
infu∈Ul(t, x, ϑ, u)>∞;
Ex,ϑ hRT−t
0 |infu∈Ul(t+s, Xs, θs, u)|2 ds i
<∞, Ex,ϑ
hRT−t
0 |l(t+s, Xs, θs, us)|ds iα
<∞.
(1.18)
To any (t, x, ϑ)∈[0, T]×S and any control u(·)∈Awe associate a probability measurePx,ϑu,t by a change of measure of Girsanov type, as we now describe. Recalling the definition of the jump times Tn in (1.9), we define, for every fixedt∈[0, T],
Lts= exp Z s
0
Z
E
(1−r(t+σ, Xσ, θσ, y, uσ))λ(Xσ, θσ)Q(Xσ, θσ, dy)dσ
·
· Y
n>1:Tn6s
r(t+Tn, XTn−, θTn−, XTn, uTn),
1.3. Optimal control of semi-Markov processes 43
for all s∈[0, T−t], with the convention that the last product equals 1 if there are no indicesn>1 satisfyingTn6s. As a consequence of the boundedness assumption onQ andλit can be proved, using for instance Lemma 4.2 in [27], or [18] Chapter VIII Theorem T11, that for every fixedt∈[0, T] and for everyγ >1 we have
Ex,ϑ LtT−t
γ
<∞, Ex,ϑ LtT−t
= 1, (1.19)
and therefore the processLtis a martingale (relative toPx,ϑandF). Defining a prob-abilityPx,ϑu,t(dω) =LtT−t(ω)Px,ϑ(dω), we introduce the cost functional corresponding tou(·)∈Aas
J(t, x, ϑ, u(·)) =Ex,au,t
Z T−t 0
l(t+s, Xs, θs, us)ds+g(XT−t, θT−t)
, (1.20) where Ex,ϑu,t denotes the expectation under Px,ϑu,t. Taking into account (1.17), (1.18) and (1.19), and using H¨older inequality it is easily seen that the cost is finite for every admissible control. The control problem starting at (x, ϑ) at time s = 0 with terminal times= T−t consists in minimizing J(t, x, ϑ,·) over A. We finally introduce the value function
v(t, x, ϑ) = inf
u(·)∈AJ(t, x, ϑ, u(·)), t∈[0, T], (x, ϑ)∈S.
The previous formulation of the optimal control problem by means of change of probability measure is classical (see e.g. [49], [57], [18]). Some comments may be useful at this point.
Remark 1.3.2.
1. The particular form of cost functional (1.20) is due to the fact that the time-homogeneous Markov process (Xs, θs)s>0 satisfies
Px,ϑ(X0 =x, θ0 =ϑ) = 1;
the introduction of the temporal translation in the first component allows us to define J(t, x, ϑ, u(·)) for allt∈[0, T].
2. We recall (see e.g. [18], Appendix A2, Theorem T34) that a process u is F-predictable if and only if it admits the representation
us(ω) =X
n>0
u(n)s (ω) 1(Tn(ω),Tn+1(ω)](s)
where for each (ω, s)7→u(n)s (ω) isF[0, Tn]⊗B(R+)-measurable, withF[0, Tn]= σ(Ti, XTi,06i6n) (see e.g. [18], Appendix A2, Theorem T30). Thus the fact that controls are predictable processes admits the following interpreta-tion: at each time Tn (i.e. immediately after a jump) the controller, having observed the random variables Ti, XTi,(0 6 i 6 n), chooses his current action, and updates her/his decisions only at timeTn+1.
3. It can be proved (see [75] Theorem 4.5) that the compensator of p(ds dy) under Px,ϑu,t is
r(t+s, Xs−, θs−, y, us)λ(Xs−, θs−)Q(Xs−, θs−, dy)ds,
whereas the compensator ofp(ds dy) underPx,ϑwasλ(Xs−, θs−)Q(Xs−, θs−, dy) ds. This explains that the choice of a given controlu(·) affects the stochastic system multiplying its compensator byr(t+s, x, ϑ, y, us).
4. We call control law an arbitrary measurable function u : [0, T]×S → U. Given a control law one can define an admissible control u setting us = u(s, Xs−, θs−).
Controls of this form are called feedback controls. For a feedback control the compensator of p(ds dy) is r(t+s, Xs−, θs−, y,u(s, Xs−, θs−)) λ(Xs−, θs−) Q(Xs−, θs−, dy) ds under Px,ϑu,t. Thus, the process (X, θ) under the opti-mal probability is a two-dimensional Markov process corresponding to the transition measure
r(t+s, x, ϑ, y,u(s, x, ϑ))λ(x, ϑ)Q(x, ϑ, dy)
instead ofλ(x, ϑ)Q(x, ϑ, dy). However, even if the optimal control is in the feedback form, the optimal process is not, in general, time-homogeneous since the control law may depend on time. In this case, according to the definition given in Section 1.2, the processX under the optimal probability is not a semi-Markov process.
Remark 1.3.3. Our formulation of the optimal control should be compared with another approach (see e.g. [125]). In [125] is given a family of jump measures on E {Q(x, b,·), b ∈ B} with B some index set endowed with a topology. In the so called strong formulation a control u is an ordered pair of functions (λ0, β) with λ0 :S →R+,β :S→B such that
λ0 and β areS−measurable;
∀x∈E,∃t(x)>0 : Rt(x)
0 λ0(x, r)dr <∞;
Q(·, β, A) isB+-measurable∀A∈E.
If A is the class of controls which satisfies the above conditions, then a control u= (λ0, β)∈Adetermines a controlled processXu in the following manner. Let
Hu(x, s) := 1−e−R0sλ0(x,r)dr, ∀(x, s)∈S,
and suppose that (X0u, θ0u) = (x, ϑ). Then at time 0, the process starts in state x and remains there a random time S1>0, such that
Px,ϑ{S1 6s}= Hu(x, ϑ+s)−Hu(x, ϑ)
1−Hu(x, ϑ) . (1.21)
At time S1 the process transitions to the state XSu1, where Px,ϑ
XSu1 ∈A|S1 =Q(x, β(x, S1), A).
The process stays in state XSu1 for a random timeS2>0 such that Px,ϑ
S2 6s|S1, XSu1 =Hu(XSu1, s) and then at timeS1+S2 transitions toXSu
1+S2, where Px,ϑ
XSu1+S2 ∈A|S1, XSu1, S2 =Q(XSu1, β(XSu1, S2), A).
1.3. Optimal control of semi-Markov processes 45
We remark that the processXuconstructed in this way turns out to be semi-Markov.
We also mention that the class of control problems specified by the initial data
We also mention that the class of control problems specified by the initial data