MULTI-PERIOD MEAN VARIANCE OPTIMAL CONTROL OF MARKOV JUMP WITH MULTIPLICATIVE NOISE SYSTEMS

on the occasion of his 70th birthday

MULTI-PERIOD MEAN VARIANCE OPTIMAL CONTROL OF MARKOV JUMP WITH MULTIPLICATIVE NOISE SYSTEMS

OSWALDO L.V. COSTA and RODRIGO T. OKIMURA

We consider the multi-period mean variance stochastic optimal control problem of discrete-time Markov jump with multiplicative noise linear systems. First, we consider the performance criterion to be a linear combination of the final variance and expected value of the output of the system. We analytically derive an optimal control policy for this problem. By using this solution, we consider next the cases in which the performance criterion is to minimize the final variance subject to a restriction on the final expected value of the output, and to maximize the final expected value subject to a restriction on the final variance of the output of the system. The optimal control strategies are obtained from a set of interconnected Riccati difference equations.

AMS 2000 Subject Classification: 49N10, 60J10, 91B28, 93E20.

Key words: mean variance control, Markov jump system, multiplicative noise, sto- chastic optimal control.

1. INTRODUCTION

The uni-period mean-variance optimization is a classical financial prob- lem introduced by [10] which paved the foundation for the modern portfo- lio theory. Using a stochastic linear quadratic theory developed in [1], the continuous-time version of Markowitz’s problem was studied in [15], with closed-form efficient policies derived, along with an explicit expression of the efficient frontier. In [5] the authors extended the mean-variance allocation problem to the discrete-time multi-period case while in [16] they considered a multi-period generalized mean-variance formulation for the risk control over bankruptcy. A geometric approach to these problems was presented in [4], considering assets as well as liabilities in the portfolios. In [13] the authors considered the discrete-time multi-period mean-variance allocation problem in the case where the parameters are subject to Markovian jumps, following an approach closely related to that in [5], while in [14] they studied a simi- lar problem under a different point of view. As pointed out in [16], one of

MATH. REPORTS9(59),1 (2007), 21–34

the key difficulties in solving the multi-period mean variance problem is the non-separability in the associated stochastic control problem in the sense of dynamic programming. Due to that reason, a tractable (from the dynamic programming point of view) auxiliary problem is introduced.

In this paper we consider the multi-period mean variance stochastic op- timal control problem of discrete-time Markov jump with multiplicative noise linear systems. As in [5] we introduce a tractable auxiliary stochastic qua- dratic optimal control problem, where the performance criterion consists of a linear part and a quadratic cost of the state variable at the final time T. There is no penalty on the control variable, so that the standard techniques for the LQG problems cannot be used. It should be pointed out that problems with indefinite weighting matrices have been intensively studied lately as can be seen, for instance, in [8], [11], and for the case with Markov jumps and multiplicative noise in [3], [6], [7], [9].

The paper is organized as follows. In Section 2 we present the notation and some preliminary results that will be required for the solution of the auxil- iary stochastic quadratic optimal control problem. In Section 3 we present the three problems that we will consider. In Problem P1 it is asked to maximize the final expected value subject to a restriction on the final variance of the output of the system while in Problem P2 it is asked to minimize the final variance subject to a restriction on the final expected value of the output. In Problem P3 it is asked to minimize a performance criterion which is a linear combination of the final variance and expected value of the output of the sys- tem. An analytical optimal control policy for these problems can be obtained through an auxiliary stochastic quadratic optimal control problem, solved in Section 4 in terms of a set of interconnected Riccati difference equations. The paper is concluded in Section 5 with a solution for the 3 problems stated in Section 3, expressed in terms of some key parameters. These parameters are explicitly written as functions of the parameters of the system and in terms of the set of interconnected Riccati difference equations.

A possible application of the results in this paper would be in an asset liabilities management (ALM) model for defined-benefit (BD) pension funds with regime switching. We could assume that the market parameters depend on the market mode that switches according to a Markov chain among a finite number of states. The ALM for DB pension funds problem can then be written as a Markov jump with multiplicative noise LQ optimal control problem with linear and quadratic costs, so that the results presented here can be applied to solve the problem.

2. PRELIMINARIES

We denote by Rⁿ the n-dimensional real Euclidean space and by B(Rⁿ,R^m) the normed bounded linear space of all m×n real matrices, with B(Rⁿ) := B(Rⁿ,Rⁿ). For a matrix A ∈ B(Rⁿ,R^m), N(A) denotes the null space of A, R(A) the range of A and A the transpose of A. As usual, for A ∈ B(Rⁿ), A ≥ 0 (A > 0 respectively) means that the matrix A is posi- tive semi-definite (positive definite), and tr(A) denotes the trace of A. The operator expected value will be denoted by E(.). Set H^n,m for the linear space made up of all N-sequences of real matrices V = (V₁, . . . , VN) with V_i ∈ B(Rⁿ,R^m), i = 1, . . . , N, and, for simplicity, set Hⁿ := H^n,n. We say that V = (V₁, . . . , VN) ∈Hⁿ⁺ (Hⁿ) if V ∈Hⁿ and, for each i= 1, . . . , N, Vi

is a positive-semidefinite (symmetric) matrix. For V = (V₁, . . . , V_N) ∈ Hⁿ, R= (R₁, . . . , R_N)∈Hⁿ, we write V ≥R ifV_i−R_i≥0 for each i= 1, . . . , N. We denote byB(Hⁿ,H^m) the space of all bounded linear operators fromHⁿto H^m and, in particular, B(Hⁿ) := B(Hⁿ,Hⁿ). We say that T ∈ B(Hⁿ⁺,H^m+) ifT ∈ B(Hⁿ,H^m) and is such that T(V) ∈H^m+ whenever V ∈ Hⁿ⁺. For a sequence ofndimensional square matricesA(0), . . . , A(T), we use the follow- ing notation: _t

=sA() =A(t). . . A(s) for t≥s,I fort < s. We define 1_S as the usual indicator function, that is, 1S(ω) = 1 if ω ∈S, zero elsewhere. We need the following definition (see [12], pages 12–13)).

Definition 1. For a matrix A ∈ B(Rⁿ,R^m), the generalized inverse of A (or Moore-Penrose inverse of A) is defined to be the unique matrix A^† ∈ B(R^m,Rⁿ) such that i) AA^†A = A, ii) A^†AA^† = A^†, iii) (AA^†) =AA^†, and iv) (A^†A) =A^†A.

We recall the result below (see [12], pages 12–13).

Proposition1 (Schur’s complement). The following assertion are equiv- alent.

a) Q=

Q₁₁ Q₁₂ Q₁₂ Q₂₂

≥0.

b) Q₂₂≥0, Q₁₂=Q₁₂Q^†₂₂Q₂₂ and Q₁₁−Q₁₂Q^†₂₂Q₁₂≥0.

c) Q₁₁≥0, Q₁₂=Q₁₁Q^†₁₁Q₁₂ and Q₂₂−Q₁₂Q^†₁₁Q₁₂≥0.

The following result will be useful in the sequel.

Proposition 2. Consider Y ∈ B(Rⁿ) and M ∈ B(R^m) with Y ≥ 0.

Let A and B be stochastic matrices (that is, each entry of them is a random variable) in B(Rⁿ) and B(R^m,Rⁿ), respectively. Then

(1) E(AY A)−E(AY B)

E(BY B)_†

E(BY A)≥0

and

(2) E(AY B) =E(AY B)

E(BY B) +M_†

E(BY B) . Proof. Consider the stochastic matrix

(3) Q=

Q₁₁ Q₁₂ Q₁₂ Q₂₂

=

Y Y B BY R

,

where R = BY B. Clearly we have Q₁₁ = Y ≥ 0, and Q₁₂ = Y B = Q₁₁Q^†₁₁Q₁₂ = Y Y^†Y B since Y Y^†Y = Y from Definition 1. Furthermore, using again Y Y^†Y = Y, we have Q₂₂−Q₁₂Q^†₁₁Q₁₂ = R−BY Y^†Y B = R−BY B= 0. Thus, by Schur’s complement (Proposition 1), Q≥0, hence

AY A AY B BY A R

=

A 0 0 I

Y Y B

BY R

A 0 0 I

≥0.

Taking the expected value of the above equation, we get Q₁₁ Q₁₂

Q₁₂ Q₂₂

=

E(AY A) E(AY B) E(BY A) E(R)

≥0.

By Schur’s complement again, we have

0≤Q₁₁−Q₁₂Q^†₂₂Q₁₂=E(AY A)−E(AY B)E(R)^†E(BY A) and Q₁₂=E(AY B) =Q₁₂Q^†₂₂Q₂₂=E(AY B)E(R)^†E(R), showing the result stated.

3. PROBLEM FORMULATION

On a probabilistic space (Ω,P,F) consider the Markov Jump Linear System with multiplicative noise

x(k+ 1) =

A¯_θ(k)(k) +

ε^x

s=1

A_θ(k),s(k)w_s^x(k) x(k)+

(4)

+

B¯_θ(k)(k) +

ε^u

s=1

B_θ(k),s(k)w_s^u(k) u(k), x(0) =x₀, θ(0) =θ₀,

where θ(k) is a time-varying Markov chain taking on values in {1, . . . , N}

with transition probability matrixP(k) = [p_ij(k)], {w_s^x(k); s = 1, . . . ε^x, k = 0,1, . . . , T −1} are zero-mean random variables independent of the Markov chain{θ(k)}, with variance equal to 1 and E(w^x_i(t)w_j^x(k)) = 0, for all t =k

and i = j. Similarly, {w^u_s(k); s = 1, . . . ε^u, k = 0,1, . . . , T −1} are zero- mean random variables independent of the Markov chain{θ(k)}, with variance equal to 1 and E(w_i^u(t)w_j^u(k)) = 0, for all t = k and i = j. The initial conditionsθ₀ and x₀ are assumed to be independent of {w_s^x(k)} and{w^u_s(k)}, with x₀ an n-dimensional random vector with finite second moments. Set µ_i(0) = E(x₀1_{θ0=i}),µ(0)∈R^{N n} asµ(0) = (µ₁(0) · · · µ_N(0)), and Q_i(0) = E(x(0)x(0)1_{θ0=i}), Q(0) = (Q₁(0), . . . , Q_N(0)) ∈ Hⁿ⁺. The correlation of w_s^x₁(k) andw^u_s2(k) is denoted byE(w^x_s1(k)w^u_s2(k)) =ρ_s1,s2(k). Without loss of generality, assume thatε=ε^x =ε^u. For eachk= 0,1, . . . , T−1, we also have

A(k) = ( ¯¯ A₁(k), . . . ,A¯_N(k))∈Hⁿ,

A_s(k) = (A_1,s(k), . . . ,A_N,s(k))∈Hⁿ, s= 1, . . . , ε, B(k) = ( ¯¯ B₁(k), . . . ,B¯N(k))∈H^m,n,

Bs(k) = (B₁,s(k), . . . ,BN,s(k))∈H^m,n, s= 1, . . . , ε.

Set π_i(k) = P(θ(k) = i), let Ft be the σ-field generated by {(θ(s), x(s));

s= 0, . . . , τ}, and write

U(τ) ={u_τ = (u(τ), . . . , u(T−1));u(k) is an m-dimensional random vector with finite second moments that isFk-measurable for each k=τ, . . . , T−1}.

Consider the scalar output

(5) y(t) =Lx(t)

of system (4), whereL∈B(Rⁿ,R).

The multi-period mean-variance problem aims at selecting u ∈ U(0) which yields the greatest expected terminal value of the output E(y^u(T)) given a maximal terminal output σ² for the variance Var(y^u(T)), or which produces the lesser terminal output variance Var(y^u(T)) given a maximal ex- pected terminal value for the output E(y^u(T)). Formally, these problems, caled respectively P1

σ²

and P2 ( ), can be stated as P1

σ²

: min

u∈U(0) −E(y^u(T)) subject to Var (y^u(T))≤σ², (6)

P2 ( ) : min

u∈U(0) Var (y^u(T)) subject to E(y^u(T))≥ . (7)

Alternatively, an unconstrained form would be P3 (ν) : min

u∈U(0) νVar (y(T))−E(y(T)), (8)

where ν ∈ [0,∞) is a risk aversion coefficient, giving a trade-off preference between the expected terminal wealth and the associated risk level. Since problem P3(ν) involves a non-linear function of the expectation in Var(V(t)) =

E(V(t)²)−E(V(t))², it cannot be directly solved by dynamic programming.

A solution procedure to seek an optimal dynamic control policy for problem P3(ν) based on a tractable auxiliary problem is proposed in [16]. We will adopt the same procedure in this paper, and consider the auxiliary problem

A(λ, ν) : min

u∈U(0) E

νy(T)²−λy(T) . (9)

4. SOLUTION OF THE AUXILIARY PROBLEM

Let us consider the following intermediate problems for problem (9). At each time k∈ {0, . . . , T −1} define

J(x(k), θ(k), k) = min

u_k∈U(k)E

νy(T)²−λy(T)| Fk

.

Define next fork= 0, . . . , T−1 the operatorsE(k, .)∈B(Hⁿ),A(k, .) ∈B(Hⁿ), G(k, .) ∈ B(Hⁿ,H^n,m), R(k, .) ∈ B(Hⁿ,H^m), P(k, .) ∈ B(Hⁿ), V(k, ., .) ∈ B(Hⁿ×H^n,1,H^n,1),D(k, ., ., .)∈B(Hⁿ×H^n,1×H¹,H¹), andH(k, .)∈B(H^n,1, H^n,m). ForX∈Hⁿ,V ∈H^n,1,γ ∈H¹, and i= 1, . . . , N, set

Ei(k, X) = N j=1

p_ij(k)X_j,

Ai(k, X) = ¯Ai(k)Ei(k, X) ¯Ai(k) + ε s=1

Ai,s(k)Ei(k, X)Ai,s(k),

Gi(k, X) =

A¯i(k)Ei(k, X) ¯Bi(k)+

+ ε s₁=1

ε s₂=1

ρs₁,s₂(k)Ai,s₁(k)Ei(k, X)Bi,s₂(k)

,

Ri(k, X) = ¯B_i(k)Ei(k, X) ¯B_i(k) + ε s=1

B_i,s(k)Ei(k, X)B_i,s(k), Pi(k, X) =Ai(k, X)− Gi(k, X)Ri(k, X)^†Gi(k, X),

Vi(k, X, V) =Ei(k, V)( ¯Ai(k)−B¯i(k)Ri(k, X)^†Gi(k, X)), Di(k, X, V, γ) =Ei(k, γ)−1

4Ei(k, V) ¯B_i(k)Ri(k, X)^†B¯_i(k)Ei(k, V), Hi(k, V) = ¯B_i(k)Ei(k, V).

It is easy to see that E(k, .) ∈ B(Hⁿ⁺), A(k, .) ∈ B(Hⁿ⁺) and R(k, .) ∈ B(Hⁿ⁺,H^m+). The next proposition will be useful in the sequel. It justi- fies the definition of the operators above. Notice that this result is closely related to the optimality Bellman equation.

Proposition 3. Let P = (P₁, . . . , PN)∈Hⁿ⁺,V = (V₁, . . . , VN)∈H^n,1 andγ ∈H¹. Then P(k, P)∈Hⁿ⁺ and

Gi(k, P) =Gi(k, P)Ri(k, P)^†Ri(k, P).

(10)

Moreover, for anyu_k ∈U(k), u(k) =u, x(k) =x and θ(k) =i we have E

νx(k+ 1)P_θ(k+1)x(k+ 1)−λV_θ(k+1)x(k+ 1) +λ²

ν γ_θ(k+1)|Fk = (11)

=ν[xAi(k, P)x+ 2xGi(k, P)u+uRi(k, P)u]−

−λ[Ei(k, V)A¯_i(k)x+ ¯B_i(k)u ] +λ²

ν Ei(k, γ), and, if for eachi,

(12) Ei(k, V) ¯B_i(k)∈R(Ri(k, P)), then (11) can be rewritten as

ν[xAi(k, P)x+ 2xGi(k, P)u+uRi(k, P)u]− (13)

−λ[Ei(k, V)A¯i(k)x+ ¯Bi(k)u ] + λ²

ν Ei(k, γ) =

=ν[xPi(k, P)x+ (u+a(x))Ri(k, P)(u+a(x))]−

−λVi(k, P, V)x+λ²

ν Di(k, P, V, γ), where

a(x) =Ri(k, P)^†

Gi(k, P)x− λ

2νB¯i(k)Ei(k, V) . (14)

Proof. SettingA= ¯A_i(k)+

ε s=1

A_i,s(k)w^x_s(k),B = ¯B_i(k)+

ε s=1

B_i,s(k)w_s^u(k), Y = Ei(P) in Proposition 2, from inequation (1), and the hypothesis on {w_s^x(k)} and {w^u_s(k)} we have

Pi(k, P) = ¯Ai(k)Ei(P) ¯Ai(k) + ε s=1

Ai,s(k)Ei(P)Ai,s(k)−

−Gi(k, P)Ri(k, P)^†Gi(k, P)≥0,

hencePi(k, P) ≥0. It also follows from Proposition 2 and equation (2) that (10) holds. We then have

E

x(k+ 1)P_θ(k+1)x(k+ 1)|Fk = (15)

=x

A¯_i(k)Ei(k, P) ¯A_i(k) + ε s=1

A_i,s(k)Ei(k, P)A_i,s(k) x+

+2x

A¯_i(k)Ei(k, P) ¯B_i(k) + ε s₁=1

ε s₂=1

ρ_s1,s2(k)A_i,s1(k)Ei(k, P)B_i,s2(k) u+

+u

B¯_i(k)Ei(k, P) ¯B_i(k) + ε s=1

B_i,s(k)Ei(k, P)B_i,s(k) u

and

E

V_θ(k+1)x(k+ 1)|Fk =Ei(k, V)

A¯i(k)x+ ¯Bi(k)u , (16)

E

γ_θ(k+1)|Fk =Ei(k, γ).

(17)

Equations (15), (16) and (17) yield (11). Considering now on the right hand side of (11) only the terms dependent onu and calling themf(u), we have

f(u) =νuRi(k, P)u+ 2

νxGi(k, P) −λ

2Ei(k, V) ¯B_i(k) u.

(18)

It follows from (10) and (12) that (18) can be written as f(u) =νuRi(k, P)u+

(19)

+2ν

xGi(k, P)− λ

2νEi(k, V) ¯B_i(k) Ri(k, P)^†Ri(k, P)u.

Writing a(x) as in (14), equation (4) can be rewritten as f(u) =ν[uRi(k, P)u+ 2a(x)Ri(k, P)u]

=ν[(u+a(x))Ri(k, P)(u+a(x))−a(x)Ri(k, P)a(x)].

Notice now that

−a(x)Ri(k, P)a(x) =−

xGi(k, P)Ri(k, P)^†Gi(k, P)x− (20)

−λ

νEi(k, V) ¯B_i(k)Ri(k, P)^†Gi(k, P)x+

+λ²

4ν²Ei(k, V) ¯Bi(k)Ri(k, P)^†(Ei(k, V) ¯Bi(k)) ,

where we have used the fact that Ri(k, P)^†Ri(k, P)Ri(k, P)^† = Ri(k, P)^†. Thus we have

ν[xAi(k, P)x+ 2xGi(k, P)u+uRi(k, P)u] +λ²

ν Ei(k, γ)− (21)

−λEi(k, V)[ ¯A_i(k)x+ ¯B_i(k)u] =

=νxAi(k, P)x−λEi(k, V) ¯Ai(k)x+λ²

ν Ei(k, γ) +f(u) =

=ν[x(Ai(k, P)− Gi(k, P)Ri(k, P)^†Gi(k, P))]x−

−λ[Ei(k, V) ¯A_i(k)− Ei(k, V) ¯B_i(k)Ri(k, P)^†Gi(k, P)]x+

+ν(u+a(x))Ri(k, P)(u+a(x))+

+λ² ν

Ei(k, γ)−1

4Ei(k, V) ¯B_i(k)Ri(k, P)^†(Ei(k, V) ¯B_i(k)) =

=νxPi(k, P)x−λVi(k, P, V)x+λ²

ν Di(k, P, V, γ)+

+ν(u+a(x))Ri(k, P)(u+a(x)),

showing (13) and completing the proof of the proposition.

Fork=T, T −1, . . . ,0 define

P(k) =P(k, P(k+ 1)), P(T) = (LL, . . . , LL), (22)

V(k) =V(k, P(k+ 1), V(k+ 1)), V(T) = (L, . . . , L), (23)

γ(k) =D(k, P(k+ 1), V(k+ 1), γ(k+ 1)), γ(T) = 0.

(24)

Theorem 4. If

(25) Ei(k, V(k+ 1)) ¯B_i(k)∈R(Ri(k, P(k+ 1)))

for eachk= 0,1, . . . , T−1, then the value functionJ(x(k), θ(k), k) is given by J(x(k), θ(k), k) =E

νx(k)P_θ(k)(k)x(k)−λV_θ(k)(k)x(k) +λ²

ν γ_θ(k)(k), (26)

and an optimal control law is given by u(k) =−Rθ(k)(k, P(k+ 1))^†

Gθ(k)(k, P(k+ 1))x(k)− (27)

−λ

2νHθ(k)(k, V(k+ 1)) .

Proof. For k = T there is no control to take, and it follows that J(x(T), θ(T), T) =νE(x(T)LLx(T)−λLx(T)), showing (26) from the defi- nition of P(T), V(T) and γ(T) = 0. Suppose from the induction hypothesis

that (26)-(27) hold fork+ 1. From the Bellman equation, (11) and (13), for x(k) =x,θ(k) =i, we have

J(x, i, k) = inf

u∈R^m

E

J(x(k+ 1), θ(k+ 1), k+ 1)|Fk = (28)

= inf

u∈R^m

νE

x(k+ 1)P_θ(k+1)(k+ 1)x(k+ 1)−

−λV_θ(k+1)(k+ 1)x(k+ 1) + λ²

ν γ_θ(k+1)(k+ 1)|Fk =

=νxPi(k, P(k+ 1))x−λVi(k, P(k+ 1), V(k+ 1))x+λ²

ν Di(k, P, V, γ), with a minimum value reached at u(k) as in (27). Now, (22), (23), (24) and (28) complete the proof.

5. SOLUTION OF PROBLEMS

In this section we solve the three mean-variance problems stated in Sec- tion 3. We assume throughout this section that (25) holds. Let Π

P1 σ²

, Π (P2 ( )), Π (P3 (ν)) and Π (A(λ, ν)) denote, respectively, the set of optimal solutions for problems P1

σ²

, P2 ( ), P3 (ν) and A(λ, ν). We recall the following results proved in [5].

Proposition 5. If u ∈Π (P3 (ν)) and λ= 1 + 2νE(y^u(T)), then u ∈ Π (A(λ, ν)). On the other hand, ifu∈Π (A(λ, ν))then a necessary condition for u∈Π (P3 (ν))is that λ= 1 + 2νE(y^u(T)).

Proposition 6. Suppose that ν≥0 andu∈Π (P3 (ν)).

a) If Var (y^u(T)) =σ² then u∈Π P1

σ² . b)If E(y^u(T)) = then u∈Π (P2 ( )).

We shall next derive some expressions forλandνsuch that the conditions of Propositions 5 and 6 will be verified, yielding a solution of problemsP1

σ² , P2 ( ) and P3 (ν). For i= 1, . . . , N define

K_i(k) =Ri(k, P(k+ 1))^†Gi(k, P(k+ 1)), (29)

U_i(k) =Ri(k, P(k+ 1))^†B¯_i(k)Ei(k, V(k+ 1)),

A^cl_i (k) = ¯A_i(k)−B¯_i(k)K_i(k), C_i(k) =π(k) ¯B_i(k)U_i(k), A(k) =





p₁₁(k)A^cl₁(k) . . . p_N1(k)A^cl_N(k) ... . . . ... p_1N(k)A^cl₁(k) . . . p_{N N}(k)A^cl_N(k)



,

V(k) =









 N i=1

p_i1(k)C_i(k) ... N

i=1

p_iN(k)C_i(k)











, I=

I . . . I ,

a=L

I

T−1 =0

A()

µ(0), b= 1 2L

_T−1

t=0

I k−1

=t+1

A()

V(t)

,

c= N

i=1

tr(Pi(0)Qi(0)), d= N i=1

Vi(0)µi(0), e= N

i=1

πi(0)γi(0).

We present next an explicit formula for E(y^u(T)) and Var(y^u(T)) in terms of λ, a, b, c, d, ewhen the optimal control strategy (27) is applied to system (4).

Proposition7. Suppose that the optimal control strategy(27)is applied to system (4). Then

(30) E(y^u(T)) =a+λ

νb (31) Var(y^u(T)) =c−a²−

λ ν

b 2

2(d−a) b +4a−

λ

ν 2

1−e

b−b

. Proof. Using the control law (27) in (4), we get

x^u(k+1) =

A^cl_θ(k)(k)+

ε s=1

A_θ(k),s(k)w^x_s(k)−B_θ(k),s(k)K_θ(k)(k)w_s^u(k)

x^u(k)+

+ λ 2ν

B¯_θ(k)(k) + ε s=1

B_θ(k),s(k)w^u_s(k)

U_θ(k)(k).

(32)

Defining z_i^u(k) = x^u(k)1_{θ(t)=i}, and µi(k) = E(z_i^u(k)), µ(k) = (µ₁(k)· · · µN(k)), from [2] we have

µ_j(k+ 1) = N

i=1

p_ij(k)A^cl_i (k)µ_i(k) + λ 2ν

N i=1

p_ij(k)π_i(k) ¯B_i(k)U_i(k), or, in other words,

µ(k+ 1) =A(k)µ(k) + λ 2νV(k).

(33)

Iterating (33) we get µ(k) =

k−1 =0

A()

µ(0) + λ 2ν

k−1

t=0

k−1 =t+1

A()

V(t).

(34)

From (34) we have

E(x^u(T)) =E N

i=1

z^u_i(T)

= N

i=1

E(z_i^u(T)) = (35)

= N i=1

µi(T) =Iµ(T) =I T−1

=0

A()

µ(0) + λ 2ν

T−1 t=0

I k−1

=t+1

A()

V(t).

Sincey^u(T) =Lx^u(t), it follows that E(y^u(T)) =L

I

T−1 =0

A()

µ(0)+ λ 2νL

_T−1

t=0

I k−1

=t+1

A()

V(t)

=a+λ νb, showing (30). To show (31), notice that from (26) we have

E(νy^u(T)²−λy^u(T)) =E

νx(0)P_θ(0)(0)x(0)−λV_θ(0)(0)x(0)+λ²

ν γ_θ(0)(0)

= (36)

=ν N

i=1

tr(P_i(0)Q_i(0))−λ N

i=1

V_i(0)µ_i(0)+λ² ν

N i=1

π_i(0)γ_i(0) =νc−λd+λ² ν e.

Therefore, it follows from (30) and (36) that νVar(y^u(T)) =ν

E(y^u(T)²)−E(y^u(T))²

=

=νE(y^u(T)²)−λE(y^u(T)) +λE(y^u(T))−νE(y^u(T))² =

=νc−λd+λ² ν e+

λ−ν

a+λ

νb a+λ νb

, thus showing (31).

Next, we obtain the values of λ and ν such that the conditions in Propositions 5 and 6 hold in order to obtain a solution of problems P3 (ν), P1

σ²

andP2 ( ). First, we determine the value ofλsatisfying the equation λ= 1 + 2νE(y^u(T)). From (30) we have

λ= 1 + 2νE(y^u(T)) = 1 + 2ν

a+λ νb

,

hence an optimal strategyu for problem P3 (ν) is given by (27) with

(37) λ= 1 + 2νa

1−2b .

Next, we determine the value of ν such that E(y^u(T)) = . From (30) and (37) we have

=E(y^u(T)) =a+

1+2νa 1−2b

ν b, hence

(38) ν= b

(1−2b)−a.

Finally, we determine the value of ν such that Var (y^u(T)) = σ². Define f = c−a², g = _b

2 2^(d−_b^a) + 4a , h = b

1−^e_b −b

and υ = ^λ_ν. It follows from (31) that

hυ²−gυ+ (f −σ²) = 0, so thatυ= ₂^g_h ±_g

2h

₂

−

f−σ²

h . Butυ= ^λ_ν =

1+2νa 1−2b

ν , so we have

(39) ν = 1

υ(1−2b)−2a with the signal inυchosen such that ν >0.

Acknowledgements. This work was partially supported by CNPq (Brazilian Na- tional Research Council), Grant 304866/03-2, CAPES (Brazilian Ministry of Educa- tion Agency), FAPESP (Research Council of the State of S˜ao Paulo), Grant 03/06736- 7, IM-AGIMB, and PRONEX, Grant 015/98.

REFERENCES

[1] S. Chen, X. Li and X.Y. Zhou, Stochastic linear quadratic regulators with indefinite control weight costs.SIAM J. Control Optim.36(1998), 1685–1702.

[2] O.L.V. Costa, M.D. Fragoso and R.P. Marques, Discrete-Time Markov Jump Linear Systems.Springer-Verlag, 2005.

[3] V. Dr˘agan and T. Morozan, The linear quadratic optimization problems for a class of linear stochastic systems with multiplicative white noise and markovian jumping.IEEE Trans. Automat. Control49(2004), 665–675.

[4] Markus Leippold, Fabio Trojani and Paolo Vanini,A geometric approach to multiperiod mean variance optimization of assets and liabilities.J. Econom. Dynamics Control28 (2004), 1079–1113.

[5] Duan Li and Wan-Lung Ng, Optimal dynamic portfolio selection: Multiperiod mean- variance formulation. Math. Finance10(2000), 387–406.

[6] X. Li and X.Y. Zhou,Indefinite stochastic LQ controls with Markovian jumps in a finite time horizon.Comm. Inform. and Systems2(2002), 265–282.

[7] X. Li, X.Y. Zhou and M. Ait Rami, Indefinite stochastic linear quadratic control with markovian jumps in infinite time horizon.J. Global Optim.27(2003), 149–175.

[8] A. Lim and X.Y. Zhou, Stochastic optimal LQR control with integral quadratic con- straints and indefinite control weights.IEEE Trans. Automat. Control44(1999), 1359–

1369.

[9] Y. Liu, G. Yin, and X.Y. Zhou,Near-optimal controls of random-switching LQ problems with indefinite control weight costs.Automatica41(2005), 1063–1070.

[10] H. Markowitz.Portfolio Selection: Efficient Diversification of Investments.Wiley, New York, 1959.

[11] A.C.M. Ran and H.L. Trentelman,Linear quadratic problems with indefinite cost for discrete time systems.SIAM J. Matrix Anal. Appl.14(1993), 776–797.

[12] A. Saberi, P. Sannuti and B.M. Chen,H2-Optimal Control.Prentice Hall, 1995.

[13] U. C¸ akmak and S. ¨Ozeckici,Portfolio optimization in stochastic markets.Math. Meth- ods Oper. Res.63(2006), 151–168.

[14] G. Yin and X.Y. Zhou, Markowitz’s mean-variance portfolio selection with regime switching: From discrete-time models to their continuous-time limits.IEEE Trans. Au- tomat. Control49(2004), 349–360.

[15] X.Y. Zhou and D. Li,Continuous-time mean-variance portfolio selection: A stochastic LQ framework.Appl. Math. Optim.42(2000), 19–33.

[16] Shu-Shang Zhu, Duan Li and Shou-Yang Wang,Risk control over bankruptcy in dynamic portfolio selection: A generalized mean-variance formulation. IEEE Trans. Automat.

Control49(2004), 447–457.

Received 30 September 2006 Escola Polit´ecnica da

Universidade de S˜aoPaulo Departamento de Engenharia de

Telecomunica¸c˜oes e Controle 05508-900 So Paulo SP, Brazil

oswaldo@lac.usp.br okimura@usp.br