The expected cumulative operational time for finite semi-Markov systems and estimation

RAIRO Oper. Res. 41(2007) 399–410 DOI: 10.1051/ro:2007029

THE EXPECTED CUMULATIVE OPERATIONAL TIME FOR FINITE SEMI-MARKOV SYSTEMS AND

ESTIMATION

Brahim Ouhbi

, Ali Boudi

and Mohamed Tkiouat

Abstract. In this paper we, firstly, present a recursive formula of the empirical estimator of the semi-Markov kernel. Then a non-parametric estimator of the expected cumulative operational time for semi-Markov systems is proposed. The asymptotic properties of this estimator, as the uniform strongly consistency and normality are given. As an illus- tration example, we give a numerical application.

Keywords.Expected cumulative operational time, Semi-Markov pro- cess, Non-parametric Estimation.

Mathematics Subject Classification.^60K20.

1. Introduction

Semi-Markov models are common modelling tools in the analysis of machines subject to stochastic failures (cf. Limnios and Oprishan [4]). In this paper we con- sider systems with finite number of states and random holding times in each state.

This consideration relaxes the exponential assumption and provides a rich class of models applicable in reliability, maintenance studies and survival analysis (cf.

Janssen and Limnios [16]). However, in practice, data analysis for semi-Markov processes can be quite difficult. There are many competing models that relax the exponential assumption in modelling complex multistate systems in both reliability and survival analysis (see i.e., Huzurbazar [2]). In this paper, a recursive formula

Received May 11, 2005. Accepted February 6, 2007.

1Ecole Nationale Sup´´ erieure d’Arts et M´etiers, Marjane II, Mekn`es Ismailia, B.P. 4024 B´eni M’Hamed, Mekn`es, Maroc;ouhbib@yahoo.co.uk

2Office National des Chemins de Fer, Agdal, Rabat, Maroc

3Ecole Mohammadia d’Ing´´ enieurs, Agdal, B.P. 765, Rabat, Maroc

c EDP Sciences, ROADEF, SMAI 2007

Article published by EDP Sciences and available at http://www.rairo-ro.org or http://dx.doi.org/10.1051/ro:2007029

to estimate the semi-Markov kernel is proposed. The cumulative operational time Co(t), t >0 of semi-Markov systems is the total time spent by the process in the set of up states during the time interval [0, t].This indicator is of great interest in maintenance studies because it can be used to minimize the expected cost of the maintenance process by choosing the appropriate actions. The distribution of the cumulative operational time has been the subject of many papers (Smith et al.

[12], Kulkarniet al.[3]). A closed form expression for the cumulative distribution function ofCo(t) in the semi-Markov case under the additional assumption that the sequences of operational and unoperational periods are independent was given by Rubino and Sericola [15]. Csenski [13], described a method for computing the cumulative operational time for semi-Markov processes.

To our knowledge there is no work on the estimation of the expected cumulative operational time of a semi-Markov system. In this paper we are concerned by developing an algorithm to estimate the expected cumulative operational time E[Co(t)], t > 0 of the semi-Markov systems. It is seen that this estimator is uniformly strongly consistent and converges weakly to a zero mean normal random variable.

This article begins by describing the model. In Section 3 we present a recursive method to estimate the embedded Markov transition matrix and the distribution of the holding time which in turns allow us to give an algorithm to estimate the semi- Markov kernel. In Section 4, we give the expression of the expected cumulative operational timeE[Co(t)], t > 0 of the semi-Markov systems. Then we propose an estimator of E[Co(t)], t > 0. This estimator is seen to be consistent and to converge to a normal random variable as the time of observation becomes large.

Finally, Section 5, presents a numerical example of a three state semi-Markov process.

2. Preliminaries

Consider a Markov renewal process (MRP) (J, S) = (J_n, S_n)_n≥0 defined on a probability complete space, where J_n is a Markov chain with values in E = {1, ..., s}, the state space of the process and S_n the jump times process. The random variables J₀,J₁, ...,J_n,... are the consecutive states to be visited by the MRP andX₁, X₂, ... defined byX₀ = 0 andX_n =S_n−S_n−1,for n≥1, are the sojourn times in these states taking values in [0,∞).

A MRP can be completely determined if we know its initial law and its transition probabilities defined respectively by :P(J₀=k) =p(k) and

P[J_n+1=k, X_n+1≤x|J₀, J₁, ..., J_n, X₁, X₂, ..., X_n] =Q_Jnk(x) (a.s.) for allx∈[0,+∞) and 1≤k≤s.

The probabilitiesp_ij =Q_ij(∞) (= lim

t→+∞Q_ij(t)) are the transition probabilities of the embedded Markov chainJ_n.

Let us, also consider the distribution function associated to sojourn time in state

ibefore going to state j defined by:

F_ij(.) =

p⁻¹_ij ×Q_ij(.) ifp_ij>0

0 otherwise.

So, we have:

P[J_n=j/J₀, J₁, ....J_n−1=i] =p_ij for all n >0.

P[X_n≤x/J₀, ...J_n−1=i, J_n =j] =F_ij(x) for all n≥0 and x≥0.

P[X_n1 ≤x₁, X_n2 ≤x₂, ...X_nk ≤x_k/J_n, n≥0] = k i=1

F_Jni−1Jni(x_i) (a.s.) for 0≤n₁≤n₂≤...≤n_k and x_i ≥0 for i= 1, ..., k.

The Markov renewal matrix,ψ(t) is defined by ψ(t) =E[N(t)] =

∞ l=0

Q^(l)(t),

whereN(t) is the counting process of transitions of the process up to time tand Q⁽¹⁾_ij (t) =Q_ij(t) and forl >1,Q^(l)(t) is thelth convolution ofQ(t) in the Stieltjes sense and

Q⁽⁰⁾_ij (.) =

1_{i=j}(t) ift >0

0 otherwise.

Let us recall that since the state space,E, is finite,ψ(t) is element wise finite for everyt≥0,i.e. the MRP is normal see [8]

e semi-Markov transition matrix function of the semi-Markov process, (Z_t)_t≥0, is defined by:

P_ij(t) =P[Z_t=j|Z₀=i] =P[J_Nt=j|J₀=i] i, j∈E, It is known,cf.[11], that

P_ij(t) = 1_{i=j}(1− s k=1

Q_ik(t)) +

k∈E

_t

0 P_kj(t−s)Q_ik(ds).

By solving the above Markov renewal equation,cf. [4], it is seen that, in matrix notation,

P(t) = (I−Q(t))⁽⁻¹⁾∗(I−diag(Q(t)1)), (1) wherediag(·) is a diagonal matrix ofith entry_s

j=1Q_ij(t) and1= (1,1, ...,1)^t. The semi-Markov transition matrix is a very useful matrix function in studying the semi-markov processes asymptotic properties and in their applications see [9].

3. Non-parametric estimation of the semi-Markov kernel

In the following, we will perform the non-parametric estimator given by [7], by giving a recursive formula which allows us to make estimation procedures easy in computation.

Let us also define the process (N(t), t∈IR⁺),byN(t) := sup{n≥0 :S_n≤t}, which counts the number of jumps of Z in the time interval (0, t]. Let N_i(t, t) be the number of visits ofZ to state i ∈ E between timet and time t and let N_ij(t, t) be the number of jumps from stateito statej between timetand time t. In the rest, we will denoteN_i(t) =N_i(0, t) andN_ij(t) =N_ij(0, t). that is,

N_i(t) : =

N(t)

k=1

1_{Jk=i}= ∞ k=1

1_{{Jk=i,Sk≤t}},

N_ij(t) : =

N(t)

k=1

1{Jk−1=i,Jk=j}= ∞ k=1

1{Jk−1=i,Jk=j,Sk≤t}.

Let us suppose that we have one observation in the time interval [0, T₁], (Z(t),0≤ t≤T₁),that is {J₀, X₁,· · ·, J_N(T), X_N(T1), U_T1},whereU_T1:=T₁−S_N(T1). The main problem here is that, when we have observed the history of the process until timeT₁,we can get the maximum likelihood estimator of the semi-Markov kernel from [7], but what one can do to actualize the estimator if the data fromT₁ toT₂,T₁< T₂, is available? Of course, one can consider the history of the process on [0, T₂],and then do as in [7], but this increases the complexity of calculation.

In the sequel of this section we will present a recursive formula to answer this problem.

Proposition 1. The non-parametric estimator of the transition matrix of the embedded Markov chain and the non-parametric estimator of the distribution of the sojourn time can be actualized in the following manner:

ˆ

p_ij(T + 1) =A_i(T+ 1).pˆ_ij(T) +W_ij(T + 1), Fˆ_ij(t, T + 1) =C_ij(T+ 1).Fˆ_ij(t, T) +D_ij(t, T+ 1), where,

A_i(T+ 1) = N_i(T)

N_i(T+ 1) and W_ij(T+ 1) = N_ij(T, T + 1) N_i(T+ 1) and

C_ij(T + 1) = N_ij(T)

N_ij(T+ 1) and D_ij(t, T+ 1) =

_Nij(T+1)

l=Nij(T)+11_{Xijl≤t}, N_ij(T+ 1) , whereX_ijl is thel^th sojourn time in state ibefore going to statej, N_ij(T+ 1) = N_ij(T) +N_ij(T, T + 1) andN_i(T + 1) =N_i(T) +_s

j=1N_ij(T, T + 1).

Proof. From [7], the non-parametric estimator of the transition matrix of the em- bedded Markov chain and the non-parametric estimator of the distribution of the sojourn time are given respectively by:

ˆ

p_ij(T+ 1) = N_ij(T+ 1) N_i(T+ 1) Fˆ_ij(t, T + 1) = 1

N_i(T + 1)

Nij(T+1) l=1

1_{Xijl≤t}.

For the first part of the proposition, see that : ˆ

p_ij(T+ 1) = N_ij(T+ 1) N_i(T+ 1)

= N_ij(T) +N_ij(T, T + 1) N_i(T) +N_i(T, T+ 1)

= N_i(T)

N_i(T+ 1).pˆ_ij(T) +N_ij(T, T+ 1)

N_i(T+ 1) , (2)

= A_i(T+ 1).pˆ_ij(T) +W_ij(T + 1),

the Equation (2) was obtained by dividing the numerator and the denominator by N_i^a(T).For the second part, we have that:

Fˆ_ij(t, T+ 1) = 1 N_ij(T+ 1)

Nij(T+1) l=1

1_{Xijl≤t},

= N_ij(T)

N_ij(T+ 1)Fˆ_ij(t, T+ 1) + 1 N_i(T+ 1)

Nij(T+1) l=Nij(T)+1

1_{Xijl≤t},

= C_ij(T + 1).Fˆ_ij(t, T) +D_ij(t, T+ 1), (3)

which is the desired result.

Remark 1.

(1) From (2) and (3) we get a recursive formula to estimate the semi-Markov kernel

Qˆ_ij(t, T) = ˆp_ij(T) ˆF_ij(t, T).

(2) Since the process that we consider is recurrently positive, we have that ˆ

p_ij(T) −→ ν_j, the equilibrium measure of the embedded Markov chain, and

i,j∈Umax sup

t∈IR⁺

|Fˆ_ij(t, T)−F_ij(t)| →0 a.s.,

the estimation algorithm is convergent and we must stop the algorithm when the needed degree of precision is reached.

(3) In Markov case, a similar result was obtained in Boudi [1].

4. Non-parametric estimation of the expected cumulative operational time

In reliability studies, the state spaceE is usually partitioned into two subsets.

The first one, sayU, consists in the up states and the second one, sayD, consists in the down states. The entrance into a state might correspond to the occurrence of a critical event such component failure due to some cause or repair achivement. We assume here that the system is repairable and thus the process alternates between U andD.

The cumulative operational time is defined by

Co(t) = _t

0 1_{Zu∈U}du. (4)

It is the total spent by the semi-Markov process Z in the set of up states U during the time interval [0, t] (see for example [13], who described a method for computing the cumulative operational time for semi-Markov processes). It is easy to see that when the embedded Markov Chain, ((J_n)_n≥0) is irreducible, recurrent, and aperiodic with equilibrium measureν, then

t→+∞lim Co(t)

t =

i∈Uν_im_i s k=1

ν_k.m_k ,

wherem_iis the mean sojourn time in statei. A mathematical proof of this result can be found in [4] or in Ross [14]. However, one can reason heuristically that sinceν_iis the proportion of transitions that are into statei, sincem_i is the mean sojourn time in statei, the average system availability, if stationary, should be

i∈U

ν_im_i s k=1

ν_k.m_k

·

The problem of estimation of the stationary distribution of ergodic semi-Markov processes, was considered by [5].

The quantity that we want to study is the expected cumulative operational time of a semi-Markov system which is given by

E[Co(t)] = E[

_t

0 1_{Zu∈U}du]

= _t

0 P(Z_u∈U)du

=

j∈U

_t

0 P(Z_u=j)du

= s i=1

j∈U

_t

0

α_iP_ij(u)du, (5)

whereP_ij(u) is given in (1) andαis the initial probability of the process.

The expected cumulative operational time is an important indicator in the maintenance studies since it allows us to derive the average system availability given by

A(t) =¯ 1 t

j∈U

_t

0 P_j(u)du.

Theorem 1. The expected cumulative operational time (5) can be written as:

E[Co(t)] = s

i=1

j∈U

_t

0

α_iM_j(t−x)dψ_ij(x),

whereM_j(x) =_x

0 H¯_j(u)du,

Proof. Of course, this quantity exists because the process which we consider is finite positive (ψ_ij(t)<∞cf. [6]) andM_j(t)< t.

From (5), we have that E[Co(t)] =

s i=1

j∈U

_t

0 α_iP_ij(u)du,

= s i=1

j∈U

_t

0 α_{i u}

0

H¯_j(u−x)dψ_ij(x)du,

= s i=1

j∈U

_t

0 α_i( _t−x

0

H¯_j(u)du)dψ_ij(x),

= s i=1

j∈U

_t

0 α_iM_j(t−x)dψ_ij(x), . (6) which is the desired result.

In the sequel, we are concerned with the non-parametric estimation of the ex- pected cumulative operational time of a semi-Markov system.

From ˆP(t), we propose to estimateE[Co(t)] by:

E[Co(t)] =ˆ s i=1

j∈U

α_i( ˆM_j∗ψˆ_ij)(t) (7)

where ˆM_j(t, T) = _N¹

j(T)

_Nj(T)

l=1 (X_jl∧t).

In the sequel we will prove that the proposed estimator is unifomly strongly consistent and converges in law to a normal random variable.

Theorem 2.The estimator of the expected operational time,E[Co(t)]ˆ is uniformly strongly consistent in the sense that

maxi,j sup

t∈[0,L]|E[Co(t, Tˆ )]−E[Co(t)]| −→0 a.s., as T → ∞.

Proof. From (7), we have maxi,j sup

t∈[0,L]|E[Co(t, Tˆ )]−E[Co(t)]| ≤max

i,j sup

t∈[0,L]

s i=1

j∈U

α_i|( ˆM_j∗ψˆ_ij)(t)−(Mj∗ψ_ij)(t)|

≤max

i,j sup

t∈[0,L]

s i=1

j∈U

α_i|( ˆM_j−M_j)(t)|.ψij(t) +|Mˆ_j(t)|.|ψˆ_ij(t)−ψ_ij(t)|.

From [6], we have that, maxj sup

t∈IR⁺

|Hˆ_j(t, T)−H_j(t)| −→0 a.s. when T −→ ∞,

which implies that maxj sup

t∈[0,L]

s i=1

j∈U

α_i|( ˆM_j−M_j)(t)| −→0 a.s. when T −→ ∞.

For anyL∈IR⁺, we have maxi,j sup

t∈[0,L]|ψˆ_ij(t, T)−ψ_ij(t)| −→0 a.s. when T −→ ∞, and the fact thatψ_ij(t)<∞and ˆM_j(t)< t, we get the desired result.

Theorem 3. For any fixed t, t∈[0,∞), T^1/2(E[Co(t, Tˆ )]−E[Co(t)]) converges in law to a zero mean normal random variable with variance

σ²(t) = s

i=1

s j=1

µ_ii· {(φij)²∗Q_ij−(φ_ij∗Q_ij)²

+1_{{j∈U} ∞}

0

_∞

0 (x∧(t−u)−M_i)dA_i(u) ₂

dQ_ij(x)

−

1_{j∈U}

_∞

0

_∞

0 (x∧(t−u)−M_i)dA_i(u)dQ_ij(x) ₂

+2.1_{j∈U}

_∞

0 φ_ij.(t−x) _∞

0 (x∧(t−u)−M_i)dA_i(u)dQ_ij(x)

−2.1_{j∈U}(φ_ij∗Q_ij)(t).(A_i∗((x∧.−M_i))(t)}

whereA_i(t) =_s

k=1α_kψ_ki(t)andφ_kl(t) =_s

i=1

j∈Uα_i(ψ_ik∗ψ_lj∗M_j)(t).

Proof. From (7), we see that

T^1/2(E[Co(t, Tˆ )]−E[Co(t)]) = s i=1

j∈U

α_iT^1/2( ˆM_j∗ψˆ_ij)(t)−(M_j∗ψ_ij)(t)

= s i=1

j∈U

α_iT^1/2[( ˆM_j−M_j)∗( ˆψ_ij−ψ_ij)(t) + ( ˆM_j−M_j)∗ψ_ij)(t) +M_j∗( ˆψ_ij−ψ_ij)(t)] (8) From [6], the first term of (8) converges to zero. ThenT^1/2(E[Co(t, Tˆ )]−E[Co(t)]) has the same limit as

s i=1

j∈U

α_iT^1/2 ( ˆM_j−M_j)∗ψ_ij)(t) +M_j∗( ˆψ_ij−ψ_ij)(t)

.

From [6], has the same limit as s

i=1

j∈U

α_iT^1/2

⎡

⎣ 1 N_j(T)

Nj(T) r=1

(X_jl∧t−M_j)∗ψ_ij)(t)+

_s

k=1

s l=1

M_j∗ψ_ik∗ψ_lj

∗( ˆQ_kl−Q_kl)(t)

,

which is equal to s

i=1

j∈U

α_iT^1/2

⎡

⎣ 1 N_j(T)

Nj(T) r=1

(X_jl∧t−M_j)∗ψ_ij)(t)

+ _s

k=1

s l=1

M_j∗ψ_ik∗ψ_lj

∗

⎛

⎝ 1 N_k(T)

Nk(T) r=1

(1_{{Jr+1=l,Xr≤.}}−Q_kl)(t)

⎞

⎠1_{Jr=k}

⎤

⎦,

which can be written as s

k=1

s l=1

T^1/2 N_k(T)

Nk(T) r=1

[1_{{Jr=k,k∈U,Jr+1=l}}(X_r∧t−M_k)∗A_k)(t)

+ (B_kl∗(1_{{Jr+1=l,Xr≤.}}−Q_kl)(t))1_{Jr=k}], whereA_k(t) =_s

i=1α_iψ_ik(t) andφ_kl(t) =_s

i=1

j∈Uα_i(ψ_ik∗ψ_lj∗M_j)(t). Since

NkT(T)→µ_kk,we can consider the function

f(J_r, J_r+1, X_n) =µ_kkA_k∗(X_n∧t−M_k)1_{{Jr=k,k∈U,Jr+1=l}}

+µ_kkφ_kl∗(1_{{Jr+1=l,Xr≤.}}−Q_kl)(t)1_{Jr=k}. By the central limit theorem of semi-Markov process (see [10]) or as in [6] for this

function we get the desired result.

5. Numerical application

Let us consider a three state semi-Markov system as illustrated in Figure 1, whereF₁₂(x) = 1−exp(−λ1x),F₂₁(x) = 1−exp[−(_α^x₁)^β1],F₂₃(x) = 1−exp[−(_α^x₂)^β2], and F₃₁(x) = 1−exp(−λ₂x), for x≥0. λ₁ = 0.1, λ₂ = 0.2,α₁ = 0.3, β₁ = 2, α₂= 0.1 andβ₂= 2.

The transition probability matrix of the embedded Markov chain (J_n) is:

P =

⎛

⎝ 0 1 0 p 0 1−p

1 0 0

⎞

⎠,

wherepis given by

p= _∞

0 [1−F₂₃(x)]dF₂₁(x).

In Figure 2, we present the estimation of the cumulative operational time for the three state semi-Markov system given in Figure 1. As an appliccation to maintenance studies, we give in Figure 3 the estimation of the average system availability for this system.

Figure 1. A three state semi-Markov system.

Figure 2. The estimation of the cumulative operational time for the three state semi-Markov system.

6. Concluding remarks

The main impediment to the semi-Markov use in practice is computational complexity. In this paper, we have developed a recursive formula of the empir- ical estimator of the semi-Markov kernel. By making use of this estimator, we have proposed an estimator for the expected cumulative operational time of semi- Markov systems and we have proved its asymptotic properties. This indicator is of great interest in maintenance studies. Finally, it should be mentioned that there is a need for applications of this quantity to minimize the expected cost of the maintenance process by choosing the appropriate actions.

Figure 3. The estimation of the average system availability.

References

[1] A. Boudi,Sur le contrˆole adaptatif des populations de chaines de Markov finies. Th`ese de 3`eme cycle, Facult´e des Sciences, Rabat, Morocco (1996).

[2] A. Huzurbazar, Flowgraph models for multistate Time-to-Event Data. Wiley, New York (2005).

[3] V.G. Kulkarni, V.F. Nicola and K.S. Trivedi, The completion time of a job on multimode systems.Adv. Appl. Probab.19(1987) 932–954.

[4] N. Limnios and G. Opri¸san, Semi-Markov Processes and Reliability. Birkh¨auser, Boston (2001).

[5] N. Limnios, B. Ouhbi and A. Sadek, Empirical Estimator of Stationary Distribution For Semi-Markov Processes.Commun. Stat. Theory Methods34(2003) 987–995.

[6] B. Ouhbi and N. Limnios, Non-parametric estimation for semi-Markov processes based on its hazard rate.Stat. Inference Stoch. Process.2(1999) 151–173.

[7] B. Ouhbi and N. Limnios, Non-parametric estimation for semi-Markov kernels with appli- cation to reliability analysis.Appl. Stoch. Models Data Anal.12(1996) 209–220.

[8] B. Ouhbi and N. Limnios, The Rate of Occurrence of Failures for Semi-Markov Processes and Estimation.Stat. Probab. Lett.59(2001) 245–255.

[9] B. Ouhbi and N. Limnios, Non-parametric Reliability Estimation of Semi-Markov Processes.

J. Stat. Plan. Inference109(2003) 155–165.

[10] R. Pyke and R. Schaufele, The existence and uniqueness of stationary measures for Markov renewal processes.Ann. Math. Stat.37(1966) 1439–1462.

[11] R. Pyke, Markov renewal processes: definitions and preliminary properties. Ann. Math.

Stat.32(1961) 1231–1241.

[12] R.M. Smith, K.S. Trivedi and A.V. Ramesh, Performability analysis : measures, an algo- rithm, and a case study.IEEE Trans. Comput.C-37(1988) 406–417.

[13] A. Scenski, Cumulative operational time analysis of finite semi-Markov reliability models.

Reliab. Eng. Syst. Saf.44(1994) 17–25.

[14] S. Ross,Applied probability models with optimization applications. Dover New York (1992).

[15] G. Rubino and B. Sericola, Interval availability analysis using operational periods.Perform.

Eval.14(1992) 257–272.

[16] J. Janssen and N. Limnios Eds.,Semi-Markov models and Applications. Kluwer Academic, Dordrecht (1999).