THE CONVERGENCE OF A FINITE MARKOV CHAIN
ALINA NICOLAIE
We give a sufficient condition for the convergence of a finite state nonhomogeneous Markov chain in terms of convergence of a certain series.
AMS 2000 Subject Classification: 60J10.
Key words: simulated annealing, finite Markov chain.
1. PRELIMINARIES
Consider a finite Markov chain with state space S = {1, . . . , r} and transition matrices (Pn)n≥1. We shall refer to it as the finite Markov chain (Pn)n≥1. For all integersm≥0,n > m, define
Pm,n =Pm+1Pm+2· · ·Pn= ((Pm,n)i,j)i,j∈S. Assume that the limit
(1.1) lim
n→∞Pn=P
exists and that the limit matrixP hasp≥1 irreducible aperiodic closed classes and, perhaps transient states, so that it has the form
(1.2)
S1 0 · · · 0 0
0 S2 · · · 0 0
· · · ·
0 0 · · · Sp 0
L1 L2 · · · Lp T
,
where Si are ri×ri transition matrices, i = 1, p, associated with the p irre- ducible aperiodic closed classes,T concerns the transitions of the chain as long as it stays in the r−Pp
t=1rt transient states and the Li concern transitions from the transient states into the ergodic sets corresponding to Si,i= 1, p.
Such a class of Markov chains was, e.g., proposed as the mathematical model for simulated annealing, a stochastic algorithm for global optimization.
We refer to van Laarhoven and Aarts [8] for a general exposition and historical background.
MATH. REPORTS10(60),1 (2008), 57–71
Definition 1.1. A probability distribution µ= (µ1, . . . , µr) is said to be invariant with respect to an r×r stochastic matrix P ifµP =µ.
We shall need the following result
Theorem1.2. Consider a finite homogeneous Markov chain with state space S and transition matrix P of the form (1.2). Then
(1.3) lim
n→∞Pn=
Γ1 0 · · · 0 0
0 Γ2 · · · 0 0
· · · ·
0 0 · · · Γp 0
Ω1 Ω2 · · · Ωp 0
,
where
Γi=
µ(i)1 · · · µ(i)ri
· · · · µ(i)1 · · · µ(i)ri
, i= 1, p,
are strictly positive ri×ri matrices; each row of the matrixΓi is the invariant probability vector µ(i) = (µ(i)1 , . . . , µ(i)ri) with respect to the matrix Si, i= 1, p, and
Ωi =
µ(i)1 zr1+r2+···+rp+1,i · · · µ(i)rizr1+r2+···+rp+1,i
· · · ·
µ(i)1 zr,i · · · µ(i)rizr,i
are (r −Pp
t=1rt) ×ri matrices, where zj,i= probability that the chain will enter and thus, will be absorbed in the set corresponding to Si given that the initial state is j from the set corresponding to T,j=Pp
t=0rt, r, i= 1, p [with convention r0 = 1].
Proof. For the form of Γi, i = 1, p, see, e.g., [5, p. 123] and for Ωi, i= 1, p, see, e.g., [7, p. 91].
Remark 1.3. Clearly,
(1.4) zj,i ≥0, ∀j=
p
X
t=0
rt, r, ∀i= 1, p, and
(1.5)
p
X
i=1
zj,i = 1, ∀j=
p
X
t=0
rt, r.
Theorem 1.4 (see, e.g., [6]). If P is the transition matrix of a finite Markov chain, then the multiplicity of the eigenvalue 1 is equal to the number of irreducible closed subsets of the chain.
Proof. See, e.g., [6, p. 126].
A vector x∈Cn will be understood as a column vector and x0 denotes the transpose of x.
Theorem1.5. Let A= −Ir+P with P of the form (1.2). Then there exists a nonsingular complex r×r matrix Q such that
(1.6) A=QJ Q−1,
where J is a Jordan r×r matrix, and Qreads as
Q=
1 0 · · · 0 · · ·
· · · ·
1 0 · · · 0 · · ·
0 1 · · · 0 · · ·
· · · ·
0 1 · · · 0 · · ·
0 0 · · · 0 · · ·
· · · ·
0 0 · · · 0 · · ·
0 0 · · · 1 · · ·
· · · ·
0 0 · · · 1 · · ·
zr1+r2+···+rp+1,1 zr1+r2+···+rp+1,2 · · · zr1+r2+···+rp+1,p · · ·
· · · ·
zr,1 zr,2 · · · zr,p · · ·
,
where the first column contains 1 in the 1, r1 rows, the next p−1 columns contains1in theri−1+ 1, rirows,i= 2, p, and the lastr−pcolumns comprise complex numbers. For zj,i, j=Pp
t=0rt, r, i= 1, p, we have the meaning given in Theorem 1.2. The inverse Q−1 has the form
Q−1=
µ(1)1 · · · µ(1)r1 0 · · · 0 0 · · · 0 0 · · · 0
· · · ·
0 · · · 0 0 · · · 0 µ(p)1 · · · µ(p)rp 0 · · · 0
qp+1,1 · · · qp+1,r
· · · · qr,1 · · · qr,r
,
where µ(i) are the invariant probability vectors with respect toSi, i= 1, p, and the last r−p rows comprise complex numbers.
Proof. For the existence ofQsee, e.g., [4, p. 126]. See also [1]. We shall need some spectral properties of A. We have
(1.7) λ1 = 0
is an eigenvalue of A whose algebraic multiplicity is equal to the geometric multiplicity and equal to p. All other distinct eigenvalue λ2, . . . , λl+s of A satisfy
(1.8) |λi+ 1|<1, i= 2, l+s, (1.9) Reλi <0, i= 2, l+s.
Indeed, by Theorem 1.4, the characteristic polynomial of P has the form det(P −ϑIr) = (ϑ−1)pR(ϑ),
where grad R=r−p and R(1)6= 0. Replacing P =Ir+A we obtain det(A−λIr) =λpR(λ+ 1),
where λ=ϑ−1, and so (1.7) follows.
On the other hand, if we set ϑi all other distinct eigenvalues different from 1 of P, with left eigenvectors ϕi,i= 2, l+s, becauseP is stochastic we have |ϑi|<1,i= 2, l+s, and
ϕiP =ϑiϕi, i= 2, l+s.
By subtracting ϕi from both members we obtain ϕiA= (ϑi−1)ϕi, i= 2, l+s.
Hence λi = ϑi −1, i = 2, l+s, are eigenvalues of A. Then (1.8) and (1.9) follow. From (1.7) (see [4, pp. 129–131]) we have
J =
J1 0 · · · 0 0 J2 · · · 0
· · · ·
0 0 · · · Jl 0 · · · ·
0 0 · · · 0 Jl+1 · · · ·
· · · ·
0 0 · · · 0 0 · · · Jl+s
,
where J1 = 0p, i.e., the p ×p zero matrix, Jk is a diagonal mk×mk ma- trix with entries the eigenvalues λk, k = 2, l, whose algebraic and geometric multiplicities are equals, and
Jl+i=
λl+i ε(i)1 0 · · · 0
0 λl+i ε(i)2 · · · 0 0
· · · · 0 0 · · · λl+i ε(i)m
l+i−1
0 0 · · · 0 λl+i
are ml+i×ml+i matrices corresponding to eigenvalues whose geometric mul- tiplicities are smaller then their algebraic multiplicities and ε(i)t ∈ {0,1}, t= 1, ml+i−1,i= 1, s. Clearly, p+m2+· · ·+ml+s=r.
A key step in obtaining the firstpcolumns ofQis the remark that writing (1.6) as AQ =QJ we see that A applied to the tth column ofQ yields zero vector, ∀t= 1, p, hence the vectors corresponding to the first p columns ofQ are right eigenvectors associated with the eigenvalue λ1 = 0 ofA.
Now, we shall determine the right eigenspace corresponding to λ1 by solving the linear system
(1.10) (A−0Ir)X= 0r
which is equivalent toP X =X, whereX= (x1, . . . , xr).
The firstPp
t=1rt rows of the system (1.10) are equivalent to (Si−Iri)·(xr0+r1+···+ri−1, . . . , xr1+···+ri−1+ri) = 0, i= 1, p, with the general solution
(1.11) (xr0+r1+···+ri−1, . . . , xr1+···+ri) =αi(1,1, . . . ,1), αi∈C, i= 1, p.
For the next components xr1+···+rp+1, . . . , xr we note that P X =X⇒PkX =X, ∀k≥1.
For the last r−Pp
t=1rt components, the above equation amounts to
r
X
j=1
(Pk)r1+···+rp+i,jxj =xr1+···+rp+i, i= 1, r−
p
X
t=1
rt, k≥1, Equivalently,
p−1
X
t=0
r1+···+rt+1
X
j=r0+···+rt
(Pk)r1+···+rp+i,jxj+
r
X
j=r1+···+rp+1
(Pk)r1+···+rp+i,jxj =xr1+···+rp+i. For j=Pp
t=0rt, rand i= 1, r−Pp
t=1rtwe have [see 5, p. 91]
(1.12) lim
k→∞(Pk)r1+···+rp+i,j = 0 and for i= 1, r−Pp
t=1rt from representation (1.3) we get
k→∞lim(Pk)r1+···+rp+i,r0+···+rs−1+t−1 =µ(s)t zr1+···+rp+i,s, s= 1, p, t= 1, rs. So, letting k → ∞ in the last sum and using the fact that µ(i), i= 1, p, are probability vectors, we obtain
p
X
j=1
zr1+r2+···+rp+i,jαj =xr1+···+rp+i, i= 1, r−
p
X
t=1
rt.
We thus obtain the general solution
p
X
j=1
αj(0, . . . ,0,1, . . . ,1,0, . . . ,0, zr1+···+rp+1,j, zr1+···+rp+2,j, . . . , zr,j)0, of (1.10), with αj ∈ C, j = 1, p, where the components of the row vectors corresponding to r0+· · ·+rj−1, r1+· · ·+rj are 1 and those corresponding to the remaining ones from 1,Pp
t=1rt, are 0,j = 1, p.
Next, we pick particular values for the parametersαi,i= 1, p, to obtain plinear independent vectors; choosingα1= 1,α2 =· · ·=αp = 0, thenα2= 1, α1 =α3 =· · ·=αp= 0 etc., we obtain the first pcolumns of the matrix Q.
A key step in obtaining the firstprows ofQ−1 is the remark that writing (1.6) as Q−1A=J Q−1 which means that thetth row ofQ−1, applied toA is equal to the tth rows of J applied to Q−1,∀t= 1, p. Since the matrix J Q−1 has the first p rows equal to the zero vector, we conclude that the tth row of Q−1 is a left eigenvector of A associated withλ1= 0, ∀t= 1, p.
Next, we shall determine the left eigenspace corresponding toλ1 by solv- ing the linear system
(1.13) Y0(A−0Ir) = 00r
which is equivalent to Y0P = Y0, where Y0 = (y1, . . . , yr). The first Pp t=1rt equations of system (1.13) are equivalent to
(yr0+r1+···+ri−1, . . . , yr1+···+ri)0·(Si−Iri) = 0, i= 1, p, with the general solution
(yr0+r1+···+ri−1, . . . , yr1+···+ri) =βi(µ(i)1 , . . . , µ(i)ri), βi ∈C, i= 1, p.
For the next components yr1+···+rp+1, . . . , yr we note that Y0P =Y0 ⇒Y0Pk=Y0, ∀k≥1.
For the last r−Pp
t=1rt components, the above equation amounts to
r
X
j=1
yj(Pk)j,r1+···+rp+i =yr1+···+rp+i, i= 1, r−
p
X
t=1
rt, k ≥1, Equivalently,
p−1
X
t=0
r1+···+rt+1
X
j=r0+···+rt
yj(Pk)j,r1+···+rp+i+
r
X
j=r1+···+rp+1
yj(Pk)j,r1+···+rp+i =yr1+···+rp+i. By letting k→ ∞in the last sum, and using (1.12) and (1.3) we get
yr1+···+rp+i= 0, i= 1, r−
p
X
t=1
rt.
We thus obtain the general solution
p
X
j=1
βj(0, . . . , µ(j)1 , . . . , µ(j)rj,0, . . . ,0)0,
of (1.13), with βj ∈ C, j = 1, p, where the components of the vector corre- sponding to the subscripts r0+· · ·+rj−1, r1+· · ·+rj are the components of the invariant vector associated with Sj and the others are equals to 0.
Finally, we pick particular values for the parametersβi,i= 1, p, to obtain plinear independent vectors; choosingβ1= 1,β2 =· · ·=βp = 0, thenβ2= 1, β1=β3 =· · ·=βp = 0, etc., we obtain the firstprows of the matrixQ−1. If A = (Aij) is an m ×n matrix, then (see [4, p. 295]) define the matrix norm
|||A|||∞= max
i=1,m n
X
j=1
|Aij|.
For M ⊆ {1, . . . , m},N ⊆ {1, . . . , n},M, N 6=∅we also define AM×N = (Aij)i∈M,j∈N.
Theorem1.6 (see, e.g., [6]). Let (an)n≥0 be a sequence of real numbers convergent to 0 and P∞
n=0bn an absolute convergent series. Then
n→∞lim
n
X
i=0
aibn−i = 0.
Proof. See, e.g., [6, pp. 34–36].
Theorem1.7 (see, e.g., [3]). Let ||| · |||be a matrix norm. Let (Un)n≥1
be a sequence of matrices such that |||Un||| ≤ 1, ∀n≥ 1. Then the following statements are equivalent.
(i)Q∞
n=rUn converges, ∀r≥1.
(ii)Q∞
n=1[Un+An]converges for all sequences (An)n≥1, when P∞
n=1|||An|||<∞.
Proof. See, e.g., [3, p. 101].
Theorem 1.8 (see, e.g., [4]). Let A be an n×n matrix. Then A is invertible if there exists a matrix norm ||| · ||| such that |||In−A||| < 1. If so, then
A−1 =
∞
X
k=0
(In−A)k. Proof. See, e.g., [4, p. 301].
Theorem1.9 (see, e.g., [4]). (i) If ||| · ||| is an arbitrary matrix norm, then |||I||| ≥1.
(ii) Let A be an n×n matrix. If ||| · ||| is a matrix norm such that
|||A|||<1, then
|||(In−A)−1||| ≤ |||In||| −(|||In||| −1)|||A|||
1− |||A||| .
Proof. See, e.g., [4, pp. 301–302].
2. A CONVERGENCE RESULT
In this section we give our main result following an idea of a theorem of Gidas [1]. Our result is a sufficient condition for the convergence of a finite state nonhomogeneous Markov chain in terms of convergence of a certain series.
Definition 2.1 ([9]). A finite Markov chain (Pn)n≥1 is said to be conver- gent if ∀m≥0 the sequence (Pm,n)n>m converges.
Definition 2.2 ([9]). Two Markov chains (Pn)n≥1 and (Pn0)n≥1 are said to be equivalent if
X
n≥1
|||Pn−Pn0|||∞<∞.
Theorem 2.3 ([2]). For two equivalent chains (Pn)n≥1 and (Pn0)n≥1
we have
∃ lim
n→∞Pm,n if and only if ∃ lim
n→∞Pm,n0 , ∀m≥1.
Consequently, two equivalent stochastic chains converge or diverge simultane- ously.
Proof. See [2].
The general results from Theorem 2.4 below are new. For the special case p= 1 see also [5, p. 226].
Theorem 2.4. Let (Pn)n≥1 be a nonhomogeneous Markov chain with state space S such thatPn→P as n→ ∞. Suppose that P has exactly p≥1 recurrent aperiodic classes Si, i= 1, p, and, possibly, transient states, i.e., P is of the form (1.2). Let µ(i) be the invariant probability vectors with respect to Si, i = 1, p and zj,i, j =Pp
t=1rt+ 1, r, i = 1, p, as in Theorem 1.2. Let Vn=Pn−P, ∀n≥1, where lim
n→∞Vn= 0. LetQ and Q−1 as in Theorem 1.5.
Then
n→∞lim(Pm,n)i,j = 0, i∈S, j=
p
X
t=0
rt, r, ∀m≥0.
If, moreover,
(2.1)
∞
X
n=1
|||(Q−1VnQ)S×M|||∞<∞, where M ={1, . . . , p}, then the chain (Pn)n≥1 is convergent.
Proof. By the Chapman-Kolmogorov equation we have Pm,n =Pm,n−1Pn, 0≤m < n.
(with the convention Pn,n = Ir). Subtracting Pm,n−1 from both members, we obtain
(2.2) Pm,n−Pm,n−1 =Pm,n−1[−Ir+Pn], 0≤m < n.
Setting
(2.3) x(i)n =x(i)n (m) = ((Pm,n)i,1, . . . ,(Pm,n)i,r), 0≤m < n, and
x(i)m =ei, i∈S,
where (ei)i=1,r is the canonical basis of the linear spaceRr, then equation (2.2) reads as
(2.4) x(i)n −x(i)n−1 =x(i)n−1[−Ir+Pn], 0≤m < n, i∈S.
We remark that the x(i)n defined in (2.3) are solutions of equations of the type (2.5) xn−xn−1 =xn−1[−Ir+Pn], n≥1,
with
xn= (xn,1, . . . , xn,r), n≥0, under the conditions
(2.6) xn,i≥0, i= 1, r,
r
X
i=1
xn,i= 1, n≥0.
We are interested in the asymptotic behaviour of (2.5) under conditions (2.6).
Setting
A=−Ir+P
we can benefit of the result given in Theorem 1.5. Also, setting
(2.7) yn=xnQ, n≥0,
and
(2.8) Ven=Q−1VnQ, n≥1,
equation (2.5) becomes
(2.9) yn−yn−1 =yn−1J+yn−1Ven, n≥1.
From (2.7) we have xn=ynQ−1, hence (2.10) xn,i=µ(1)i yn,1+
r
X
j=p+1
yn,jqji, i= 1, r1,
(2.11) xn,r1+···+rt−1+i =µ(t)i yn,t+
r
X
j=p+1
yn,jqj,r1+···+rt−1+i, t= 2, p, i= 1, rt,
(2.12) xn,i=
r−p
X
j=1
yn,p+jqp+j,i, i=
p
X
t=0
rt, r.
Next, equations (2.9) amount to yn,i−yn−1,i=
p
X
j=1
yn−1,j(Ven)j,i+
r
X
j=p+1
yn−1,j(Ven)j,i, i= 1, p,
yn,i−yn−1,i=λ2yn−1,i+
r
X
j=1
yn−1,i(Ven)j,i, i=p+ 1, p+m2,
yn,i−yn−1,i =λtyn−1,i+
r
X
j=1
yn−1,j(Ven)j,i, i=p+
t−1
X
s=2
ms+ 1, p+
t
X
s=2
ms, t= 3, l, yn,p+Pl
s=2ms+1−yn−1,p+Pl
s=2ms+1=λl+1yn−1,p+Pl
s=2ms+1+ +
r
X
j=1
yn−1,j(Ven)j,p+Pl
s=2ms+1, yn,i−yn−1,i=λl+1yn−1,i+ε(1)k yn−1,i−1+ +
r
X
j=1
yn−1,j(Ven)j,i, i=p+
l
X
t=2
mt+ 2, p+
l+1
X
t=2
mt, k= 1, ml+1−1, and similar equations for i=p+Pl+1
t=2mt+ 1, r.
Next, we shall study the asymptotic behaviour ofyn,i,i=p+ 1, r. The boundedness of yn, n≥0, and the fact that lim
n→∞Vn= 0, allow one to assert that
(2.13) Wn,i:=
r
X
j=1
yn−1,j(Ven)j,i→0 as n→ ∞, i=p+ 1, r.
In the above system we shall be concerned with the equations corresponding toi∈ {p+ 1, . . . , p+Pl
t=2mt+ 1}. Equivalently, we can write yn,i= (λ2+ 1)yn−1,i+Wn,i, n≥1, i=p+ 1, p+m2, respectively,
yn,i= (λt+ 1)yn−1,i+Wn,i, n≥1, t= 3, l, i=p+
t−1
X
s=2
ms+ 1, p+
t
X
s=2
ms, Therefore,
(2.14) |yn,i| ≤ |y0,i| |λ2+ 1|n+
n
X
s=1
|Ws,i| |λ2+ 1|n−s, n≥1, i=p+ 1, p+m2. and similar equations do hold for i=p+Pt−1
s=2ms+ 1, p+Pt
s=2ms,t= 3, l.
By Theorem 1.6, using P∞
n=0|λ2+ 1|n<∞(see (1.8)) and (2.13), the second term on the right side of (2.14) converges to zero asn→ ∞. Using also (1.8), we get
n→∞lim yn,i= 0, i=p+ 1, p+m2, regardless of the initial data y0 and, similarly,
(2.15) lim
n→∞yn,i= 0, i=p+m2+ 1, p+
l
X
t=2
mt+ 1.
Next, we pay attention to the equations corresponding toifromp+Pl
t=2mt+2 to p+Pl+1
t=2mt. The case ε(i)k = 0 for some k = 1, ml+1−1 is similar to the preceding one. We are interested now in cases for whichε(i)k = 1. As in (2.14), we obtain
(2.16) |yn,i| ≤ |y0,i| |λl+1+ 1|n+
n
X
s=1
[|ys,i−1|+|Ws,i|]|λl+1+ 1|n−s. For i= p+Pl
t=2mt+ 2, p+Pl+1
t=2mt, the first term on the right hand side of (2.16) converges to 0 as n→ ∞. SinceP∞
n=0|λl+1+ 1|n <∞ (see (1.8)), by Theorem 1.6 (see (1.8), (2.13) and (2.15)), the second term from (2.16) converges to 0 as n→ ∞ fori=p+Pl
t=2mt+ 2. Hence
n→∞lim yn,p+m2+···+ml+2 = 0.
Similarly,
n→∞lim yn,i= 0, i=p+
l
X
t=2
mt+ 3, p+
l+1
X
t=2
mt.
We can thus conclude that
(2.17) lim
n→∞yn,i= 0, i=p+ 1, r, regardless of the initial data y0.
Setting
Rn,i=
r
X
j=p+1
yn,j(Ven+1)j,i, i= 1, p, n≥0.
equation (2.9) for the first p components becomes
(2.18) yn,i−yn−1,i=yn−1,1(Ven)1,i+yn−1,2(Ven)2,i+· · ·+yn−1,p(Ven)p,i+Rn−1,i, i= 1, p. Letm≥0 andi∈S be arbitrarily fixed. Let
xn=x(i)n , n > m,
where x(i)n are defined like in (2.3). It follows using (2.17) and (2.12), that
(2.19) lim
n→∞xn,k = 0, k=
p
X
t=0
rt, r.
and the first conclusion of theorem. In matrix notation, system (2.18) becomes
yn,1
yn,2
· · · yn,p
0
=
yn−1,1
yn−1,2
· · · yn−1,p
0
Cn+
Rn−1,1
Rn−1,2
· · · Rn−1,p
0
,
where
Cn=
1 + (Ven)1,1 (Ven)1,2 · · · (Ven)1,p (Ven)2,1 1 + (Ven)2,2 · · · (Ven)2,p
· · · ·
(Ven)p,1 (Ven)p,2 · · · 1 + (Ven)p,p
.
Setting
Y(n) =
yn,1
· · · yn,p
and R(n) =
Rn,1
· · · Rn,p
, n > m, the matrix equation above becomes
(Y(n))0 = (Y(n−1))0Cn+ (R(n−1))0, n > m.
The solution of this matrix equation is (2.20)
(Y(n))0 = (Y(m))0
n
Y
s=m+1
Cs+
(R(n−1))0+
n−2
X
s=m
(R(s))0
n
Y
t=s+2
Ct
, n > m.
Setting
Cn=Ip+Bn, n > m, we have
|||Bn|||∞= max
k=1,p p
X
j=1
|(Ven)k,j|, n > m.
The convergence ofQn
s=m+1Csfrom (2.20) asn→ ∞follows from Theo- rem 1.7. We shall now prove that the second term from (2.20) converges as n→ ∞, too. It follows from (2.1) that
∃n0 > m+ 2 such that|||Ip−Cn|||∞=|||Bn|||∞<1, n≥n0. So, by Theorem 1.8 we have
∃(Cn)−1, ∀n≥n0 and (Cn)−1=
∞
X
t=0
[Ip−Cn]t=
∞
X
t=0
[−Bn]t, ∀n≥n0, and we can thus write
n−2
X
s=m
(R(s))0
n
Y
t=s+2
Ct+ (R(n−1))0 =
n0−2
X
s=m
(R(s))0
n
Y
t=s+2
Ct+
+
n−1
X
s=n0−1
(R(s))0
s+1
Y
l=n0
(Cs+n0+1−l)−1
n
Y
t=n0
Ct, ∀n≥n0. To prove that the expression above converges amounts to show that
T(n) :=
n
X
s=n0−1
(R(s))0
s+1
Y
l=n0
(Cs+n0+1−l)−1, n≥n0, converges. From the boundedness of (yn)n and condition (2.1) we have (2.21)
∞
X
n=1
|||Rn|||∞<∞.
Let u∈Nand n≥n0. We have
|||T(n+u)−T(n)|||∞=
=
R(n+1)
n+2
Y
l=n0
(Cn+1+n0+1−l)−1+· · ·+R(n+u)
n+u+1
Y
l=n0
(Cn+u+n0+1−l)−1 ∞≤
≤ |||R(n+ 1)|||∞
n+2
Y
l=n0
|||(Cl)−1|||∞+· · ·+|||R(n+u)|||∞
n+u+1
Y
l=n0
|||(Cl)−1|||∞
expression which, by Theorem 1.9, is smaller then
|||R(n+ 1)|||∞ n+2
Q
l=n0
[1− |||Bl|||∞]
+· · ·+ |||R(n+u)|||∞ n+u+1
Q
l=n0
[1− |||Bl|||∞]
≤
≤(|||R(n+ 1)|||∞+· · ·+|||R(n+u)|||∞) 1 cn0, where cn0 :=Q∞
l=n0
1− |||Bl|||∞
∈(0,1]. The inequalities above and (2.21) implies that (T(n))n is a Cauchy sequence. Therefore it is convergent. This implies that lim
n→∞yn,t,∀t= 1, p exists and, using also (2.10), (2.11) and (2.17), that lim
n→∞xn,k, k= 1,Pp
t=1rt exists. Sincei∈S and m ≥0 were arbitrarily chosen, it follows using also (2.19) that lim
n→∞Pm,n exists for allm≥0, i.e., the chain (Pn)n≥1 is convergent.
Example 2.5. Consider the chain (Pn)n≥1 given by
Pn=
1
2 −n1 12+ n+11 n(n+1)1 1−n+11 n+11 0
0 0 1
, ∀n≥2,
and P1 = I3. The chain (Pn)n≥1 is convergent because the conditions of Theorem 2.4 are fulfilled.
Acknowledgements. The author wishes to thank Dr. Udrea P˘aun for the interest he showed in this paper.
REFERENCES
[1] B. Gidas, Nonstationary Markov chains and convergence of the annealing algorithm.
J. Statist. Phys.39(1985), 73–131.
[2] J. Hajnal,Weak ergodicity in nonhomogeneous Markov chains.Proc. Cambridge Philos.
Soc.54(1958), 233–246.
[3] D.J. Hartfiel,Nonhomogeous Matrix Products.World Scientific, River Edge, 2002.
[4] R.A. Horn and C.R. Johnson,Matrix Analysis.Cambridge Univ. Press, New York, 1985.
[5] M. Iosifescu, Finite Markov Processes and Their Applications. Wiley, Chichester and Editura Tehnic˘a, Bucure¸sti, 1980; corrected republication by Dover, Mineola, N.Y., 2007.
[6] D.L. Isaacson and R.W. Madsen,Markov Chains:Theory and Applications. Wiley, New York, 1976.
[7] S. Karlin and H.M. Taylor,A First Course in Stochastic Processes.Academic Press, New York, 1975.
[8] P.J.M. van Laarhoven and E.H.L. Aarts,Simulated Annealing:Theory and Applications.
Reidel, Dordrecht, 1987.
[9] A. Mukherjea and R. Chaudhuri,Convergence of non-homogeneous stochastic chains, II.
Math. Proc. Cambridge Philos. Soc.90(1981), 167–182.
Received 16 March 2007 “Transilvania” University of Bra¸sov
Department of Mathematical Analysis and Probability Str. Iuliu Maniu 50
500157 Bra¸sov, Romania alinanicolae@unitbv.ro
