Y~= ~ P~Y~ Determine II when P is 9iven. THE CALCULATION OF THE ERGODIC PROJECTION FOR MARKOV CHAINS AND PROCESSES WITH A COUNTABLE INFINITY OF STATES

THE CALCULATION OF THE ERGODIC PROJECTION FOR MARKOV CHAINS AND PROCESSES WITH A COUNTABLE

INFINITY OF STATES

BY

DAVID G. K E N D A L L and G. E. H. R E U T E R Magdalen College Victoria University

Oxford Manchester

1. I n t r o d u c t i o n

t . 1 . L e t P ~ {p,~ :i, j = 0, 1, 2 . . . . } be the m a t r i x of one-step transition probabilities for a temporally homogeneous Markov chain with a countably infinite set of states (labelled as 0, 1, 2 . . . . ). The p r o b a b i l i t y p~j of a transition in n steps from state i to state j will then be the (i, j)th element of the m a t r i x pn, so t h a t the specification of P (or equivalently of A = P - I ) completely determines the system. I t is known (Kolmogorov [24]) t h a t the Cess limits

Jtij ~ lim 1 ~ P~s (1)

n - - ~ o o n r - 1

always exist. L e t II denote the m a t r i x whose (i, j)th element is 7 ~ , and consider P r o b l e m A : Determine II when P is 9iven.

This problem has obvious importance for practical applications. A n u m b e r of special techniques are available for its solution in particular cases (see, e.g., Feller [12], Ch. 15, Foster [14], [15] a n d Jensen [18]); also Feller [12], pp. 332-4, has given a general iterative m e t h o d of solution. We shall give another ( n o n - i t e r a t i v e ) g e n e r a l method in w 2: it will involve the non-negative solutions of

xj = Y. x~ p~j

r162

such t h a t Z x a < co, a n d the non-negative solutions of Y~= ~ P~Y~

such t h a t sup y~ < ~ .

104 D A V I D G . K E N D A L L A N D G . E . ]:I. R E U T E R

1.2. L e t P, ~ {p,j (t) : i, ] - 0, 1, 2 . . . . }, for each t >_ 0, denote the m a t r i x of transi- tion probabilities for a t e m p o r a l l y homogeneous Markov process with a countably infinite set of states, such t h a t

p~j(t)--~6ii

as t ~ 0. We m a y then define a one-para- m e t e r semigroup

{Pt:t->O}

of transition operators on the Banach space l b y setting

(Pt x)j -~ ~ x~ p,j (t)

(2)

Ct-0

for each element

x=(Xo, x~, x 2 .. . . )

of I. I n exceptionally simple cases we can write

Pt

= exp (~ t), where ~ is a bounded operator on l, but even for so simple an example as the b i r t h - a n d - d e a t h process (for this see, e.g., Feller [12], pp. 371-5) this is no longer possible. H o w e v e r a generating operator ~ (analogous to A in the chain case) can always be defined by writing

~) x ~ strong lim (Pt x -

x)/t

(3)

t 4 0

for x C ~ (~), where ~ (~) is b y definition t h a t set of elements x E l for which the limit in (3) exists. I n general ~ will be an unbounded operator, b u t its domain D (s is always dense in 1 and ~ determines the system uniquely (hence it is called the

in]initesimal generator).

I t is also known (L6vy [27]) t h a t the ordinary limits

~ j ~ lim p~j (t) (4)

t --~r162

exist (no Cess averaging being needed now); thus if we again write II for the m a t r i x whose (i, j)th element is z~j, we are led to consider

P r o b l e m B :

Determine II when ~ is given.

A solution to this problem will be given in w 3: it will involve the non-negative elements in the nullspaces of ~ and ~ * (the operator adjoint to ~ ) .

1.3. F o r a n y such Markov process, the limits

t~O

exist, and t h e y are finite except Always

(5)

perhaps when

i = j

(Doob [7], Kolmogorov [25]).

and in m a n y cases of practical interest

(a) all qtt are finite, and (b)

~ q t ~ = O

for each i.

T H E C A L C U L A T I O N O F T H E E R O O D I C P R O J E C T I O N F O R M A R K O V C H A I N S 1 0 5

Conversely, given a conservative q-matrix, i.e. a matrix Q - { q i j } with non-negative elements off the main diagonal and satisfying both (a) a n d (b), then there exists at least one process for which p~j(+ 0 ) = q~s, b u t in general this process ! is not unique (Feller I l l ] , Doob [8]). When the process is unique the conservative q-matrix Q will be called regular and the associated process will be called the Feller process determined by Q. That this is the normal situation in practical problems is the sole justifica- tion for the common practice of specifying a Markov process merely b y writing down a conservative q-matrix Q. Necessary and sufficient conditions for regularity have been given by Feller I l l ] and (in a somewhat different form) b y K a t o [20].

I t will now be clear t h a t for practical purposes the solution to Problem B will be insufficient, and t h a t one must also consider

P r o b l e m C: Determine [IF, the If-matrix associated with the Feller process, when a regular conservative q-matrix Q is given.

A solution to this problem will be given in w 4: it will involve the non-negative solutions of

x~ q~j - 0 such t h a t E x~ < ~ , and the non-negative solutions of

q~ ~ y~ = 0 such t h a t sup ya < ~ .

t.4. Throughout this paper we shall confine ourselves to " h o n e s t " chains and processes, i.e. systems satisfying ~ p,a = 1. However, a "dishonest" chain or process can always be imbedded in an honest one (obtained b y adjoining a single "absorbing"

state) and b y means of this device our methods can easily be adapted to the gen- eral case.

We have also confined ourselves strictly to the problem of calculating the H-matrix and it will be recalled t h a t one cannot decide from an examination of the zts for a Markov chain whether or not a given dissipative state is recurrent. B u t this limita- tion also is only apparent; it has often been remarked t h a t the question of recurrence can be settled b y calculating the H - m a t r i x for a modified chain in which the given state is made absorbing.

1.5. I n some respects this paper is a sequel to our paper [22] in which we considered the ergodie properties of one-parameter semigroups of operators on an ab-

106 DAVID G. K E ~ D A L L AND G. E. H. REUTER

stract Banach space, and readers of the earlier p a p e r will find a discussion of the present problem from t h e standpoint of the general t h e o r y in w 6. Others who are mainly interested in probabilistic applications m a y prefer not to read beyond w 5, in which some examples are given to illustrate our methods. We h a v e deliberately chosen the simplest examples which would serve this purpose, b u t we believe t h a t the methods of this p a p e r can usefully be applied to some more complicated systems which arise in practice. I n particular, we intend to t r e a t elsewhere the ergodic pro- perties of two Markov processes which describe (i) the competition between two species, and (ii) the development of a stochastic epidemic.

2. M a r k o v chains: the solution to Problem A

2 . t . We first recall some results with which the reader will doubtless be familiar (perhaps in a different terminology). 1 The limits re, j defined at (1) always exist a n d will clearly satisfy

z~j>-0, ~ g ~ _ < 1.

The structure of the m a t r i x H -- { ~ j : i, j >_ 0} is m o s t conveniently described b y classi- fying the state j as

positive

when gss > 0 and as

dissipative

when r~jj = 0. T h e collec- tion of positive states (if there are any) is t h e n further divided into disjoint

positive classes,

the positive states ?" and ]c being in the same class if and only if gjk > 0.

(This relation can be shown to be reflexive, s y m m e t r i c a n d transitive.) I f we write gJ-=~sJ for each positive state ], t h e n the g,j can be expressed in terms of the gj a n d a set of numbers ~ (i, C), defined for each state i a n d each positive class C, where 0 ~ ( / , C ) _ < I . I n fact

(i) ~ j = 0 for all i, if ?" is dissipative.

(ii)

~ , s = ~ ( i , C)rej

for all i, if ~" belongs to the positive class C.

Also ~j a n d ~ (i, C) h a v e the following properties:

(iii) ~ ~j = 1 for each positive class C.

(iv) I f / is a positive state and C a positive class, ' ~ (i, C) = if i ~ C .

' F o r e x p o s i t o r y a c c o u n t s of t h e t h e o r y , see C~UNG [4], FELLER ([12], Ch. 15), LO~.VE ([28], p p . 28-42).

T H E C A L C U L A T I O N O F T H E E R G O D I C P R O J E C T I O N F O R M A R K O ~ r C H A I N S 107

(v)

I f i is dissipative a n d {C e : ~ = 1, 2 . . . . } are the positive classes, t h e n

$ (i, C Q) _< 1.

(vi) I f C is a positive class, then

{~i if j E C , c ~ p~j= 0 if j ~ C, and ~ p~ = ~ (~, C) = ~ (i, C) for all i.

(vii) F o r all i a n d j,

(6) (7)

(8)

( I n (7) and (8) summations are over r162 1, 2 . . . . ; we shall adhere to this conven- tion from now on.)

The classification of states a n d the description of ntj have been given above in purely analytical terms: however, t h e y have probabilistic meanings which (although t h e y are n o t needed for w h a t follows) the reader m a y usefully keep in mind. Positive states are precisely those which are recurrent with finite m e a n recurrence time #j, given b y /zj = 1/7~j; dissipative states are either recurrent with infinite m e a n recurrence time, or non-recurrent (transient); finally ~ (i, C) is the probability t h a t the system, starting at state i, will ultimately enter (and thereafter remain in) the positive class C.

These interpretations depend on a detailed a n d deep analysis of the a s y m p t o t i c pro- perties of p~- as n - - > ~ , first m a d e b y K o l m o g o r o v [24]; the properties of I] which we have s t a t e d lie less deep and have been p r o v e d more simply b y u & K a k u - t a n i [32] and Doob [7].

2.2. The calculation of I] when P is given involves two steps:

(a) the classification of states a n d determination of the reciprocal m e a n recur- rence times ~j,

(b) the calculation of the absorption probabilities ~ (i, C).

Step (b) will of course be superfluous (by (i) of 2.1) when there are no positive states, a n d will be trivial (by (iv)) when i is a positive state.

To perform step (a) we introduce the Banach space 1 whose elements are real sequences x = (x0, xl, x~ . . . . ) such t h a t

1 0 8 D A V I D G. K E N D A L L A N D G . E . H . R E U T E R

T h e n P a n d [I d e t e r m i n e b o u n d e d l i n e a r o p e r a t o r s on l as follows:

W e h a v e

Px>_O, H x > _ O , a n d also (because of (8))

II nxll IIxll=lIPxl[,

I I ~ = P II = II P = II.

w h e n e v e r x >_ 0,

(9) F o r e a c h p o s i t i v e class C ~ we d e f i n e n ~ E l b y

{~j if ~ E C ~ (~o)j= 0 if ~'~C~

t h u s he>_0, ] [ : ~ [ [ = 1 ( b y (iii) of 2.1), n ~ h a s C ~ as i t s s u p p o r t / a n d f r o m ( 6 ) w e see t h a t P : r ~ ~ so t h a t :z ~ b e l o n g s t o t h e n u l l s p a c e 7 / ( A ) of t h e o p e r a t o r A - ~ P - I . T h e f o l l o w i n g t h e o r e m is well k n o w n , * b u t we s t a t e i t in a f o r m w h i c h differs f r o m t h e u s u a l one a n d t h e r e f o r e s k e t c h i t s p r o o f .

T H E O R E M 1. A n l.vector x lies in ~ ( A ) i/ and only i] there exist real numbers { ~ e : e = l , 2 . . . . } such that 5 ] ) t e [ < o o and x = ~ , ~ e r d . Also x ~ O i/ and only i/ ~>_O

Q Q

/or each e"

Proo/. F r o m t h e f a c t t h a t A : r ~ 0 i t e a s i l y follows t h a t A x = 0 w h e n e v e r x is of t h e s t a t e d f o r m . O n t h e o t h e r h a n d , if x E 1 a n d A x = 0 , t h e n

a n d so on l e t t i n g n--> oo we o b t a i n

Z x~ ~aj = Xj.

o~

F r o m (i) of 2.1 i t n o w follows t h a t x s = O w h e n ] is d i s s i p a t i v e , so t h a t

Q ~ e C 0

w h e r e t h e f i r s t s u m m a t i o n is o v e r a l l p o s i t i v e classes. T h e f i r s t a s s e r t i o n of t h e t h e o r e m n o w follows o n s e t t i n g

~ e C 0

t h e s e c o n d a s s e r t i o n follows f r o m t h e f a c t t h a t t h e v e c t o r s n e a r e p o s i t i v e a n d h a v e d i s j o i n t s u p p o r t s .

1 The support of an /-vector x is the set of states j such that x I * O.

z See, e.g., CHu~IG ([4], Th. 9), or Lo~vE ([28], p. 41).

T H E C A L C U L A T I O N OF T H E E R G O D I C P R O J E C T I O ~ F O R M A R K O V C H A I N S 109 W e can deduce t w o i m p o r t a n t corollaries on using t h e fact t h a t for each ~, t h e s u p p o r t of n ~ is precisely t h e corresponding positive class C ~ L e t us d e n o t e b y ~ + (A) t h e set of positive vectors in ~ ( A ) . T h e n we h a v e

C o R o L L X R u 1.1. A state j is positive i / a n d only i/ it lies in the support o / a t least one 1-vector x E ~+ (A ). T w o positive states ~ and ]c lie in di//erent positive classes i/ and only i/ there exists an x C ~+ (A) whose support contains ] but not k.

C O R O L L A R Y 1.2. Let ~ be a positive state. T h e n amongst the elements x E ~ + (A) such that x j = l there is a least, x s, and xi/iixJiI=Jz q where C a is the positive class con.

raining ].

These corollaries show t h a t step (a) can be p e r f o r m e d when the nullspace ~ (A) has been [ound.

2.3. W e n o w have to c a r r y o u t step (b) in so far as it is non-trivial, so t h a t we m u s t give a m e t h o d for calculating ~ (k, C) w h e n k is a dissipative state a n d C is a positive class. This will involve t h e nullspace ~ ( A * ) o f t h e o p e r a t o r A* ad- joint to A.

L e t m be t h e B a n a c h space of real sequences y ~ (Y0, YI, Y2 .. . . ) such t h a t

Ilyll---sup ly l<

T h e n m is t h e a d j o i n t space to l, a n d we shall write (y, x) ~ ~ y~ x~ w h e n y E m a n d x E1.

T h e o p e r a t o r P* a d j o i n t t o P is given b y

(y P*)~ - ~ p ~ y:, (y E m), a n d ( y P * , x) = (y, P x )

w q E m b y

w h e n y E m a n d x E l . F o r each positive class C e, we define

(~Q)~-~(!,C~) (i=0, 1,2

. . . . ).

Properties (iii) a n d (iv) of 2.1 show t h a t

0>0, I1 011=1,

(Wq, ~o) = 1,

a n d (wQ)~=O if i E C ~ ' ( a : v ~ ) . F r o m (7) we h a v e ~ Q P * = w Q , so t h a t ~ q E ~ + ( A * ) , t h e positive p a r t of t h e nullspaee ~ ( A * ) of t h e o p e r a t o r A * - - P * - / . T h e nullspaee of A* is often m u c h bigger t h a n this result suggests, 1 b u t f o r t u n a t e l y we can still

1 The structuro of ~ (A*) has been studied by BLACKWELL [2] and FELLER [13].

1 1 0 D A V I D G . K E N D A L L A N n G . E . H . R E U T E R

e h a r a c t e r i s e R e b y a m i n i m a l p r o p e r t y s i m i l a r t o t h a t i n v o l v e d in C o r o l l a r y 1.2. W e f o r m u l a t e t h i s a s

T H E O R E M 2. Let C e be a positive class and let ne be the associated 1-vector. Then w e is the least element y o/ ~+ (A*) such that (y, ~ ) = 1.

Proo/. W e a l r e a d y k n o w t h a t y - - - R e satisfies t h e t h r e e c o n d i t i o n s y _> 0, y A* = 0, (y, ~e) = 1.

L e t y be a n y m - v e c t o r d i s t i n c t f r o m R e a n d s a t i s f y i n g t h e s e c o n d i t i o n s . T h e n

n

f r o m w h i c h i t follows (on using t h e p o s i t i v i t y of y) t h a t

~ y~ _< y~. (10)

W e can w r i t e (10) in t h e f o r m

~ ( i , C ~) ~ zr~y~<y~,

a ~ E C (~

w h e r e t h e a - s u m m a t i o n is o v e r all p o s i t i v e classes, a n d t h i s is e q u i v a l e n t t o

~ (i, C ") (y, ~") § ~ (i, C ~ _< y~, f r o m w h i c h i t follows t h a t R e < y a s r e q u i r e d .

W e o u g h t t o m e n t i o n h e r e t h a t F e l l e r ([12], p. 332, (8.2) a n d (8.3)) h a s g i v e n a set of r e c u r r e n c e r e l a t i o n s w h i c h u n i q u e l y d e t e r m i n e t h e a b s o r p t i o n p r o b a b i l i t i e s (i, C). T h i s r e c u r r e n t p r o c e d u r e m a y , h o w e v e r , b e as d i f f i c u l t t o c a r r y o u t as t h e c a l c u l a t i o n of t h e ~ij d i r e c t l y f r o m t h e i r d e f i n i t i o n (1). As F e l l e r r e m a r k s , one h a s t h e n t o r e s o r t t o his e q u a t i o n (8.4) a n d t h e s o l u t i o n t o t h i s e q u a t i o n is in g e n e r a l n o t u n i q u e . T h e n o n - t r i v i a l , p a r t of o u r T h e o r e m 2 singles o u t t h e r e l e v a n t solution.

2.4. W e c a n n o w s t a t e o u r solution to Problem A as

T H E O R E M 3. I / the positive classes (Ce : ~ = 1, 2 . . . . } and the associated l-vectors 7e Q and m-vectors ~ are determined by Corollaries 1.1 a n d 1.2 and Theorem 2 above, then the matrix-elements o/ the "ergodic projection-operator" II will be given by

7r~j=O (all i)

THE CALCULATION OF THE ERGODIC FROJECTION FOR MARKOV CHAINS

when ] is not a positive state, and by

~ij = (~e)~ (~e)j (all i) when ] lies in the positive class C Q.

111

3. Markov processes: the solution to Problem B

I f {p~j (t) : i, j = 0, 1, 2 . . . . ; t _> 0} is the a r r a y of transition probabilities for

3 . ~ [ ,

a Markov process, a n d if (as we shall always assume) the continuity condition p ~ j ( t ) - ~ j as t ~ 0

holds, t~hen the limits xe~j-lim p~j(t) (11)

t - - > ~

exist. This result is due to L 6 v y [27], a n d can be proved b y considering the chain defined b y setting p~j ~ p~j (n~). This chain is aperiodic because p~ (n r) > 0, and there- fore lira pit(n~) exists for each fixed ~ > 0 . F r o m the uniform continuity of Pij(" )

n - - ~

for fixed i and j, it follows t h a t this limit is independent of v and t h e n t h a t the limit a t (11) exists.

Clearly we can calculate 7rij b y applying the procedure of w 2 to the chain whose m a t r i x of one-step transition probabilities is P~-= {p~j(~):i, ?'=0, 1, 2 . . . . ), for a n y one

> 0, because

~ j = lim p~j (n~) = lim (P~)ij.

n~-~oo n-->oo

To do this, we m u s t consider the nullspaees }/(A~) and 7/(A*), where A~--P~--I, (P~x)j----~x~p~j(~) (x61).

(x

The dependence of this procedure on the choice of v is, of course, only apparent, a n d this fact is exhibited m o s t conveniently b y introducing the one-parameter semigroup {Pt:t>-O) of operators on l associated with the process. This has the properties: 1

(a) P 0 = I , P~P~=Pu+v (u_>0, v>_0).

(b) Ptx>_O and

IIP ll=[l lt

whenever O_<xCl.

(c) as t r for each

I t s infinitesimal generator s is d e f i n e d b y

s - strong lim (Pt x - x ) / t

t~0

~ ^{. o}

1 F o r t h e special p r o p e r t i e s of s u c h t r a n m t l o n s e m i g r o u p s " on l, see KI~NDALL a n d REUTER [23]

or KENDALL [21]; for t h e general t h e o r y of s e m i g r o u p s of o p e r a t o r s , see HILLE [17].

112 D A V I D G . K E N D A L L A N D G . E . H . R E U T E R

whenever this limit exists, the domain ~ (s of ~/ is dense in

l,

and

t t

P t x = x + ] P u ~ l x d u = x + f a P u x d u

(12)

0 0

for x 6 ~(~-/) and t_>0. The adjoint operator ~/* (on the adjoint space m of l) is defined by setting y ~ / * - z , where

(z, x) = (y, ~ x)

for all x 6 ~ (g/), whenever such an element z (necessarily unique) exists.

We now have the crucial

LEMMA. 7/ ( ~ ) = 7 / (A~)

/or each

3 > 0 ; (13)

7/(g/*) = N 7/(A*). (14)

1~>0

Proo/.

If x 6 7 / ( ~ / ) , t h e n (12) at once gives

P ~ x = x ,

so t h a t x67/(A~). Con- versely if x 6 7 / ( A ~ ) for

one

3 > 0 , t h e n

P ~ x = x

and hence

H x = x

(as i n t h e p r o o f of Theorem 1). On using (9), we obtain t h a t

P t x = P t I I x = I I x = x

for

all t>O,

and so finally t h a t

glx=O.

This proves (13).

Next, if y 6 7 / ( g / * ) , t h e n (12) gives

T

(y, P~ x) = (y, x) + f (y, g / P u x) d u (all x 6 ~ (~/))

0 T

= (y, x) + f (y gl*, P~ x) d u = (y, x).

0

p ~

Hence (y 3, x ) = (y, x) for all x 6 O(g/), and because D(~/) is dense this implies t h a t

p* p *

y ~ = y , i.e. t h a t y 6 7 / ( A * ) . Conversely if y , = y for

all

3 > 0 , then

( y , P ~ x ) =

(y, x) and

( y , ~ x ) = l i m y, = 0 for all x E ~ ( f ~ ) ,

31,0

so t h a t y 67/(f2*). This proves (14).

3.2. We now classify the states and describe t h e structure of IF[. L e t us say t h a t the state j is

positive

if ~jj > 0,

dissipative

if zjj = 0, and t h a t two positive states j and k are in the same

positive class

if ~jk > 0 (this is an equivalence relation between positive states). Clearly this classification coincides with t h a t of w 2 for each of the chains P~ derived from the process. Using (13), we can at once transcribe Corol- laries 1.1 and 1.2 in t e r m s of 7/+(g/), the positive p a r t of 7/(~/), and obtain

T H E C A L C U L A T I O I ~ O F T H E E R G O D I C P R O J E C T I O i ~ F O R M A R K O u C H A I I ~ S 113 T HEOREVt 4. A state is positive i/ and only i/ it lies in the support o/ some x E 7</+ (f2); two positive states lie in di[/erent positive classes i/ and only i/ there exists an x E ~+ (f~) whose support contains one o/ the states but not the other.

I / j is a positive state then amongst elements x E ~+ ( ~ ) with xj = 1 there is a least, x j, and i/ ~q-xS/HxJll then 7t ~ depends only on the class C e containing j, and has C ~ as its support.

To find the analogue of Theorem 2, we observe t h a t the same /-vector ~ and m-vector ~ e are associated with a positive class C ~ for each chain P~, and in each case w o is the least element y of 71 + (A*) such t h a t (y, ~ ) = 1. Hence ~ is also the least element y in

n u+ (h:)

v > 0

or (by (14)) in )l + (f~*), such t h a t (y, ~ ) = 1. Thus we have

T ~ E O R E M 5. I [ C ~ is a positive class and 7t ~ is the associated l-vector o/ Theo- rem 4, then amongst the elements y o/ ~+ (~*) such that (y, 7t ~) = 1 there is a least, ~ .

B y combining Theorems 4 and 5 we obtain as a solution to Problem B :

T ~ E O R ~ M 6. I / the positive classes C Q and the corresponding l-vectors ,~ and m-vectors w e have been determined as in Theorems 4 and 5, then the matrix-elements o/

the ergodic projection-operator II will be given by

~ij = 0 (all i) when ~ is not a positive state, and by

~ i = (we)~ (zte)j (all i) when j lies in the positive class C q.

I t should be pointed out t h a t whilst we have proved our results for processes b y considering the chains P~ and using the results of w 2, this is only a matter of convenience. Direct proofs can be given, b y methods similar to those of w 2, based on the properties (analogous to (8))

(for each t >_0), (15)

which where proved b y Doob ([7], Theorems 6 and 7).

8 - - 5 7 3 8 0 4 . Acta mathematica. 97. I m p r i r n 6 le 12 a v r l l 1957.

114 D A V I D G . K E N D A L L A N D G . ~ . H . R E U T E R

4. Feller processes: the solution to ProMem C

4 . t . Suppose t h a t a m a t r i x Q of finite real elements q*t is given, such t h a t q~j>_0 when i - j , Z q i ~ - - 0 for all i.

Suppose f u r t h e r t h a t the conservative m a t r i x Q is

regular;

i.e. t h a t the " m i n i m a l "

process constructed b y Feller [11] is

honest

(satisfies ~ p ~ ( t ) = 1), or equivalently t h a t there is

exactly one

Markov process such t h a t p[j ( + 0) = q~j. (See Doob [8], R e u t e r [31]).

We write x e D (Q) whenever (a) x El,

(b)

~x~q:~i

is absolutely convergent for each ?',

(c) ZIZx q ;l<

1 a

and we define an operator Q with domain ~ (Q) b y setting

(Qx)j~Zx~q~,j (xEO(Q)).

(16)

q.

The set O 0 of " f i n i t e " vectors (those with only finitely m a n y non-zero components) is contained in D(Q), and we define Q0 to be the restriction of Q to O ( Q 0 ) ~ ) 0 . Because O0 is dense in 1 we can define the adjoint Q~ of Q0, a n d this can be shown to be given b y

(yQ~),-~q,~ya (yeD(Q~))

(17)

O~

the domain D(Q~) of Q~ consisting of those vectors y E m for which (17) defines an element

y Q~

of m.

N o w let

~F

generate the Feller semigroup associated with the given q's. Then 1

Q0 - ~ r - ~ Q , (18)

and hence Q * ~ ~ ~ Qo * *; (19)

f u r t h e r for each ~ > 0 and x E l, the equation

2 ~ - ~ ~ = z (20)

has exactly one solution ~ ( I ) ~ x in ~ ( ~ ) . When x_>0, t h e n (I)~x_>0 and ~ ( I ) ~ x is the least positive solution (in 0 (Q)) of the equation

2 8 - Q S = x .

(21)

1 The properties of ~F which we state here can be proved without undue difficulty by exam-

~6 . . )

ining FELLER'S construction [11] of his rmmmal ' semigroup; see [31].

T H E C A L C U L A T I O : N O F T H E E R G O D I C P R O J E C T I O : N F O R M A R K O V C H A I N S 115 W e n o w m a k e use of our a s s u m p t i o n t h a t Q is regular, i.e. t h a t g2r generates a

transition

semigroup. This implies t h a t 2(I)~ is a t r a n s i t i o n operator, a n d 1 t h a t t h e nullspace

T I ( 2 I - Q ~ )

contains o n l y t h e zero vector, for each 2 > 0 . Using these facts, we are able t o eliminate t h e explicit references to ~ in T h e o r e m s 4 a n d 5, a n d re- place t h e m b y s t a t e m e n t s involving t h e q's alone. To do this, we prove

T H E O R E M 7.

tion semigroup, then

I / Q is regular, and ~ generates the (unique) associated transi-

Tl + ( ~ ) = Tl + (Q), (22)

~ + ( ~ ) = ~ + (Q~). (23)

Proo[.

Clearly (18) implies t h a t ~ + (~p)___~+ (Q), a n d (22) will follow if we can prove t h e reverse inclusion. Suppose t h e n t h a t x ~> 0, x E ~ (Q), a n d Q x = 0, a n d choose some fixed 2 > 0 . W e shall t h e n h a v e 2 x - Q x = 2 x ~ 0 , so t h a t ~ = x is a positive solution of t h e e q u a t i o n

2 ~ - Q ~ = 2 x > _ O

(~ e ~ ( Q ) ) . (24) B u t (cf. (21)) the least positive solution of (24) is ~ = ( P ~ ( 2 x ) , so t h a t x>_(I)~(2x)=

2 (I)z x ; also we c a n n o t h a v e x ~= 2 (I)~ x because this would n o w give [[ x [[ > [I 2 (I)x x [I, con- t r a d i c t i n g t h e fact t h a t 2 (I)~ is a t r a n s i t i o n operator. I t follows t h a t x = (I)~ (2x)E D ( ~ r ) , so t h a t x E T/+ ( ~ r ) ; this proves t h a t ~ + ( Q ) ~ + (~F), a n d (22) follows.

W e shall d e d u c e (23) f r o m t h e sharper s t a t e m e n t t h a t

$ $

~ F = Q0, (25)

a n d to p r o v e this it will suffice, b y (19),. to prove t h a t

O (D~) = ~ (Q~). (26)

W e shall need here one f u r t h e r fact f r o m semigroup t h e o r y , n a m e l y t h a t w h e n 2 > 0 a n d x E D (~F), t h e n

(I)~ (21 -- ~F) x = x.

F r o m this it follows at once t h a t

a n d t h a t w O~ ( 2 1 - ~ ) = w

for

all w Em.

1 The condition, ~ (Z I - Q~) = {0}, is given in KATe'S paper [20]. The condition can be proved to be equivalent to the regularity of Q; see [31].

1 1 6 D A V I D G. K E N D A L L A N D G. E . t I . R E U T E R

N o w if (26) were false, we could find y u= 0 such t h a t y E ~) (Q~) b u t y ~ ~ ( ~ ) . F i x 2 > 0, a n d define

z = - y ( , ~ I - Q ~ ) C p ~ . T h e n z E ~ (g2~), so t h a t z ~= y, b u t also

z ( 2 I - Q~)) = z (21 - g]~) = y ( 2 I - Q~) O~ (21 - F~*)

= y ( 2 I - Q ~ ) ,

so t h a t ( z - y ) would be a non-zero element of ~ ( ~ I - Q ~ ) , c o n t r a d i c t i n g t h e r e g u l a r i t y of Q. T h u s (26) m u s t hold, a n d (23) follows. (For (26), see also [31], w 7.2.)

4.2. To o b t a i n a solution to Problem C, w e h a v e m e r e l y to combine T h e o r e m 7 w i t h T h e o r e m s 4 to 6. This leads us to

T H E O R E M 8. Let Q=- (q~j : i, j = 0 , 1, 2 . . . . } be a regular conservative q-matrix, and let (p~j (t) : t >_ O} be the unique process such that p[j ( + O) = q~j. Then:

(i) A state is positive i/ and only i/ it lies in the support of some x E ~ + ( Q ) ; two positive states lie in di//erent positive classes i / a n d only i/ there exists an x E Tl + (Q) whose support contains one o/ the states but not the other.

(ii) I / ~ is a positive state, then the set o/ l-vectors x E ~+ (Q) with x j = 1 has a least member x j, and i/ no`-xJ/[[xJH then ~o` depends only on the positive class C ~ containing ~ and has C ~ as its support.

(iii) I] Co` is a positive class, then the set o/ m-vectors y E ~+ (Q~) with (y, ~zo`)= 1 has a least member v90`.

(iv) The limits z~j = lim p~j (t)

are given by z ~ j = 0 (all i) when j is not a positive state, and by

xe~j ⁼ (wq)~ (7~o`)j (all i) when ] lies in the positive class Co`.

I n order t o a p p l y T h e o r e m 8, we h a v e o n l y to use t h e facts t h a t ~ + (Q) consists of all positive vectors x E l such t h a t

~x~q~j=o

( ] = 0 , 1 , 2 . . . . ), (27)

THE CALCULATIOI~ OF THE ERGODIC P R O J E C T I O 5 FOR MARKOV CHAINS 117

a n d ~/+ (Q~) consists of all p o s i t i v e v e c t o r s y E m s u c h t h a t

q ~ y ~ = O ( i = O , 1, 2 . . . . ). (28)

c ~ = O

T h e r e l e v a n c e t o t h e ergodic p r o b l e m of t h e p o s i t i v e c o n v e r g e n t s o l u t i o n s t o (27) a n d of t h e p o s i t i v e b o u n d e d s o l u t i o n s t o (28) has, of course, l o n g b e e n r e c o g n i s e d b y s t a t i s t i c i a n s ; t h e i m p o r t a n c e of T h e o r e m 8 is t h a t i t allows t h e a p p r o p r i a t e s o l u t i o n s t o be i d e n t i f i e d in cases of n o n - u n i q u e n e s s .

5. E x a m p l e s

5.1. T h e r a n d o m w a l k . W e b e g i n w i t h a n e x a m p l e i l l u s t r a t i n g t h e s o l u t i o n t o P r o b l e m A ; i t is a f a m i l i a r one a n d h e r e t h e ergodic b e h a v i o u r is well u n d e r s t o o d (ef. F o s t e r [14], H a r r i s [16], J e n s e n [18], K a r l i n & M c G r e g o r [19]). W e l a b e l t h e s t a t e s of a M a r k o v c h a i n as . . . . - 1 , 0, 1 . . . . a n d t h e n p u t

[

P:I if ? ' = i + l , if i = i - 1 ,

if j : ~ i •

w h e r e Pt > 0, q~ > O a n d p~ + q~ = 1, for all i. To f i n d t h e n u l l s p a c e of A we m u s t solve t h e d i f f e r e n c e e q u a t i o n s

x] = P ] - I xj 1 -~ q]+l X]+l (all j), a n d we w r i t e t h e s e as

u j - p j x j - p j _ l x j _ l = q j + l Xj + l - q j x j .

E v i d e n t l y 1 u E l if x E l, so t h a t we m u s t h a v e

~} Xj = Uj -~ U]_l -~ Uj_2 + . . . , qj xj = - (uj + uj+l + uj+2 + "" "),

(29)

a n d we t h e r e f o r e p u t

J

- r - o c J

so t h a t (29) b e c o m e s p~ x j = v j , qs x j = v j - l - a .

B e c a u s e b o t h x a n d u a r e in l, we k n o w t h a t x.,. a n d vj t e n d t o zero w h e n j - - > - ~ ; t h u s a = O. T h e l a s t p a i r of e q u a t i o n s n o w gives v j = (Ps/qJ)vj-1, a n d h e n c e

' A typical /-vector now takes the form x - ( . . . . x , , %, x, . . . . ), with I I x l l -

- o o

118 D A V I D G . K E N D A L L A N D G . E . H . R E U T E R

L e t us set

/PIP2 ... ~ v

~ qx q2 ... qJ o (J >0), v j =

|qJ+~qJ+~ "-': qJ v o (j<O).

( PJ+I Pj+2 ... P0

- 1 1 qj+l ... q o q _ l + ~, 1 - - ~ , V l " ' " ~i9"/

(30)

- - - .

R - = _ ~ P j P j + I P0 Po 1 P j q 1 . - - q j

I f R = o~, t h e n Vo=0 , t h e nullspace of A contains t h e zero v e c t o r only, a n d e v e r y state is dissipative. I f R < o~, t h e n t h e nullspaee of A is s p a n n e d b y a single posi- tive vector, e v e r y state is positive, a n d t h e states f o r m a single positive class. I n b o t h cases, we shall h a v e

i1 qJ+ ...qo 1

Pj p j + l . . . P 0 R if j < 0 ,

:z,j= ] 1 / ( p o R ) if j = 0 , (31)

] l p l " ' ' p ' I if j > O , [ PJ ql ... qj R

for all i (these expressions all being zero w h e n R = o~).

T h e nullspace of A* can readily be f o u n d for t h e present example, b u t (cf. 2.2) this is never required when, as here, ~ (A) is either zero-dimensional or is one-dimen- sional with a strictly positive g e n e r a t i n g vector.

5.2. The "/lash o/ /lashes". W e n e x t give t w o examples illustrating t h e solution t o problem B, a n d we h a v e chosen for this p u r p o s e t h e t w o m o s t pathological pro- cesses which we know. T h e first of these has a c o n s e r v a t i v e q-matrix which is so h i g h l y non-regular as t o be associated with a c o n t i n u u m of processes all satisfying t h e same set of " b a c k w a r d " a n d " f o r w a r d " differential equations. P r o b l e m C would here be quite meaningless a n d t h e process will be specified b y giving its infinitesimal g e n e r a t o r ~ .

T h e space 1 will n o w be so labelled t h a t a t y p i c a l l-vector becomes x=--(..., x -1, x ~ x 1, ...)

where xS---( .... xS-1, x0,S xl,S ...)

a n d t h e x s n are real n u m b e r s such t h a t

- o o - o o

T~E CALCULATIO~r OF THE ]~RGODIC rROJECTION FOR M ~ K O V C~AINS 119 N o w l e t a s (s, n = . . . , - 1 , 0 , 1 . . . . ) b e p o s i t i v e r e a l n u m b e r s such t h a t ~ I / a S < ~ , a n d d e f i n e t h e c o n s e r v a t i v e q - m a t r i x Q b y

rs [ - as~ if n = m ,

q m n = O ( r : ~ s ) ; qmnS~ = i +ares if n = m + l , 0 o t h e r w i s e .

T h e o p e r a t o r ~ is t h e n d e f i n e d t o be t h e r e s t r i c t i o n of t h e m a t r i x - o p e r a t o r Q ( d e f i n e d as a t (16)) t o t h e d o m a i n of / - v e c t o r s x such t h a t

(i)

Y Y l a g - l x g - , - a g x g l < ~ ,

(ii) U ~ x = L S + Z x ( s = . . . , - 1 , 0 , 1 . . . . ), (iii) l i m U " x = l i m U ' x ,

o--~-t- oo a - - ~ , - o o

w h e r e U s x = l i m a~ x ~ n ~ L ~ x-= lim a~ x~.

( I t is s h o w n in K e n d a l l [21] t h a t s g e n e r a t e s a t r a n s i t i o n s e m i g r o u p . ) W e m u s t n o w f i n d t h e n u l l s p a c e of ~ . I f x fi 7/(C2), t h e n

s x s a s x s

a n - 1 n - 1 - - n n ~ 0

for a l l s a n d n, a n d so

x s - cS/a ~ r t - - / rt , w h e r e c s = U s x = L ~ x.

C o n d i t i o n (ii) t h e n s h o w s t h a t c s is i n d e p e n d e n t of s, a n d t h e o t h e r c o n d i t i o n s a r e n o w s a t i s f i e d a u t o m a t i c a l l y . T h u s ~ ( ~ ) is o n e - d i m e n s i o n a l a n d is s p a n n e d b y t h e v e c t o r w h o s e (~)th c o m p o n e n t is 1 / a ~ . I t follows t h a t all t h e s t a t e s a r c p o s i t i v e , t h a t t h e y f o r m a single p o s i t i v e class, a n d t h a t

~ j = ~ , = ~ - (32)

a~

for all v a l u e s of r, s, m a n d n. A s in 5.1 t h e r e is no n e e d to c a l c u l a t e T/(~*).

5.3. A sequence o] ]lashes c o m m u n i c a t i n g v i a a n i n s t a n t a n e o u s state. I n t h i s e x a m p l e (also t a k e n f r o m K e n d a l l [21]) t h e M a r k o v p r o c e s s h a s a h i g h l y n o n - c o n - s e r v a t i v e q - m a t r i x ; one of s t a t e s , l a b e l l e d 0, is " i n s t a n t a n e o u s " (has % 0 = - c ~ ) a n d t h e c o r r e s p o n d i n g r o w of t h e q - m a t r i x c a n be w r i t t e n a s

( - o ~ , 0 , 0 , 0 . . . . ).

As in 5.2 t h e process will be specified b y g i v i n g i t s i n f i n i t e s i m a l g e n e r a t o r ~ . T h i s t i m e 1 is so l a b e l l e d t h a t a t y p i c a l l - v e c t o r b e c o m e s

120 D A V I D O . K E N D A L L A N D G . E . H . R E U T E R z - - ( x ~ x 1, x ~ . . . . )

where x ~ is a real n u m b e r a n d x s is a real v e c t o r

X S ~ ( . . . . X__I,S X ~ , X ~ . . . . )*

T h e n o r m is defined b y

I1 11 1 ~ §

(*=1 c~- - oo

5: Izgl< .

N o w choose positive n u m b e r s a~ ( s = l , 2, 3 . . . . ; n = . . . , - 1 , 0, 1 . . . . ) such t h a t

~ . ~ 1 / a ~ < c<, a n d define ~ ( ~ ) t o be t h e set of vectors x which satisfy (i) E E l a a - l a Xa:_a I - -

a~,xr

* = x ~ (all s ~ > l )

a n d (ii) L* x =- lira a= xn

n---> - oo

(such vectors necessarily belong to l). F i n a l l y define ~Qx f o r x e D ( ~ ) b y

oo

(f~ x ) ~ - E ( U " x - x ~

s s s s ( s > l , all n), ( ~ x ) ~ ---an-1 X n - l - - a n z~

where U s x ~- lim a~ x~

~/.--~ -I- OO

(this last limit necessarily exists a n d t h e series e q u a t e d t o (f2x) ~ is necessarily ab- solutely convergent). E v i d e n t l y s 0 if a n d o n l y i f a s xS~ = c ~ x ~ a n d so ~ / ( ~ ) is one-dimensional a n d is s p a n n e d b y t h e v e c t o r whose c o m p o n e n t s are

x ~ = 1, xn-* - 1 / a ~ .

T h u s all t h e states are positive a n d f o r m a single positive class, a n d re ~176 = A , 7~.~~ = A / a ~ ,

YO r s __

J

: 7 " ~ m . = A , ~mn - A / a S ~ , ( 3 3 )

where 1 / A ~ 1 § ~ ~ ]//a~. Once again there is no n e e d to find ~/(~2").

5.4. A g e n e r a l M a r k o v i a n q u e u i n g p r o c e s s . W e n o w t u r n to P r o b l e m C: t h e cal- culation of t h e ergodic projection for Feller processes. As a first example we consider t h e M a r k o v process h a v i n g t h e q-matrix

- b o b o 0 0 ...

a 1 - ( a l + b z ) b 1 0 . . .

0 a 2 - ( a ~ § b 2 . . .

0 0 a 3 - ( a a + b 3) . . .

. ~ ~ . . . . . . . . . . 9

(34)

T H E C A L C U L A T I O N O F T H E E R G O D I C P R O J E C T I O N F O R M A R K O V C H A I N S 121 w h e r e b0, a 1 bl, a 2 . . . . a r e g i v e n positive r e a l n u m b e r s . 1 I n t h e classical q u e u i n g process of E r l a n g we h a v e a 1 = % . . . a a n d b 0 = b l = b 2 . . . b; t h e s t a t e - l a b e l r is e q u a l t o t h e n u m b e r of p e r s o n s w a i t i n g o r b e i n g s e r v e d a n d so a can be i d e n t i f i e d as t h e r e c i p r o c a l of t h e m e a n service t i m e a n d b a s t h e r e c i p r o c a l of t h e m e a n t i m e b e t w e e n a r r i v a l s , e a c h of t h e s e t i m e s h a v i n g a n e g a t i v e - e x p o n e n t i a l d i s t r i b u t i o n . I n t h e m o r e g e n e r a l s y s t e m c o n s i d e r e d h e r e e a c h of a a n d b is a l l o w e d t o d e p e n d on t h e n u m b e r r of p e r s o n s p r e s e n t in t h e queue. A s i m i l a r e x t e n s i o n of t h e classical q u e u i n g s i t u a - t i o n h a s b e e n c o n s i d e r e d p r e v i o u s l y b y J e n s e n [18].

T h e q - m a t r i x a t (34) is c o n s e r v a t i v e : i t will b e r e g u l a r a n d so d e f i n e a u n i q u e M a r k o v p r o c e s s if a n d o n l y if t h e s e t of e q u a t i o n s

(~ + b0) Yo = b0 Yi,

( ~ + a r + b r ) y r = a r y r - l § (r_> 1),

h a s no b o u n d e d s o l u t i o n o t h e r t h a n Yo = Yl = Y2 . . . 0 for a n y one ( a n d t h e n for all)

~ > 0. I n l o o k i n g for a n o n - n u l l s o l u t i o n we m a y c l e a r l y a s s u m e t h a t Y0 = 1 a n d t h e I l t h e r e l a t i o n s

y l = 1 § ~/bo,

b r ( y r + ~ - y , ) = 2 y ~ + a ~ ( y ~ - y r - 1 ) , ( r > _ l ) t (35) show i n d u c t i v e l y t h a t 1 = Y0 < Yl < Y, < "" ". W e t h e r e f o r e p u t Y-= l i m y, _< ~ . F r o m (35) we f i n d t h a t

l y , a~ yr-1 a . . . . a l Y0/

Y r + l - Y r = ~ ~- + ~ 4 " ' + b ~ b lbo] ~ (36) for all r_>0. N o w if

t h e n (36) gives

1 ar ar ... a 1

= - - -~- § (r >_ O),

c~-- br ' b~br_l b . . . . bib o

~Cr<_yr+l--yr<_~Cryr ( r ~ 0 ) ,

O o

a n d so l + 2 ~ c r _ < Y_< I ~ I ( l + 2 c r ) .

0 0

W e c o n c l u d e t h a t the q . m a t r i x (34) is regular i / and only i /

r-0 +b, (37/

T h i s r e s u l t is d u e t o D o b r u ~ i n [6], w h o d e d u c e d i t f r o m F e l l e r ' s r e g u l a r i t y c r i t e r i o n ; t h e a d v a n t a g e s of t h e a l t e r n a t i v e a n a l y t i c a l c r i t e r i o n for r e g u l a r i t y a r e v e r y well s h o w n b y t h i s e x a m p l e .

1 We now revert to 0, 1, 2, ... a s t h e s t a t e - l a b e l s .

1 2 2 D A V I D G . K E N D A L L A N D G . E . H . R E U T E R

W e n o w s u p p o s e t h a t (37) is satisfied, so t h a t (34) is a s s o c i a t e d w i t h a u n i q u e M a r k o v process. T h e s o l u t i o n t o P r o b l e m C r e q u i r e s t h a t we f i n d all p o s i t i v e / - v e c t o r s x such t h a t

- b o x o + a l x 1 = 0, ( 3 8 )

br_lxr_l-(ar+br) xr+ar+lXr+l~O (r>_ 1). (39)

T h e s e e q u a t i o n s h a v e a o n e - d i m e n s i o n a l s e t of s o l u t i o n s g e n e r a t e d b y

b 0 b 0 b I b 0 ... br 1

x 0 ~ l ~ X l ~ - - , X 2 - ~ . . . , X r : - - ~ . . . ,

a 1 a 1 a 2 a 1 . . . a r

so t h a t we m u s t d i s t i n g u i s h b e t w e e n t w o cases, d e p e n d i n g on t h e n a t u r e of

S----l+ ~ b~ "'" b~-i or (40)

r = 1 a l a 2 9 9 9 a r

I f S = o~, t h e n e v e r y s t a t e is d i s s i p a t i v e , w h i l s t if S < 0o t h e n all t h e s t a t e s a r e p o s i t i v e a n d f o r m a single p o s i t i v e class. I n b o t h cases

1 b 0 ... bi-1 1

: r i 0 = ~ , ~r~i=a 1 . . . a j S (]_>1), (41)

for all i.

I t s h o u l d be n o t e d t h a t S = ~ a n d S < ~ a r e e a c h c o n s i s t e n t w i t h t h e r e g u l a r i t y c o n d i t i o n (37). I f a~=a a n d b~ = b for all v a l u e s of r t h e n ( 3 7 ) w i l l h o l d a n d we f i n d t h a t S < ~ if a n d o n l y if b < a (a w e l l - k n o w n result). T h e i m p o r t a n c e of t h e series (40) in q u e u i n g p r o b l e m s h a s b e e n p r e v i o u s l y n o t e d b y J e n s e n [18].

5.5. The general birth-and-death process. W e n o w c o n s i d e r t h e M a r k o v p r o c e s s a s s o c i a t e d w i t h t h e c o n s e r v a t i v e q - m a t r i x (34) w h e n b 0 = 0 a n d a~ > 0, br > 0, for r _> 1.

N a t u r a l l y we m u s t a g a i n r e q u i r e t h e q - m a t r i x t o be r e g u l a r , a n d t h e r e g u l a r i t y condi- t i o n will b e d e r i v e d in a m o m e n t . T h e s t a t e - l a b e l r will n o w be i d e n t i f i e d w i t h t h e n u m b e r of i n d i v i d u a l s in a p o p u l a t i o n for w h i c h t h e c h a n c e s of a b i r t h o r d e a t h o c c u r r i n g in t h e s h o r t t i m e i n t e r v a l 5 t a r e

b ~ t + o ( S t ) a n d a ~ t + o ( ~ t ) ,

r e s p e c t i v e l y . T h e c o n d i t i o n b 0 = 0 ensures t h a t t h e p o p u l a t i o n c a n n o t r e c o v e r if i t once b e c o m e s e x t i n c t . T h e f a m i l i a r " s i m p l e " b i r t h - a n d - d e a t h p r o c e s s c o r r e s p o n d s t o t h e choice

a f t r a , br--rb for t h e p a r a m e t e r s .

T H E C A L C U L A T I O N O F T H E E R G O D I C P R O J E C T I O N F O R M A R K O V C H A I N S 1 2 3

T h e q - m a t r i x will be r e g u l a r if a n d o n l y if, for s o m e one ( a n d t h e n for all)

> 0, t h e set of e q u a t i o n s

2y0 = 0,

(,~ + a~ + br ) y~ = a~ y r - l + b~ yr + ~ ( r > 1),

h a s no n o n - n u l l b o u n d e d solution. W i t h o u t loss of g e n e r a l i t y we m a y p u t Y0 = 0 a n d Yl = 1, a n d t h e n we s h a l l h a v e

Y2 = 1 + (~

+ al)/bi,

b ~ ( y r + l - y ~ ) = 2 y ~ + a r ( y ~ - y ~ - ~ ) (r_> 2),

so t h a t 1 = Y l <Yz < " " . A g a i n i t is c o n v e n i e n t t o p u t Y ~ lira y~, a n d v e r y m u c h a s

r - - ~

b e f o r e we f i n d t h a t

2d~ + e~ -<y~+x - Yr -< (2dr + er) y~

] ar ar ... a s

w h e r e dr ~ ~ + ~ - ] 4 " - + b~ b 2 b I '

(r_> 1), ar ... a 2 a l b~ ... b~ b I '

oo oo

a n d so 1 + ~ (er § ~ dr) <_ :Y ~ I-[ (1 + er + 2 dr).

1 1

I t follows t h a t the q - m a t r i x (34), w i t h b o = 0, i s r e g u l a r i / a n d o n l y i /

t l ar a . . . . a2 I = (42)

T h i s r e s u l t also is t o b e a t t r i b u t e d t o D o b r u ~ i n [6]. 1 N o t e t h a t (37) is in effect a c o n d i t i o n on bl, a2, b 2 . . . . o n l y , a n d t h a t i t is e q u i v a l e n t t o (42), so t h a t t h e r e g u l a r i t y of t h e s y s t e m does n o t d e p e n d on w h e t h e r t h e s t a t e r = 0 is or is n o t a b s o r b i n g .

W e n o w a s s u m e t h a t (42) h o l d s a n d f i n d t h e s o l u t i o n t o P r o b l e m C. F i r s t we m u s t f i n d all p o s i t i v e / - v e c t o r s x such t h a t

a 1 x 1 = 0~

- ( a I § bl) x 1 § a s x 2 = 0 ,

b~ - l X~- l - ( ar + br ) x~ + ar + l X~ + l = O (r_> 2),

a n d we see a t once t h a t t h e g e n e r a l s o l u t i o n is a n a r b i t r a r y n o n - n e g a t i v e m u l t i p l e o f t h e v e c t o r

u ~ 0, 0, 0 . . . . ].

T h u s t h e z e r o - s t a t e is p o s i t i v e a n d f o r m s b y i t s e l f t h e o n l y p o s i t i v e class C; all o t h e r s t a t e s a r e d i s s i p a t i v e . A c c o r d i n g l y we shall h a v e

1 See also KARLIN & MCGREGOR [19].

1 2 4 D A V I D G . K E N D A L L A N D G . E . t I . R E U T E R

~,j = 0 (j_> 1, all i)

a n d 7~i0 = ~5~ (i, C) (all i),

so t h a t t h e missing c o l u m n of t h e H - m a t r i x consists of t h e e x t i n c t i o n - p r o b a b i l i t i e s (i, C) ~ lira p~ 0 (t) (i = 0, 1, 2 . . . . ) ;

t--~+ ~

it is clear t h a t this limit is a t t a i n e d m o n o t o n i c a l l y , a n d t h a t it could be i n t e r p r e t e d as t h e chance t h a t t h e p o p u l a t i o n , initially of size i, will u l t i m a t e l y b e c o m e extinct.

Our solution to P r o b l e m C shows t h a t t h e v e c t o r ~ whose i t h c o m p o n e n t is (i, C) is t h e least n o n - n e g a t i v e b o u n d e d solution y t o t h e e q u a t i o n s

aryr 1 - ( a r + b r ) y ~ + b ~ y ~ + l = O ( r = 1, 2, 3 . . . . ), (43) subject to t h e r e q u i r e m e n t t h a t

(y, u ~ = 1.

T h e general solution to t h e difference e q u a t i o n (43) is ( r ~ al ... as)

y o = A , y ~ = A + B 1 + ( r > l ) ,

where A a n d B are a r b i t r a r y constants, a n d t h e n a t u r e of its b o u n d e d solutions will d e p e n d on

a l . . . a s

T ~ 1 + (44)

s=l bl bs"

I f T = ~ , we m u s t h a v e ~ [ 1 , 1, 1 . . . . ]

a n d e x t i n c t i o n is a l m o s t certain w h a t e v e r t h e initial state. I f h o w e v e r T < ~ , t h e n

~ ( Q ~ ) will be t w o - d i m e n s i o n a l a n d t h e m i n i m a l i t y condition comes into p l a y ; we find t h e n t h a t t h e v e c t o r ~ a n d t h e i n d i v i d u a l extinction probabilities are g i v e n b y

~ 0 = 7~00 = 1, /

1 ~

^{a 1...}

a~ / (45)

~ = zti o = ~' s=~'-" b 1 .. . bs (i >~ 1).

As in 5.4 it should be n o t e d t h a t T = ~ a n d T < ~ are b o t h consistent w i t h t h e r e g u l a r i t y condition (42): for i n s t a n c e if a ~ = r a a n d b ~ = r b (the " s i m p l e " b i r t h - a n d - d e a t h process) t h e n (42) a l w a y s holds, whilst T < ~ if a n d o n l y if a < b. I t is also w o r t h o b s e r v i n g t h a t t h e conditions S = ~ (in 5.4) a n d T = ~ (in 5.5) are e n t i r e l y different in character. R o u g h l y s p e a k i n g a queuing process will be c o m p l e t e l y dissi- p a t i v e (S = ~ ) w h e n t h e ratios b~/a~ are too large, whilst t h e c o r r e s p o n d i n g b i r t h - a n d - d e a t h process will h a v e t h e u n i v e r s a l e x t i n c t i o n p r o p e r t y ( T = ~ ) w h e n t h e s e

T H E C A L C U L A T I O N O F T H E E R G O D I C P R O J E C T I O N F O R M A R K O V C t t A I N S 1 2 5

ratios are too small. This is of course quite reasonable from the intuitive standpoint.

Finally it should also be mentioned t h a t the H - m a t r i x for the examples in 5.4 and 5.5 has been previously found b y L e d e r m a n n and R e u t e r [26] assuming the q-matrix to be regular (the regularity conditions (37) and (42) were not known to t h e m ) ; the present method is however m u c h simpler.

6. The calculation of R ( I I ) and ~'/(FI)

6.t. A di//erent approach to Problem B.

I n [22] we t r e a t e d a generalisation of Problem B: to determine the subspace F of "ergodic" vectors x E X and the associated

"ergodic projection operator" [I (defined on F), for a one-parameter semigroup ( P t : t _ > 0 } of operators on a Banach space X. A vector x was there called ergodic whenever [I x ~ lim

P t x

existed as some kind of generalised ]imit and with regard

t-->co

to some suitable topology for X. This work formed a natural continuation of earlier investigations in general ergodic theory b y Dunford [10], Hille ([17], Ch. X I V ) and Phillips [30], b u t for us it was also p r o m p t e d b y a desire to solve Problem B of the present paper. I f we set

X ~ l

in [22], the resulting theorems are not always ap- plicable to Problem B, b u t the methods of [22] can be a d a p t e d to yield a partial solution. This has an entirely different character from the solution given in w 3, where the lattice properties of 1 were exploited: we shall now have to s t u d y the interplay of a v a r i e t y of weak topologies. I t should be emphasised t h a t the solution of w 3 is superior in t h a t it is always availabJe; the methods to be described now m a y sometimes fail altogether b u t have their own merits whenever t h e y can be applied.

Problem A can also be treated b y similar m e t h o d s ; the analysis is then simpler a n d we leave the details to t h e reader.

As in w 3, we consider a Markov process whose a r r a y of transition probabilities is {p~j(t):i, ] = 0 , l, 2 . . . . }. L e t ~ be the infinitesimal generator of the associated transition semigroup

(Pt

: t ~ 0 } on l; also, with z i j - - l i m

pij(t)

as before, define the ergodic projection operator II b y

(H x)j--=

~ x~,zt~s (xel).

r

We recall (el. (9) and (15)) t h a t

I I 2 = P t I I = I I P t = H

for all t_>0. (46) I t is i m p o r t a n t to note t h a t the operator II is in general

not

the same as the operator 1] of [22].

126 D A V I D G . K E N D A L L A N D G . E . H . R E U T E R

Because II is idempotent we can write a n y x E l uniquely in the form x = x ~ + x2, where x 1E ~ ( [ I ) and x 2 E ~ (II), and x 1 - I x. Thus we have the direct decomposition

1 = ~ (I]) Q ~/(II). (47)

If we can find the summands ~(1-I) and ~/(H) in (47), then we can find H x , for any x E1, by taking the component of x in ~ ( [ I ) . I t will now be seen t h a t we m a y rephrase Problem B as follows.

P r o b l e m B I : Determine ~ ( I I ) and ~ ( I I ) when ~ is given.

We shall see t h a t ~ ( I I ) is easy to identify, and coincides with ~/(~). The deter- mination of ~/(II) is more troublesome ; it turns out t h a t ~ (g2) _c ~/(1]), and we shall t r y to reach T/(II) by closing ~ ( ~ ) with regard to several weak topologies for l.

6.2. WeaIc topologies. I t will be convenient for future reference to state some results which we shall need concerning weak topologies for a Banach space X. (Cf.

[5] and [3], Ch. IV.)

Let G be a linear set of funetionals g E X * which is " t o t a l " for X, i.e. such t h a t the vanishing of (g, x) for all g E G implies x = 0. Then the G-weak topology for X is the (Hausdorff) topology generated by the sub-basic open sets

{x:xeX,~<(g,x)<~} (~<~),

where ~, fl range over the real numbers and g ranges over G. If E _ X then we shall write E for the strong and [E]G for the G-weak closures of E. Also if A G X and B c _ X * we shall write A • and B T for the annihilators of A in X* and B in X :

A • * and ( y , x ) = O for all x E A ) , (48)

B T - ( x : x E X and ( y , x ) = 0 for all y E B } . (49) The following fact (Dieudonn~ [5], Th. 5) will often be used:

when L is a linear subset o/ X , then

[L]~ = (L~ n G) T. (50)

I n particular, strongly closed linear subsets of X being also weakly closed (i.e. closed in the weak topology obtained b y taking G ~ X * ) :

when L is a linear subset o/ X , then

L = (L• T. (51)

T H E C A L C U L A T I O N OF T H E E R G O D I C P R O J E C T I O N F O R M A R K O V C H A I N S 127 The preceding considerations a p p l y equally well, of course, to the Banach space X*. We shall need only the " w e a k * " topology (cf. Loomis [29], w 9): this is the weak topology induced on X* b y X (regarded as a subspace of X**). The s t a t e m e n t at (50) t h e n becomes:

when M is a linear subset o/ X*, then the weak* closure o] M is (MT) •

Finally we shall need the fact t h a t every/inite-dimensional linear subset o / X * is weak*

closed (see Bourbaki [3], Ch. I, w 2, No. 3).

The preceding results will be applied chiefly to the space X - - l and its adjoint X * ~ m , b u t at one point it will be useful to note t h a t t h e y apply equally to the space X ~ e o and its adjoint X*=--l; some care will t h e n be necessary in manipulating the symbols ( )l and ( )T. (We assume t h a t the reader is familiar with the ele- m e n t a r y properties of the spaces Co, l and m ; these can be found in Banach [1].

As usual we write c o for t h a t subspaee of m which consists of sequences z~(zo, z i . . . . ) such t h a t z~-->0 when ~ - ~ . )

I n w h a t follows we shall often require the G-weak closure of the linear subset

~ ( ~ ) of X ~ l for various total linear subsets G of X * ~ m . I t is readily seen ([22], p. 167, eq. (42/) t h a t ( ~ ( ~ ) ) l = ~ ( ~ * ) ; this shows incidentally t h a t 7/(fl*), being of the form E • (where E E l ) ,

from (51) and (50) t h a t

and

is always a weak* closed subset of m = l * . I t follows

R ( ~ ) = ( ~ (n*)) T (52)

[~ (~)]G = (7/(~*) N G) T. (53)

6.3. A partial solution to Problem B r We shall write u ~ and v j for the i t h and j t h unit vectors in l and m, so t h a t (u*)j = (vS),=(~ij and p,j(t)= (v j, Ptu*). Then, for each i and j, (v j, ( P t - H ) u * ) - ~ 0 as t - - > ~ . B u t I I P t - I I H < 2 so t h a t a double ap- plication of the Banach-Steinhaus theorem, z together with the facts t h a t the linear sets spanned b y the u ~ a n d the v j are dense in l and in co, yields the i m p o r t a n t relation lim (z, Pt x) = (z, I I x) (x E l, z E co). (54) We now introduce the transition operator 2 J a defined for all ;t > 0 b y

2 J ~ x - - f t e - ~ t P t x d t (xEl) (55)

0 I S e e H I L L E [17], T h . 2 . 1 2 . 1 .