Chernoff-type bound for finite Markov chains

HAL Id: hal-00940907

https://hal-enac.archives-ouvertes.fr/hal-00940907

Submitted on 1 Apr 2014

HAL

is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire

destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Pascal Lezaud

To cite this version:

Pascal Lezaud. Chernoff-type bound for finite Markov chains. Annals of Applied Probability, Institute

of Mathematical Statistics (IMS), 1998, 8 (3), pp 849-867. �hal-00940907�

CHERNOFF-TYPE BOUND FOR FINITE MARKOV CHAINS

By Pascal Lezaud

CENA and Universit´e Paul Sabatier

This paper develops bounds on the distribution function of the empiri- cal mean for irreducible finite-state Markov chains. One approach, explored by Gillman, reduces this problem to bounding the largest eigenvalue of a perturbation of the transition matrix for the Markov chain. By using esti- mates on eigenvalues given in Kato’s bookPerturbation Theory for Linear Operators, we simplify the proof of Gillman and extend it to nonreversible finite-state Markov chains and continuous time. We also set out another method, directly applicable to some general ergodic Markov kernels having a spectral gap.

1. Introduction. LetX_n‘be an irreducible Markov chain on a finite set Gwith transition matrixPand stationary distributionπ:Then, for any func- tion f and any initial distribution q, the weak law of large numbers states that for almost every trajectory of the Markov chain, the empirical mean n⁻¹Pn

i=1fXi‘ converges to πf = P

yπy‘fy‘: This result is the basis of the Markov chain simulation method to evaluate the mean of the function f: In this paper, we will quantify this rate of convergence by studying the probability

P_q

n⁻¹

n

X

i=1

fX_i‘ −πf≥γ

;

whereP_q denotes the probability measure of the chain with the initial distri- bution q; and the size of deviation, γ; is a small number, such as πf/10 or πf/100:

The corresponding rate of convergence for sums of independent random variables has been given by Kolmogorov (1929), Cram´er (1938), Chernoff (1952), Bahadur and Ranga Rao (1960) and Bennett (1962), but the sam- ples in Markov chains are generally correlated with each other, even if this correlation decreases exponentially with the number of steps. In that case, the theory of large deviations gives an asymptotic rate of convergence, since the empirical mean satisfies the large deviation principle with the good rate functionIz‘ =sup_r∈R”rz−logβ₀r‘•;where β₀r‘is the largest eigenvalue (i.e., Perron–Frobenius eigenvalue) of a perturbation of the transition matrix P[see Dembo and Zeitouni (1993)]. This asymptotic result is not satisfactory if one wants to achieve bounds that are useful for fixed n: Using perturba- tion theory, Gillman (1993) estimated the rate of convergence for reversible finite-state Markov chains by bounding the eigenvalueβ₀r‘:He got a bound in terms of the spectral gap of the original transition matrix P; defined

Received December 1996; revised June 1997.

AMS1991subject classification. 60F10.

Key words and phrases. Markov chain, Chernoff bound, eigenvalues, perturbation theory.

849

byεP‘ =1−β₁P‘;whereβ₁P‘denotes the second largest eigenvalue ofP:

Using a technique adapted from Rellich, Dinwoodie (1995) improved the bound obtained by Gillman, but only for γsmall. Improvements of such bounds are motivated by their wide use in simulation.

The aim of this work, based on Kato’s perturbation theory, is to obtain a bound which depends on the variance off and which is applicable for allγ:

More precisely, this bound provides a Gaussian behavior for the small values of γ and a Poissonian behavior for the large values. Moreover, the result we achieved can also be easily extended to nonreversible Markov chains and con- tinuous time. For instance, we get in the reversible case the following theorem, proved in Section 3.1, where · ₂denotes the l²π‘-norm.

Theorem 1.1. LetP; π‘be an irreducible and reversible Markov chain on a finite setG:LetfxG→Rbe such thatπf=0,f_∞≤1and0<f²₂≤b²: Then, for any initial distributionq;any positive integernand all 0< γ≤1;

P_q

n⁻¹

n

X

i=1

fX_i‘ ≥γ

≤e^εP‘/5N_qexp

− nγ²εP‘

4b²1+h5γ/b²‘‘

; (1)

whereN_q= q/π₂and

hx‘ = ¹₂ p

1+x− 1−x/2‘

: Therefore, ifγ≪b²andεP‘ ≪1;the bound is

1+o1‘‘N_qexp

− nγ²εP‘

4b²1+o1‘‘

:

Remark 1. For γ > 1; the probability of deviation is zero; thus, we can assume that γ ≤ 1: Moreover, applying Theorem 1.1 to −f gives the same bound on P_q’n⁻¹Pn

i=1fX_i‘ ≤ −γ“: Thus, inequality (1) implies bounds on all moments of the positive random variable Žn⁻¹Pn

i fX_i‘Ž; and quantifies convergence in eachl^pnorm, 1≤p <∞:

Remark 2. hx‘is an increasing function of x ≥ 0 such thathx‘ ≤ x/2 for allx:Thus, forγ≤2b²/5, we get

P_q

n⁻¹

n

X

i=1

fX_i‘ ≥γ

≤e^εP‘/5N_qexp

−nγ²εP‘

4b²

1− 5γ 2b²

; whereas forγ >2b²/5;

P_q

n⁻¹

n

X

i=1

fX_i‘ ≥γ

≤e^εP‘/5N_qexp

−nγεP‘

10

1−2b² 5γ

:

Remark 3. Obviously, we can takeb²=1 in inequality (1), which gives Pq

n⁻¹

n

X

i=1

fXi‘ ≥γ

≤e^εP‘/5Nqexp’−nγ²εP‘/12“:

(2)

The constantN_qis thel₂π‘-norm of the density ofqrelated to the station- ary distribution π: This norm can be large but is always bounded by π∗^−1/2, whereπ_∗ =min”πy‘xy∈G•:However, it is worth noting thatNq=1 when the initial distribution is the stationary one. This fact will be used to im- prove the convergence, by splitting the simulation in two steps, as suggested in Aldous (1987). The first step, the initial transient period, may be defined as the period of time which must be discarded before the chain comes close toπ;

whereas the second step, the observation period, may be defined as the period of time where observations must be collected to reach the desired precision.

A short description of the paper is as follows. Section 2 of this paper sets out preliminaries on the perturbation theory of linear operators. The results taken from Kato (1966) are based on a function-theoretical study of the resolvent, in particular on the expression of the eigenprojections as contour integrals of the resolvent. However, the specific results needed here are easier than in Kato (1966), since we restrict our study to the case of operators having an eigenvalue whose algebraic multiplicity is 1.

Section 3.1 proves Theorem 1.1. The bound is extended to nonreversible Markov chains in Section 3.2, by considering the multiplicative symmetriza- tion of the operator P;namely, K = P^∗P; where P^∗ is the adjoint of P on l₂π‘:Explicitly,

P^∗x; y‘ =πy‘Py; x‘/πx‘:

Dinwoodie (1995) also obtained a bound for nonreversible Markov chain, but his bound depended on the random covering time Tdefined by T=inf”n≥ 1x ”X₀; X₁; : : : ; X_n• =G•:Bounding this covering time can be difficult.

Section 3.3 shows how the previous bounds extend to continuous-time Markov chains, by employing a method suggested by Fill (1991). We thus get an exponential bound in terms of the smallest nonzero eigenvalue of

−3+3^∗‘/2; where3is the generator of the semi-groupP_t:

Perturbation theory of linear operators is a fruitful method to achieve ex- plicit bounds on finite Markov chains. Mann (1996) used it to obtain a Berry–

Esseen bound for Markov chains with finite or countably infinite state space, such that

βx=sup

Pg₂

g₂

<1;

(3)

where the sup is over all complex-valued functions on the state spaceGwith πg=0:Nagaev (1957, 1961) used also the perturbation method for studying some limit theorems for Markov chains under the Doeblin condition. Following the method used in a setting of the finite-state space, we recently have been able to achieve a Chernoff-type bound for general Markov operatorP under the assumptions that 1 is an isolated eigenvalue ofPand the dimension of the eigenspace for the eigenvalue 1 is finite. In particular, these assumptions are fulfilled for Markov chains satisfying the Doeblin condition or the relation (3).

We will publish this result in a forthcoming paper. Instead of presenting these results, we would rather set out another method, directly applicable to a gen-

eral irreducible Markov kernel assuming condition (3). Section 4 introduces this method, due to Bakry and Ledoux, which gives the following inequality forγ≤b²:

P_π

n⁻¹

n

X

i=1

fX_i‘ −πf≥γ

≤exp

−n㲐1−β‘

21b²

:

Note that this last inequality involves β and not εP‘: For instance, if the state space is finite and the Markov chain is periodic, then β=1:Thus, this method is not applicable in this case, unlike the perturbation method.

Finally, Section 5 shows how the running time of simulation can be im- proved by considering an initial burning period whose effect is to decreaseN_q: 2. Perturbation theory of linear operators. LetTbe a linear operator on some finite-dimensional vector spaceX: We denote the spectrum ofT by 6T‘:The resolvent ofT, defined byRζ‘ = T−ζ‘⁻¹, is an analytic operator- valued function on the domainC^/ \6T‘;called the resolvent set. Furthermore, the only singularities ofRζ‘are the eigenvaluesλ_h,h=1; : : : ; s;ofT:

The Laurent series of the resolvent at a simple eigenvalue λ_h takes the form [Kato (1966), page 38]

Rζ‘ = −ζ−λ_h‘⁻¹P_h+

∞

X

n=0

ζ−λ_h‘ⁿSⁿ⁺¹_h ;

whereP_his the eigenprojection operator for the eigenvalueλ_handS_his called the reduced resolvent ofTwith respect to the eigenvalueλ_h: More precisely, S_h is the inverse of the restriction of T−λ_h in the subspaceI−P_h‘X:We deduce thatPh is the residue of−Rζ‘inλh;and

P_h= − 1 2πi

Z

Ŵ_hRζ‘dζ;

whereŴ_his a positively oriented small circle enclosingλ_h;but excluding other eigenvalues ofT:

Consider a family of operator-valued functions with the form Tχ‘ =T+χT^1‘+χ²T^2‘+ · · · :

Then the resolvent Rζ; χ‘ = Tχ‘ −ζ‘⁻¹ of Tχ‘ is analytic in the two variablesζ; χin each domain in whichζ is not equal to any of the eigenvalues ofTχ‘[Kato (1966), page 66]. So it can be expanded into the following power series inχwith coefficients depending onζ:

Rζ; χ‘ =Rζ‘ +

∞

X

n=1

χⁿR^n‘ζ‘;

(4)

where eachR^n‘ is an operator-valued function. This series is uniformly con- vergent if

∞

X

n=1

ŽχŽⁿT^n‘Rζ‘<1:

(5)

Letrζ‘be the value ofŽχŽsuch that the left member of (5) is equal to 1:Then (5) is satisfied forŽχŽ< rζ‘:Letλbe one of the eigenvalues ofT=T0‘with multiplicitym=1;andŴbe a positively oriented circle, in the resolvent set of T;enclosingλbut no other eigenvalues ofT:The series (4) is then uniformly convergent forζ∈Ŵif

ŽχŽ< r₀=min

ζ∈Ŵ rζ‘:

(6)

In the special case in which X is a Hilbert space and T is normal (i.e., T^∗T=TT^∗), we get

Rζ‘ =1/distζ; 6T‘‘;

r₀=min

ζ∈Ŵ

a

distζ; 6T‘‘+c −1

for every ζ in the resolvent set of T; where a = T^1‘ and c is such that

T^n‘ ≤acⁿ⁻¹: If we choose asŴthe circleŽζ−λŽ =d/2, where d= min

µ∈6T‘\”λ•Žλ−µŽ;

we obtainr₀= 2ad⁻¹+c‘⁻¹:

The existence of the resolventRζ; χ‘ for ζ ∈ Ŵimplies that there are no eigenvalues ofTχ‘onŴ: The operator

Pχ‘ = − 1 2πi

Z

ŴRζ; χ‘dζ (7)

is the eigenprojection for all the eigenvalues ofTχ‘lying insideŴ: In partic- ular [Kato (1966), page 68], for allŽχŽsufficiently small, we have

dimPχ‘X=dimPX=1:

Therefore, only the eigenvalue λχ‘ of Tχ‘ lies inside Ŵ; and Pχ‘ is the eigenprojection for this eigenvalue.

As the only eigenvalues ofTχ‘Pχ‘are 0 andλχ‘; we will consider λχ‘ −λ=trTχ‘ −λ‘Pχ‘‘:

Combining (7) and substitution for Rζ; χ‘ from (4) give the Taylor series expansion [Kato (1966), page 79]

λχ‘ −λ= − 1 2πitrZ

Ŵζ−λ‘Rζ; χ‘dζ =

∞

X

n=1

χⁿλ^n‘; where

λ^n‘ =

n

X

p=1

−1‘^p p

X

ν₁+···+ν_p=n k₁+···+k_p=p−1

ν_i≥1; k_j≥0

tr’T^ν1S^k1‘· · ·T^νpS^kp‘“;

(8)

withS^0‘= −P, S^n‘=Sⁿ:HereSis the reduced resolvent ofTwith respect to the eigenvalueλ:

3. Chernoff-type bound for Markov chains. In this section, we con- sider an irreducible finite Markov chain with transition matrix P and sta- tionary distribution π: This transition matrix defines an operator acting on functions by

Pfx‘ = X

y∈G

Px; y‘fy‘:

We are interested in bounding the rate of convergence of P_q’n⁻¹t_n≥γ“; wheret_n=

n

X

i=1

fX_i‘ −πf:

If the Markov chain is reversible, the operator P is self-adjoint on l₂π‘

endowed with the inner product

Œf; g = X

x∈G

fx‘gx‘πx‘:

This operator P has largest eigenvalue 1 and because of irreducibility, the constant functions are the only eigenfunctions with eigenvalue 1:These eigen- values are denoted by

β₀P‘ =1> β₁P‘ ≥β₂P‘ ≥ · · · ≥βŽGŽ−1P‘ ≥ −1;

and the spectral gap by εP‘ =1−β₁P‘:

IfP; π‘ is not reversible, we will consider the multiplicative symmetriza- tionK=P^∗PofP:This operatorKis a reversible Markov kernel with same stationary distributionπ:We will assume thatKis irreducible. For instance, this will be the case as soon asPx; x‘>0 for everyx∈G:

Since we will always work with a self-adjoint operator onl₂π‘, every norm considered will be the l₂π‘-norm. Moreover, except for constant functions without interest, replacing fby f−πf‘/f−πf_∞ shows that there is no loss of generality in assuming

πf=0; f_∞≤1; f²₂≤b²≤1:

3.1. Reversible Markov chains. This section proves Theorem 1.1 stated in the Introduction, by using the method of Gillman (1993). We first prove the following lemma.

Lemma 3.1. Referring to the setting of Theorem 1.1, letr >0:Then, for any positive integern,

P_q’t_n/n≥γ“ ≤e^−rnγq^TPr‘ⁿ1;

wherePr‘ = e^rfy‘Px; y‘‘:

Proof. By Markov’s inequality,

P_q’t_n≥nγ“ ≤e^−rnγE_qexprt_n‘;

where E_q denotes the expectation given the initial distribution q: Observe that

E_qexprt_n‘ = X

x₀; x₁;:::;x_n

exprt_n‘qx₀‘

n

Y

i=1

Px_i−1; x_i‘;

where the summation is evaluated over all possible trajectoriesx₀; x₁; : : : ; x_n: By introducing the operatorPr‘ = e^rfy‘Px; y‘‘;we obtain the expression

E_qexprt_n‘ =q^TPr‘ⁿ1:

Lemma 3.2. Letr >0 andN_q = q/π₂:Then q^TPr‘ⁿ1≤e^rN_qβⁿ₀r‘;

whereβ₀Pr‘‘is the largest eigenvalue ofPr‘.

Proof. Let us recall that Pis a self-adjoint operator in l₂π‘;and intro- duce the diagonal matrixE_r=diage^rfy‘‘; so thatPr‘ =PE_r:ThenPr‘ = pE⁻¹_r p

E_rPp E_rp

E_ris similar to the self-adjoint matrix Sr‘ =p E_rPp

E_r; so its eigenvalues are real values forr≥0:In this case, the Perron–Frobenius theorem says thatSr‘_2→2=β₀Pr‘‘;whereβ₀Pr‘‘is the largest eigen- value ofPr‘: Applying the Cauchy–Schwarz inequality inl₂π‘gives

q^TPr‘ⁿ1= q

π; Pr‘ⁿ1

π

≤N_q1₂E_r/2_2→2E_−r/2_2→2Sr‘ⁿ_2→2

≤e^rN_qβⁿ₀Pr‘‘:

Thus, Lemma 3.2 gives the inequality

P_q’t_n/n≥γ“ ≤e^rN_qexp”−n’rγ−logβ₀Pr‘‘“•:

(9)

LetD=diagfx‘‘; then we can write Pr‘ =P+

∞

X

i=1

rⁱ i!PDⁱ:

Since P ≤ Pr‘ ≤ e^rP; we get 1 ≤ β₀Pr‘‘ ≤ e^r [see Kato (1966), Theo- rem I-6.44]. It follows that β₀Pr‘‘ is the perturbation of the eigenvalue 1 of P: The irreducibility of P implies that 1 is a simple eigenvalue and the eigenprojection for the eigenvalue 1 is the operatorπ defined by

πfx‘ x=X

x

fx‘πx‘:

Thus (8) can be used to obtain the coefficients β^n‘ in the Taylor series of β₀Pr‘‘ −1;so long asr < r₀, wherer₀is the convergence radius defined by

(6). We obtain β^n‘=

n

X

p=1

−1‘^p p

X

ν₁+···+ν_p=n k₁+···+kp=p−1

ν_i≥1; k_j≥0

1

ν₁!: : : ν_p!Œf^ν1; S^k1‘PD^ν2· · ·S^kp−1‘Pf^νp;

(10)

where we used the following relation whatever the matrixM:

trπPD^ν1MD^νp‘ =trπD^ν1MD^νp‘ = Œf^ν1; Mf^νpπ: For instance,

β^1‘= Œf;1_π =0; β^2‘= Œf;−Sf_π−¹₂Œf; f_π:

An explicit calculation gives β^2‘ = lim_n→∞Eπt²_n/n; which is precisely the asymptotic variance oft_n:

The number of terms in the formula (10) is Cn‘ x=

n

X

p=1

n−1 p−1

2p−1‘

p−1 1

p: Furthermore, as1/n!‘ ≤2¹⁻ⁿand ^2k_k

<2^2kkπ‘^−1/2, we obtain the following inequalities forn≥3:

Cn‘<1+π^−1/2

n−1

X

p=1

n−1 p

1 p+14^p≤

25 4n

5ⁿ⁻²:

Thus, forn≥7;we getCn‘ ≤5ⁿ⁻²;but direct computations show this bound is also valid forn≥3:Finally, the Cauchy–Schwarz inequality gives

β^n‘≤ b²/5‘5/εP‘‘ⁿ⁻¹; n≥2:

Hence, provided thatr < εP‘/5;we obtain β₀Pr‘‘ ≤1+ b²

εP‘r²

1− 5r εP‘

−1

: (11)

Combining inequalities (9) with (11) and log1+x‘ ≤xgives, forr < εP‘/5;

P_q’t_n/n≥γ“ ≤e^εP‘/5N_qexp−nQr‘‘;

whereQr‘ =γr− b²/εP‘‘1−5r/εP‘‘⁻¹r²is maximized when

r= γεP‘

b²1+5γ/b²+p

1+5γ/b²‘:

We can check that r < εP‘/5; and simple computations yield inequality (1). ✷

3.2. Nonreversible Markov chains. This section extends the previous bound to nonreversible Markov chains. We now consider the multiplicative sym- metrization of P; namely, K = P^∗P: Set Kr‘ = P^∗r‘Pr‘ and denote its largest eigenvalue by β₀r‘: As in the previous section, Markov’s inequality gives

P_q’t_n/n≥γ“ ≤e^−rnγq^TPr‘ⁿ1

≤e^−rnγN_q1₂Pr‘ⁿ_2→2

=e^−rnγNqPr‘^∗‘ⁿPr‘ⁿ^1/2_2→2

=e^−rnγN_qq

β_{0; n}r‘;

where β_{0; n}r‘ denotes the largest eigenvalue of Pr‘^∗‘ⁿPr‘ⁿ: Result anal- ogous to Lemma 3.2 requires the use of Marcus’ Theorem [see Marshall and Olkin (1979), Theorem 9.H.2.a]. This theorem states that β_{0; n}r‘ ≤ β₀r‘ⁿ; whence the following inequality:

P_q’t_n/n≥γ“ ≤N_qβ^n/2₀ r‘e^−rnγ:

Here again, the operatorKr‘ will be considered as a perturbation of K;

since we have

Kr‘ =K+

∞

X

i=1

rⁱ i

X

j=0

1

j!i−j‘!D^jKD^i−j

:

In the remainder of this section, we assume thatKis irreducible. From here, the arguments are exactly the same as the ones used in the proof of Theo- rem 1.1. The coefficientsβ^n‘ in the Taylor series ofβ₀r‘ −1 are bounded by

β^n‘≤2b²2/εK‘‘ⁿ⁻¹Cn‘ ≤ 2/5‘b²10/εK‘‘ⁿ⁻¹: Therefore, provided thatr < εK‘/10;we obtain

β₀r‘ ≤1+ 4b² εK‘r²

1− 10r εK‘

−1

y hence,

P_q’t_n/n≥γ“ ≤N_qexp

−n

rγ− 4b²

εK‘r²1− 10r‘/εK‘‘⁻¹

: Optimizing inr < εK‘/10;we may therefore state the following.

Theorem 3.3. Let P; π‘ be an irreducible Markov chain on a finite set G; such that its multiplicative symmetrization K = P^∗P is irreducible. Let fx G→ Rbe such that πf=0, f_∞ ≤ 1 and 0< f²₂ ≤ b²: Then, for any initial distributionq;any positive integernand all 0< γ≤1;

P_q’t_n/n≥γ“ ≤N_qexp

− nγ²εK‘

8b²1+h5γ/b²‘‘

;

whereεK‘is the spectral gap ofKand hx‘ = ¹₂ p

1+x− 1−x/2‘

:

Obviously, with some modifications, the remarks stated in the Introduction are also valid.

3.3. Continuous-time Markov chain. In this section, we consider an irre- ducible continuous-time Markov chain on a finite-state space. Ifπdenotes its unique stationary distribution, the weak law of large numbers states that, for any functionsf;

P 1

t Z^t

0

fX_s‘ds−πf≥γ

→0 ast→ ∞:

We can express the previous integral as the following limit:

1 t

Z^t

0 fX_s‘ds= lim

k→∞k⁻¹

k

X

i=1

fX_it/k‘:

Fixk and assume thatπf=0; f_∞ ≤1 and 0<f²₂ ≤b²: The Markov kernelPt/k‘satisfies

Pt/k‘x; y‘>0 ∀t >0 ∀x; y‘ ∈G²:

It follows that the Markov kernelKt/k‘ =P^∗t/k‘Pt/k‘is irreducible. Let β₁t/k‘be the second largest eigenvalue ofKt/k‘and denote its spectral gap byεt/k‘ =1−β₁t/k‘:Hence, for all 0≤γ≤1;Theorem 3.3 implies

P_q

k⁻¹

k

X

i=1

fX_it/k‘ ≥γ

≤N_qexp

− kγ²εt/k‘

8b²1+h5γ/b²‘‘

: (12)

If 3 denotes the generator of the semigroup P_t‘; the generator of P^∗_t‘ is 3^∗:So, we deduce that, for k→ ∞; Kt/k‘ =I+ t/k‘3+3^∗‘ +o1/k²‘and εt/k‘ =2t/k‘λ₁+o1/k²‘;whereλ₁denotes the smallest positive eigenvalue of −3+3^∗‘/2: Now applying Fatou’s lemma for k → ∞ to inequality (12), gives the following theorem.

Theorem 3.4. LetPt; π‘be an irreducible continuous-time Markov chain on a finite setG;and3its infinitesimal generator. LetfxG→Rbe such that πf=0,f_∞≤1and0<f²₂≤b²:Then, for any initial distributionq; all t >0and all 0< γ≤1;

P_q 1

t Z^t

0

fX_s‘ds≥γ

≤N_qexp

− γ²λ₁t 4b²1+h5γ/b²‘‘

; whereλ₁is the smallest positive eigenvalue of −3+3^∗‘/2and

hx‘ = ¹₂ p

1+x− 1−x/2‘

:

Finally, boundingb²by 1 gives the boundN_qexp−γ²λ₁t/12‘;which is valid for allγ≤1:

Remark. Applying Lemma 3.1 with the transition matrixPt/k‘and us- ing the following formula valid for all matricesA; B:

k→∞limexpA/k‘expB/k‘‘^k=expA+B‘;

give

P_q 1

t Z^t

0 fX_s‘ds≥γ

≤e^−rtγN_qexp3r‘t‘_2→2;

where 3r‘ = 3+rdiagfx‘‘: Thus, we get a linear perturbation of the infinitesimal generator 3:Moreover, exp3r‘t‘_2→2≤exp−λ₀r‘t‘; where λ₀r‘ is the smallest eigenvalue of −3+3^∗‘/2−rD: By using the Trotter product formula [see Trotter (1959)], this method can be extended to general state space ergodic Markov processes.

4. Direct method. This section develops another method to obtain upper bound on the distribution function of the empirical measure. This method does not require the perturbation theory of linear operators and works easily in a more general setting.

From now on, we consider an ergodic Markov kernelPx; dy‘defined on a general state spaceG;with the stationary distributionπ:This kernel defines an operatorPacting onL²π‘by

Pgx‘ =Z

Ggy‘Px; dy‘:

Assume now that the condition (3) is fulfilled by P: Our goal is to obtain an upper bound for E_πexprPn

i=1fX_i‘‘; where 0 ≤ r and f is such that

f²₂≤b², πf=0 andf_∞≤1: Writeg=exprf‘; then E_πexp

r

n

X

i=1

fX_i‘

=Z

Q^n‘gdπ;

where, for allh∈L²π‘,

Q^1‘hx‘ =Z

Px; dy‘hy‘;

Q^n+1‘=Z

Px; dy‘gy‘Q^n‘hy‘:

The principle of the proof, due to Bakry and Ledoux, consists of centering each successive term on which the operatorPacts. Introduce some notation:

a₀=1; ai=Eπexp

r

i

X

j=1

fXj‘

; 1≤i≤n;

g₁=g−Z

g dπ; g_i=gPg_i−1−Z

gPg_i−1dπ; 2≤i≤n;

b₁=Z

g dπ; bi=Z

gPgi−1dπ=Z

g₁Pgi−1dπ; 2≤i≤ny

then

a_n=a_n−1b₁+Z

Q^n‘g₁dπ

=a_n−1b₁+a_n−2b₂+Z

Q^n−1‘g₂dπ :::

=

n

X

i=1

b_ia_n−i:

Now, each b_i can be bounded by using (3), since the operator P acts on the centered functionsg_i−1:Actually, letα=βe^r,a=b²e^r/2; then

b₁=Z

exprf‘dπ=1+r²

2E_πf²+r³

3!E_πf³+ · · · ≤1+ar²; b_i ≤βαⁱ⁻²g₁²₂; 2≤i < n;

where we used the inequalityg_i₂ ≤αⁱ⁻¹g₁₂, 1≤i≤ n:Furthermore, by Jensen’s inequality, we get R

exprf‘dπ≥1, so

g₁²₂=Z

exp2rf‘dπ− Z

exprf‘dπ 2

≤1+2r²e^2rE_πf²−1≤4ar²e^r:

Chooser <1−β;soα <1, since logβ≤ −1−β‘:Now, as induction hypothesis, suppose thata_k ≤φ^kr‘;whereφr‘ =1+Cr² andC≥4ais independent of k:As

a₁≤ g₂≤ 1+4ar²‘^1/2≤ 1+2ar²‘;

the hypothesis is true fork=1 and the induction hypothesis gives ak ≤

1+ar²‘φ^k−1r‘ +4ar²α^k−1

k−2

X

i=0

φr‘/α‘ⁱ

≤

1+ar²‘φ^k−1r‘ +4ar²αφ^k−1r‘

φr‘ −α

≤φ^k−1r‘”1+4ar²1−α‘⁻¹•:

Choosing C= 4a1−α‘⁻¹; it follows that the induction hypothesis is valid.

Hence, for every 0≤γ <4b² andr=γ1−β‘/4b²‘;we obtain, by Markov’s inequality,

Pπ’tn/n≥γ“ ≤exp”−nrγ−2b²r²e^r1−βe^r‘⁻¹‘•

=exp

−n1−β‘γ² 8b²

2−1−β‘exp’γ1−β‘“/4b²‘ 1−βexp’γ1−β‘“/4b²‘

:

Assume now thatγ≤2b²:Ase^x≤1+4x/3;forx≤1/2;we get the following:

Pπ’tn/n≥γ“ ≤exp

−n1−β‘γ² 8b²

1− γ

2b²

: (13)

This method is directly applicable to general Markov chains and gives the same type of asymptotic bound as the one obtained by the perturbation method for finite nonreversible Markov chains. Observe, however, that for finite re- versible Markov chains the bound (13) involves the constantβ;instead of the second largest eigenvalue of the kernelP:

Inequality (13) can be extended to continuous time by assuming thatf is continuous and bounded and

sup”Ptg₂/g₂; πg=0• ≤e^−tλ ∀t≥0 andλ >0:

(14)

Arguing as in the proof of Theorem 3.4, we get that, for everyγ≤2b²; P_π

1 t

Z^t

0

fX_s‘ds≥γ

≤exp

−γ²tλ 8b²

1− γ

2b²

:

An exponential inequality applicable for all 0 ≤ γ ≤ 1 can be obtained by following the proof of the Prokhorov’s inequality for independent random variables given in Stout (1974). We obtain the following inequality, when qπ,E_πf=0, f_∞≤aandf²₂≤b²:

P_π’t_n/n≥γ“ ≤exp

− γ

2aarcsinh

nγa1−β‘

4b²

:

Proof. For everyx≥0,e^x≥ex:Therefore,E_πexprt_n‘ ≤exp’E_πexprt_n‘−

1“:LetQx‘ =P_π’t_n≤x“y we get E_πexprt_n‘ ≤exp

Z

exprx‘ −1‘dQx‘

: Moreover,Qassigns mass 1 to the interval’−na; na“and

Z x dQx‘ =0; Z

x²dQx‘ =E_πt²_n‘ ≤2nb²1−β‘⁻¹: Thus,

E_πexprt_n‘ ≤exp Z

exprx‘ −1−rx‘dQx‘

≤exp

2Z

coshrx‘ −1‘dQx‘

: As the functionhx‘ =coshrx‘ −1 is even and convex, we obtain

Z

hx‘dQx‘ ≤Z

hx‘dQ₁x‘;

where Q₁ assigns mass only to 0 and to na; with as much mass as possible assigned tonawithout violating

Z x²dQ₁x‘ ≤2nb²1−β‘⁻¹y thus,Q₁na‘ ≤ 2b²‘n1−β‘a²‘⁻¹:This yields

E_πexprt_n‘ ≤exp

4b²

na²1−⑐coshrna‘ −1‘

: Hence, for any r≥0;

P_π’t_n/n≥γ“ ≤exp

−nγr+ 4b²

na²1−⑐coshrna‘ −1‘

: Differentiation shows that

r= na‘⁻¹arcsinhnaγ1−⑐4b²‘⁻¹‘ x= na‘⁻¹θ

minimizes the right-hand side of the above inequality. It suffices to show coshθ−1≤ θsinhθ‘/2:But the above inequality can easily be deduced from

∀x∈R; e^x2−x‘ +e^−x2+x‘ ≤4:

Another proof, due to Ledoux, can be developed for continuous-time Markov chain under the weaker assumption that the functionfis bounded, measur- able and hypothesis (14) is fulfilled. LetY_t=Rt

0fX_s‘ds; then, for all k≥1, Y^k_t =k!Z^t

0

ds₁Z^t

s₁ds₂· · ·Z^t

s_k−1fX_s1‘fX_s2‘ · · ·fX_sk‘ds_k; E_πY^k_t =k!Z^t

0

du₁Z^t−u1

0

du₂· · · Z^t−u1+···+u_k−1‘

0

du_kZ

fP_u2f· · · fP_ukf‘ · · ·‘dπ:

Let a_2;:::;k= ŽR

fP_u2f· · · fP_ukf‘ · · ·‘dπŽ; then, using (14) and the centering method gives the inequality

a_2;:::;k≤ f²₂

exp−u₂+ · · · +u_k‘λ‘ +a₂exp−u₄+ · · · +u_k‘λ‘

+ · · · +a_2;:::;k−2exp−u_kλ‘

: An iteration of this inequality shows that a_2;:::;k is bounded by a sum of at most 2^k−1 of such terms:b²‘^s+1exp−m₂+ · · · +m_k‘λ‘; wherem_i =0 oru_i andsis the number ofm_i 6=u_i with 0≤s≤ ˆk/2‰ −1:This implies

E_πexprY_t‘ ≤1+

∞

X

k=2

r^k2^k−1

ˆk/2‰

X

s=1

b²t‘^s s! λ^s−k

≤1+ 2−4r/λ‘⁻¹

∞

X

s=1

4b²r²t/λ‘^s s! ;

so long as 2r/λ <1:For 2r/λ≤1/2;this yieldsE_πexprY_t‘ ≤exp’4b²tr²λ⁻¹“:

Optimizing in r≥ λ/4 together with Markov’s inequality, it follows that, for every 0≤γ≤2b²;

P_π 1

t Z^t

0 fX_s‘ds≥γ

≤exp

−γ²tλ 16b²

:

5. Some reflections on simulation. This last section shows how Theo- rem 1.1 can be combined with quantification of the closeness to stationarity, in order to compute the simulation time required to obtain a sufficiently ac- curate estimate. More precisely, we will consider the algorithm suggested by Aldous (1987).

The constantN_q. As it has already been said before, the coefficientN_q=

q/π₂may be large but always bounded byπ∗^−1/2;whereπ_∗=min”πy‘xy∈ G•: Furthermore,N_q =1 when q = π: This fact suggests that it might be better to begin the computation oftnonly when the distribution of the Markov chain is close toπ:For instance, if the Markov chain is ergodic (i.e., irreducible and aperiodic), the distribution π_n of X_n converges to π exponentially asn tends to infinity. In that case, the simulation can be split into two phases.

During the first one, the Markov chain approaches the stationary distribution sufficiently close to eliminate the “bias” from the initial positionX₀;while the second one is the period of time necessary beforet_n is a reasonable estimate of πf:

Letπ_n be the distribution ofX_nfor the initial distribution q;and N²_n= π_n/π²₂=X

x

πx‘

X

y

qy‘Pⁿy; x‘/πx‘

2

:

Then, the time until the distribution is close to stationary can be formalized by a parameterτ;defined as

τ=min

nxN_n≤p

1+1/e² =min”nx π_n/π‘ −1₂≤1/e•:

From now on, we consider reversible and aperiodic Markov chains. Instead of working with Nn; we will use the chi-square distance πn/π‘ −1₂: We have

π_n/π‘ −1²₂≤ X

x; y

πx‘qy‘Pⁿy; x‘/πx‘ −1‘²

≤sup

y Pⁿy;·‘/𐷑 −1²= Pⁿ−π²_2→∞

≤ Pⁿ¹²_2→∞Pⁿ²−E_π²_2→2;

where the first inequality follows from the Cauchy–Schwarz inequality and n₁+n₂=n:From the last inequality, we deduce

πn/π‘ −1₂≤ min

n₁+n₂=nDn₁‘Pⁿ²−Eπ_2→2

≤ 1/π_∗−1‘^1/2µn‘;

(15)

whereDn₁‘ = Pⁿ¹_2→∞ andµn‘ = Pⁿ−E_π_2→2:Finally, an upper bound forµn‘is accomplished by using the inequalityµn‘ ≤β_∗P‘ⁿ;where

β_∗P‘ =max”Žβ_ŽGŽ−1P‘Ž; β₁P‘•:

See Diaconis and Saloff-Coste (1993) and Fill (1991) for more details. Combin- ing this inequality with (15) gives

π_n/π‘ −1₂≤ 1/π_∗−1‘^1/2β_∗P‘ⁿ: (16)

We are interested in determining the sufficient total number of steps for the simulation,n_e=τ+n; such that

P_q

n⁻¹

τ+n

X

i=τ+1

fX_i‘

≥γ

≤α;

(17)

where α < 1 is a small constant such as 0:05; for instance. Then with the setup of Theorem 1.1, the inequality (2) stated in Remark 3 gives

P_q

n⁻¹

τ+n

X

i=τ+1

fX_i‘

≥γ

≤2 exp’−nγ²εP‘/12“:

(18)

Moreover, different techniques can be used to get a bound onτ;such as eigen- values and geometric techniques or coupling [see Aldous and Fill, Fill (1991), Diaconis and Saloff-Coste (1996a, b)]. For instance, Poincar´e inequality gives upper bound forµn‘[see Diaconis and Saloff-Coste (1996a)], whereas, when applicable, Nash and Log-Sobolev inequalities allow us to bound Dn‘ [see Diaconis and Saloff-Coste (1996b)].

Examples. The following examples deal with the Metropolis algorithm, a widely used tool in simulation. Consider G = ”0;1; : : : ; N• and π a fixed distribution. We would like to construct an irreducible Markov chainMx; y‘

whose stationary distribution is π: The Metropolis algorithm begins with a base chainPx; y‘onGwhich is modified by an auxiliary randomization. We will assume that Pis irreducible and aperiodic and thatPx; y‘>0 implies Py; x‘>0:Let

Ax; y‘ =

πy‘Py; x‘/πx‘Px; y‘‘; ifPx; y‘>0;

0; otherwise.

Formally, the Markov kernel Mis

Mx; y‘ =















Px; y‘; ifAx; y‘ ≥1; x6=y;

Px; y‘Ax; y‘; ifAx; y‘<1;

Px; y‘ + X

zxAx; z‘<1

Px; z‘1−Ax; z‘‘; ifx=y:

We can easily prove that M is reversible and aperiodic. For the present examples, we take the base chain to be the nearest-neighbor random walk

Px; x+1‘ =Px; x−1‘ =1/2; 1≤x≤N−1;

P0;1‘ =P0;0‘ =PN; N−1‘ =PN; N‘ =1/2:

Binomial distribution. The stationary distribution is πx‘ = 2^{−N N}_x : The standard Metropolis construction gives [see Diaconis and Saloff-Coste (1996a)]

Mx; y‘ =























1/2; if

(y=x+1; 0≤x≤ N−1‘/2;

y=x−1; N+1‘/2≤x≤N;

x/2N−x+1‘; ify=x−1; 1≤x≤ N+1‘/2;

N−x‘/2x+1‘; ify=x+1; N−1‘/2≤x≤N−1;

N−2x+1‘/2N−x+1‘; ify=x; 0≤x≤ N+1‘/2;

2x−N+1‘/2x+1‘; ify=x; N−1‘/2≤x≤N:

Here 1/N≤1−β₁M‘ ≤2/N,π_∗=2^−N;andβ_NM‘ ≥ −1+2 minMx; x‘ ≥

−1/2:Furthermore, it is shown in Diaconis and Saloff-Coste (1996a), by using the Log-Sobolev inequality, that

Mⁿx;·‘/π−1 ≤ 1+2e²‘^1/2e^−c for n≥ N/2‘logN+2c‘ +1:

In other words, we find that approximate equilibrium is reached for τ of the order ofNlogN, whereas inequality (16) asserts approximate randomness for τ of orderN²: We thus see, using inequality (2) (i.e., withτ =0), that order

N/γ‘² steps are sufficient for condition (17) to hold, whereas inequality (18) improves the running time, since we get n_e of the order ofNlogN+1/㲑:

Exponential fall-off. In this example, given in Diaconis and Saloff- Coste (1995), the stationary distribution is

πi‘ =za‘a^hi‘; 0< a <1; zthe normalizing constant:

Then, assuming

hi+1‘ −hi‘ ≥c≥1; 0≤i≤N−1;

the Metropolis chain is given by

Mi; i−1‘ =1/2; Mi; i‘ =2⁻¹1−ahi+1‘−hi‘‘;

Mi; i+1‘ =2⁻¹ahi+1‘−hi‘; 1≤i≤N−1;

M0;0‘ =1−2⁻¹a^{h1‘−h0‘}; M0;1‘ =2⁻¹a^{h1‘−h0‘}; MN; N−1‘ =MN; N‘ =1/2:

Here, the second eigenvalue of this chain satisfies β₁≤1−2⁻¹1−a^c/2‘²:

See Diaconis and Saloff-Coste (1995) for the proof based on a Poincar´e in- equality. Moreover, β_N ≥ −1+2 minMx; x‘ ≥ −1+21/2−a^c/2‘ = −a^c: Thus,

β_∗≤max1−2⁻¹1−a^c/2‘²; a^c‘:

(19)

Moreover, inequality (19) says orderNsteps are sufficient to reach station- arity. It follows that, using inequality (18), condition (17) holds forn_e of the order ofN; whereas applying inequality (2) givesn_e of the order ofN/γ²:

Acknowledgments. Dominique Bakry, Laurent Saloff-Coste, Persi Dia- conis and Michel Ledoux were exceptionally helpful with this paper. I would like to thank also the referee and Associate Editor for suggesting a number of improvements.

REFERENCES

Aldous, D.(1987). On the Markov chain simulation method for uniform combinatorial distribu- tions and simulated annealing.Prob. Engrg. Inform. Sci.133–46.

Aldous, D.andFill, J.Reversible Markov chains and random walks on graphs. Unpublished manuscript.

Bahadur, R. R.andRanga Rao, R.(1960). On deviations of the sample mean.Ann. Math. Statist.

311015–1033.

Bennett, G.(1962). Probability inequalities for sums of independent random variables.J. Amer.

Statist. Assoc.5733–45.

Chernoff, H.(1952). A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations.Ann. Math. Statist.23493–507.

Cram ´er, H.(1938). Sur un nouveau th´eor`eme de la th´eorie des probabilit´es.Actualit´es Sci. Indust.

736.

Dembo, A.andZeitouni, O.(1993). Large Deviations Techniques and Applications. Jones and Bartlett, Boston.

Diaconis, P.andSaloff-Coste, L.(1993). Comparison theorems for reversible Markov chains.

Ann. Appl. Probab.3696–730.

Diaconis, P.andSaloff-Coste, L.(1995). What do we know about the Metropolis algorithm?

Unpublished manuscript.

Diaconis, P.andSaloff-Coste, L.(1996a). Logarithmic Sobolev inequalities and finite Markov chains.Ann. Appl. Probab.6695–750.

Diaconis, P.andSaloff-Coste, L.(1996b). Nash inequalities for finite Markov chains.J. Appl.

Probab.9459–510.

Dinwoodie, I.(1995). A probability inequality for the occupation measure of a reversible Markov chain.Ann. Appl. Probab.537–43.

Fill, J. (1991). Eigenvalue bounds on convergence to stationarity for nonreversible Markov chains, with an application to the exclusion process.Ann. Appl. Probab.162–87.

Gillman, D.(1993). Hidden Markov chains: rates of convergence and the complexity of inference.

Ph.D. dissertation, MIT.

Kato, T.(1966). Perturbation Theory for Linear Operators. Springer, New York.

Kolmogorov, A.(1929). ¨Uber das Gesetz des iterieten Logaritmus.Math. Ann.99309–319.

Mann, B. (1996). Berry–Esseen central limit theorem for Markov chains. Ph.D. dissertation, Harvard Univ.

Marshall, A. W.andOlkin, I.(1979).Inequalities: Theory of Majorization and Its Applications.

Academic Press, New York.

Nagaev, S. V.(1957). Some limit theorems for stationary Markov chains.Theory Probab. Appl.

2378–406.

Nagaev, S. V.(1961). More exact statements of limits theorems for homogeneous Markov chains.

Theory Probab. Appl.662–81.

Stout, W. F.(1974). Almost Sure Convergence. Academic Press, New York.

Trotter, H. F.(1959). On the product of semi-groups of operators.Proc. Amer. Math. Soc.10 545–551.

Centre d’Etudes de la Navigation A ´erienne 31055 Toulouse Cedex

and

Lab. Statistique et Probabilit ´e Universit ´e Paul Sabatier 31055 Toulouse Cedex France

E-mail:[email protected]