Stochastic approximation of quasi-stationary distributions on compact spaces and applications

HAL Id: hal-01334603

https://hal.archives-ouvertes.fr/hal-01334603v3

Submitted on 6 Nov 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Stochastic approximation of quasi-stationary distributions on compact spaces and applications

Michel Benaim, Bertrand Cloez, Fabien Panloup

To cite this version:

Michel Benaim, Bertrand Cloez, Fabien Panloup. Stochastic approximation of quasi-stationary distri- butions on compact spaces and applications. Annals of Applied Probability, Institute of Mathematical Statistics (IMS), 2018, 28 (4), pp.2370-2416. �10.1214/17-AAP1360�. �hal-01334603v3�

COMPACT SPACES AND APPLICATIONS

MICHEL BENAIM, BERTRAND CLOEZ, FABIEN PANLOUP

ABSTRACT. As a continuation of a recent paper, dealing with finite Markov chains, this paper proposes and analyzes a recursive algorithm for the approximation of the quasi-stationary distri- bution of a general Markov chain living on a compact metric space killed in finite time. The idea is to run the process until extinction and then to bring it back to life at a position randomly chosen according to the (possibly weighted) empirical occupation measure of its past positions. General conditions are given ensuring the convergence of this measure to the quasi-stationary distribution of the chain. We then apply this method to the numerical approximation of the quasi-stationary distribution of a diffusion process killed on the boundary of a compact set. Finally, the sharpness of the assumptions is illustrated through the study of the algorithm in a non-irreducible setting.

Keywords. Quasi-stationary distributions ; stochastic approximation ; reinforced random walks ; random perturba- tions of dynamical systems; extinction rate; Euler scheme.

AMS-MSC.65C20; 60B12; 60J10, Secondary 34F05; 60J20; 60J60.

1. INTRODUCTION

Numerous models, in ecology and elsewhere, describe the temporal evolution of a system by a Markov process which eventually gets killed in finite time. In population dynamics, for instance, extinction in finite time is a typical effect of finite population sizes. However, when populations are large, extinction usually occurs over very large time scales and the relevant phenomena are given by the behavior of the process conditionally to its non-extinction.

More formally, letpξ_tqtě0 be a Markov process with values inEY tBuwhereE is a metric space andB REdenotes an absorbing point (typically, the extinction set or the complement of a domain). Under appropriate assumptions, there exists a distributionνonE(possibly depending on the initial distribution ofξ) such that

tÑ8limPpξtP.|ξt‰ Bq “νp¨q. (1) Such a distribution well describes the behavior of the process before extinction, and is necessar- ily (see e.g [33]) aquasi-stationary distribution(QSD) in the sense that

PνpξtP ¨|ξt‰ Bq “νp¨q.

We refer the reader to the survey paper [33] or the book [19] for general background and a comprehensive introduction to the subject.

The simulation and numerical approximation of quasi-stationary distributions have received a lot of attention in the recent years and led to the development and analysis of a class of particle systems algorithms known in the literature asFleming-Viot algorithms(see [14, 18, 20, 43]). The principle of these algorithms is to run a large number of particles independently until one is killed

Date: Compiled November 6, 2017.

1

and then to replace the killed particle by an offspring whose location is randomly (and uniformly) chosen among the locations of the other (alive) particles. In the limit of an infinite number of particles, the (spatial) empirical occupation measure of the particles approaches the law of the process conditioned to never be absorbed; see for instance [43, Theorem 1] . Combined with (1), this gives a method for estimating the QSD of the process.

In a related context the new paper [40] demonstrates the importance to simulate QSDs in computational statistics as an alternative approach to classical MCMC simulations.

Recently, in the setting of finite state Markov chains, Benaim and Cloez [6] (see also [12]) ana- lyzed and generalized an alternative approach introduced in [1] in which thespatial occupation measureof a set of particles is replaced by thetemporal occupation measureof a single par- ticle. Each time the particle is killed it is risen at a location randomly chosen according to its temporal occupation measure. The details of the construction are recalled in Section 2.

The objective of this paper is twofold: on one hand, we aim at extending the results of [6] to the setting of Markov chains with values in a general space, being killed when leaving a compact domain. Indeed, up to our knowledge, in all the previous works for this algorithm [1, 6, 12], the state spaceEis finite. On the other hand, we also explore various applications: we propose and investigate a numerical procedure, based on an Euler discretization, for approximating QSD of diffusions.

In contrast with the Fleming-Viot particle system, this algorithm requires less calculus but more memory. Also, it only depends on only one parameter (the time) and then approximates in the same time the conditioned dynamics and its long time limit; in particular, it does not require to calibrate simultaneously the number of particles and the time parameter as in the standard Fleming-Viot approach. Instead, in view of a convergence result for this algorithm, one needs to obtain some properties which are similar to the commutation of the limits of large particles and of the long time for the Fleming-Viot algorithm. For the particle system, this type of problem is not completely solved in general but some results have been obtained in some particular settings;

see for instance [14, 17, 18, 25, 28, 42]. Note that an example where the commutation property does not hold is exhibited in Section 3.1. Besides, let us cite [35], [6, Section 3] or [36] which give three different discrete-time Fleming-Viot type algorithms where the double limit is either not proved or proved under restrictive assumptions. Another difference is that the Fleming-Viot process is often developed in continuous-time although our stochastic approximation scheme is in discrete time. As a consequence it is difficult to compare our assumptions on the transition kernel with the ones of the articles mentioned previously. However, implementing the methods of [14, 28, 42] requires a discretization and then leads to the QSD of an Euler-type sequence instead of the one of the target diffusion process. Theorem 3.9 corroborates the consistence of their methods and also shows the consistence of our algorithm.

Outline. The paper is organized as follows: In Section 2 we detail the general framework, the hypotheses and state our main results. In Section 3, we first discuss our assumptions in the simple case of finite Markov chains and then focus on the application to the numerical approx- imation of QSDs for diffusions (including theoretical results and numerical tests), The sequel of the paper (Sections 4, 5, 6, and 7) is mainly devoted to the proofs and the details about their sequencing will be given at the end of Section 3. We end the paper by some potential extensions of this work to some more general settings such as non-compact domains or continuous-time reinforced strategies.

2. SETTING AND MAINRESULTS

2.1. Notation and Setting. LetE be a compact metric space¹equipped with its Borelσ-field BpEq.Throughout, we let BpE,Rqdenote the set of real valued bounded measurable functions on E andCpE,Rq Ă BpE,Rq the subset of continuous functions. For allf P BpE,Rqwe let }f}8 “ sup_xPE|fpxq| and we let 1 denote the constant map x ÞÑ 1.We let PpEq denote the space of (Borel) probabilities over E equipped with the topology of weak* convergence.

For all µ P PpEq andf P BpE,Rq,orf nonnegative measurable, we writeµpfq (or µf) for ş

Ef dµ.Recall thatµn Ñ µinPpEq provided µnpfq Ñ µpfqfor all f P CpE,Rq,and that (by compactness ofE and Prohorov Theorem), PpEq is a compact metric space (see e.g [21, Chapter 11]).

A sub-Markovian kernel on E is a map Q : E ˆBpEq ÞÑ r0,1ssuch that for all x P E, A ÞÑ Qpx, Aq is a nonzero measure (i.eQpx,Eq ą 0) and for allA PBpEq, x ÞÑ Qpx, Aqis measurable. If furthermoreQpx,Eq “1for allxPE,thenQis called aMarkov (or Markovian) kernel.

Let Q be a sub-Markovian (respectively Markovian) kernel. For every f P BpE,Rq and µPPpEq,we letQfandµQrespectively denote the map and measure defined by

Qfpxq “ ż

E

fpyqQpx, dyq and µQp¨q “ ż

E

µpdxqQpx,¨q.

IfQf P CpE,Rqwheneverf P CpE,Rq,thenQis said to beFeller. For alln P N,we letQⁿ denote the sub-Markovian (respectively Markovian) kernel recursively defined by

Q^n`1px,¨q “ ż

E

Qpy,¨qQⁿpx, dyqandQ⁰px,¨q “δx.

A probability µP PpEqis called aquasi-stationary distribution(QSD) forQifµandµQare proportional or, equivalently, if,

µ“ µQ µQ1. The number

Θpµq:“µQ1 (2)

is called theextinction rateofµ.

Note that whenQis Markovian, a quasi stationary distribution isstationary(orinvariant) in the sense thatµ“µQ.In this caseΘpµq “1, otherwiseΘpµq ă1.

From now on and throughout the remainder of the paper we assume given a Feller sub- Markovian kernelK onE.

LetB REbe acemetery point. Associated toK is the Markov kernelKs onEY tBudefined, for allxPE, APBpEq,by

$

&

%

Kpx, Aq “s Kpx, Aq,

Kpx,s tBuq “1´Kpx,Eq, and KpB,s tBuq “1.

(3) The kernelKs can be understood as the transition kernel of a Markov chainpY_nqně0onEY tBu whose transitions inEare given byKand which is "killed" when it leavesE.

1For comments about a possible extension to the non-compact case, see Section 8

Letδ:E ÞÑ r0,1sbe the function defined by δ “1´K1.

That is, for everyxPE,

δpxq “Kpx,s tBuq “1´Kpx,Eq. (4) For a givenµPPpEq,we letKµdenote the Markov kernel onEdefined by

K_µpx, Aq “Kpx, Aq `δpxqµpAq for allxPE andAPBpEq.Equivalently, for everyf PBpE,Rq, K<sub>µ</sub>fpxq “Kfpxq `δpxqµpfq.

The chain induced by Kµ behaves like pYnq until it is killed and then is redistributed in E according toµ.Note thatK_µinherits the Feller continuity fromK. For the sequel, an important feature ofKµis thatµis a QSD forKif and only if it is invariant forKµ(see Lemma 4.3 for details).

LetpΩ,F,Pqbe a probability space equipped with a filtrationtF_nuně0(i.e an increasing fam- ily ofσ-fields). We now consider anE-valued random processpXnqně0 defined on pΩ,F, Pq adapted totF_nuně0such that

X₀“xPE and@ně0, PpX_n`1Pdy|F_nq “K_µnpX_n, dyq, (5) where

µ_n“ ř_n

k“0η_kδ_Xk ř_n

k“0η_k (6)

is aweighted occupation measure. Herepη_nqně0 is a sequence of positive numbers satisfying certain conditions that will be specified below (see Hypothesis 2.1).

With the definition ofK_µ, this means that whenever the original processpY_nqně0is killed, it is redistributed inEaccording to its weighted empirical occupation measureµn. Note that such a process is a type ofreinforced random walk (see e.g [38]). It is reminiscent ofinteracting particle systems algorithms used for the simulation of QSDs such as the so-called Fleming- Viot algorithm(see [14, 18, 43] and [6, Section 3]). However, while these latter algorithms involve a large number of particles whose individual dynamics depend on thespatial occupation measureof the particles, here there is a single particle whose dynamics depends on itstemporal occupation measure. From a simulation point of view, this is of potential interest, suggesting fewer computations (but more memory) and leading to a recursive method which avoids (at least in name) the trade-off between the number of particles and the time horizon induced by Fleming-Viot algorithm.

Set, forně0,

γn“ ηn

ř_n

k“0η_k.

The occupation measure can then be computed recursively as follows:

µn`1 “ p1´γn`1qµn`γn`1δXn`1. (7) Under appropriate irreducibility assumptions (see Hypothesis 2.2 below),K_µadmits a unique invariant probabilityΠ_µ.Owing to the above characterization of QSDs as fixed points ofµÞÑ Π_µ, we choose to rewrite the evolution ofpµnqas:

µ<sub>n`1</sub> “µ<sub>n</sub>`γ_{n`1</sub>p´µ<sub>n</sub>`Π_µnq `γ<sub>n`1}ε_n (8)

whereε_n “δ_Xn`1 ´Π_µn. The processpµ_nqis therefore astochastic approximation algorithm associated to the ordinary differential equation (ODE) (for which rigorous sense will be given in Section 5):

µ9 “ ´µ`Π_µ. (9)

The almost sure convergence of pµ_nq towards µ^‹ (the QSD of K) will then be achieved by proving that :

(i) The asymptotic dynamics ofpµ_nqně0matches with that of solutions of the above ODE:

more precisely,pµnqně0 is (at a different scale) anasymptotic pseudo-trajectoryof the ODE (in the sense of Benaim and Hirsch [7], see [5] for background).

(ii) The setFixpΠq “ tµ P PpEq, µ “ Π_µu reduces toµ^‹ and is a global attractor of the ODE.

This strategy was applied in [6] whenE is a finite set. However, the proofs in [6] strongly rely on finite dimensional arguments that cannot be applied in this more general setting and the new proofs will require a careful study of the kernel familypK_µqµ.

2.2. Main results. We first summarize the standing assumptions under which our main results will be proved. We begin by the assumptions onpγ_nqně1.

Hypothesis 2.1(Standing assumption onpγ_nq). The sequence pγ_nqně0 appearing in equation (7) is a non-increasing sequence such that

ÿ

ně0

γn“ `8 and lim

nÑ`8γnlnpnq “0. (10) The typical sequence is given byγ<sub>n</sub>“ <sub>n`1</sub>¹ , which corresponds toη_n“1for allně1.

Now, let us focus on the assumptions on the sub-Markovian kernelK. We say that a non- empty setAPBpEq Y tBuisaccessibleif for allxPE

ÿ

ně1

Ksⁿpx, Aq ą0.

It is called aweak²small setifA ĂE and there exists a probability measureΨonE andą0 such that for allxPA

ÿ

ně1

Kⁿpx, dyq ěΨpdyq. (11)

Hypothesis 2.2(Standing assumptions onK).

‚ pH₁q Kis Feller.

‚ pH₂qThe cemetery pointtBuis accessible.

Assumptions H₁ andH₂ imply the existence of a quasi-stationary distribution but are not sufficient to ensure its uniqueness (see the example developed in Subsection 3.1). For this, we require the supplementary assumptions below

Hypothesis 2.3(Additional assumptions onK).

‚ pH₃qThere exists an open accessible weak small setU.

2this is a mildly weaker definition than the usual definition of small or petite sets (see e.g [22, 34])

‚ pH₄qThere exists a non increasing convex functionC:R^{`</sup>ÞÑR<sup>`}satisfying ż₈

0

Cpsqds“ 8 (12)

such that

ΨpKⁿ1q

sup_xPEKⁿ1pxq ěCpnq whereΨsatisfies equation (11).

Roughly, the latter hypothesis stipulates that the rate at which the process dies is uniformly controlled, in terms of the initial point. This is motivated by the recent work of Champagnat and Villemonais [16] in which it is proved that under mildly stronger versions ofH₃ (namely, K^lpx,q ě9 Ψfor somelindependent ofx) and H₄ (namelyCptq ě c ą 0) the sequence of conditioned laws defined by

PxpYnP ¨|YnPEq “ Kⁿpx,¨q

Kⁿ1pxq, ně0,

converges, as n Ñ 8, exponentially fast to a (unique) QSD. Here, Assumption H₄ which does not require the functions ÞÑ Cpsqto be lower-bounded does certainly not guarantee the exponential rate but is a sharper and almost necessary assumption for the uniqueness and the attractivenessof the QSD (on this topic, see also Remark 2.4 below and Proposition 3.3). More precisely, it will be shown that under H₃ and H₄, the semiflow induced by (9) is globally asymptotically stable (i.eFixpΠqis a singleton and is a global attractor).

Remark 2.4(Sufficient condition). A simple condition ensuring Hypothesis 2.3 is that, for some lě1,constantsc1, c2 ą0and someΨPPpEq

c₁Ψpdyq ďK<sup>`</sup>px, dyq ďc<sub>2</sub>Ψpdyq. (13) Indeed, under (13), forně`,

c₂ΨpK^{n´`</sup>1q ěK<sup>n</sup>1ěc<sub>1</sub>ΨpK<sup>n´`}1q while forn ď `,1 ě K<sup>n</sup>1 ě K<sup>`</sup>1 ě c₁.HenceCptq “ min

ˆc₁ c₂, c₁

˙

ą0.Note that (13), which is usual in the literature (seee.g.[9, Theorem 3.2]), is satisfied ifK^`admits a continuous and positive density with respect to a positive reference measure.

Finally, note that in Hypotheses 2.2 and 2.3 there is no aperiodicity assumption onK.

We are now able to state our main general result about the convergence of the empirical measurepµnqně0towards the QSD.

Theorem 2.5(Convergence of the algorithm). Assume Hypotheses 2.1, 2.2 and 2.3. Then, K has a unique QSDµ^‹and the sequencepµ_nqně0defined by(6)convergesa.s.inPpEqtowards µ^‹.

In fact, the previous setting also leads to the convergence in distribution of the reinforced random walk:

Theorem 2.6(Convergence in distribution ofpX_nqně0). Suppose that the assumptions of The- orem 2.5 hold. Then, for any starting distribution α, pXnqně0 defined by (5) converges in distribution toµ^‹.

The two above results thus show that the algorithm both produces a way to approximateµ^‹ and also to sample a random variable with this distribution. The convergence in law ofpXnqně0

may appear surprising due to the lack of aperiodicity assumption forpYnqně0. To overcome this problem, we prove in fact thatpX_nqně0gets asymptotically this property.

The extinction rateΘpµ^‹q, defined in (2), can be estimated through the same procedure. For this, we need to keep track of the times at which a "resurrection" occurs. We then constructpX_nq as follows. LetppUn, Xnqqbe a process adapted totF_nu,withUnPEY tBu, XnPE,satisfying X₀“U₀“x,

PpU_n`1Pdy|F_nq “KpXs _n, dyq and

X_{n`1</sub> “V<sub>n`1}1<sub>tUn`1“Bu</sub>`U_{n`1</sub>1<sub>tUn`1PEu},

where pV_nq is a sequence of independent variables such that V<sub>n`1</sub> „ µ<sub>n</sub>, conditionally on σpF<sub>n</sub>, Un`1q. Clearly,pXnqsatisfies (5) and the times at whichUn “ Bare the "resurrection"

times (at whichX_nis redistributed).

Theorem 2.7 (Extinction rate estimation). Suppose that the assumptions of Theorem 2.5 are satisfied. Then,

θs_n:“ 1 n

ÿn k“1

1_tU

k“Bu

ÝÝÝÝÑnÑ`8 1´Θpµ^‹q.

Proof. SincePpUn`1“ B|F_nq “δpXnq, we can decomposeθsnas θsn“ Mn

n `µnpδq wherepMnqis the martingale defined byMn “ ř_n

k“1p1_tUk“Bu´δpXk´1qq.Since the incre- ments ofpMnqare uniformly bounded,xMynďCnand it follows from the strong law of large numbers for martingales, that ^M_nⁿ Ñ 0a.s. asn Ñ `8. On the other hand,µnpδq ÝÝÝÝÑ<sup>nÑ`8</sup>

µ^‹pδq “1´Θpµ^‹qa.s.This ends the proof.

3. EXAMPLES AND APPLICATIONS

3.1. Finite Markov Chains. In this entire subsection, we consider the simple situation where Eis a finite set in order to discuss our main assumptions.

We use the notation

Kpx, yq “Kpx,tyuq, @x, yPE.

We say that x leads to y, written x ãÑ y, if ř

ně0Kⁿpx, yq ą 0. IfA, B Ă E we write AãÑBwhenever there existxPAandyPBsuch thatxãÑy.

KernelKis calledindecomposableif there existsx₀ PEsuch thatxãÑx₀for allxPE,and irreducibleifxãÑyfor allx, yPE.

Note that HypothesisH₁ is automatically satisfied (endowEwith the discrete topology) and thatH₃is equivalent to indecomposability (chooseU “ tx0uandΨ“δx0). From now on, we investigate separately the irreducible and non-irreducible cases.

Irreducible setting. When K is irreducible HypothesisH₄ holds withCpnq “ c ą 0.This follows from the following lemma.

Lemma 3.1. There existscą0such that

Kⁿ1pxq “Kⁿpx,Eq ěcKⁿpy,Eq “cKⁿ1pyq for allnPNandx, yPE such thatxãÑy.

Proof. Letc1“mintKpx, yq:Kpx, yq ą0u.IfxãÑy,then the path which linksxandyhas at most|E| ´1steps and hence,

Dry P t1, . . . ,|E| ´1u such that K^rypx, yq ěc^r₁^y ěc^|E|´1₁ . From the relation

Kⁿpx,Eq ěK^n`rypx,Eq “ ÿ

y

K^rypx, yqKⁿpy,Eq it comes that

Kⁿpx,Eq ěc^|E|´1₁ Kⁿpy,Eq.

This proves the result withc“c^|E|´1₁ .

As a consequence, except for the rate of convergence, we retrieve [6, Theorem 1.2] (see also [1, 12] for the convergence result in the caseγn“ _n`1¹ ).

Theorem 3.2. SupposeK is irreducible andKpx₀,Eq ă 1for some x₀ P E.ThenK has a unique QSDµ^‹and under Hypothesis 2.1,pµnqconverges almost surely toµ^‹.

Bottleneck effect and conditionH₄. Here we discuss an example demonstrating the necessity of conditionH₄for non irreducible chains. Note that this example can also be understood as a benchmark of more general processes admitting several QSDs such as general indecomposable Markov chains.

SupposeE “E₁YE₂ whereE₁ andE₂are nonempty disjoint sets such that (1) @x, yPE_ixãÑy;

(2) E₁ ãÑE₂; (3) E₂ ­ãÑE₁;

(4) E₂ ãÑ B,(that isDxPE₂Kpx,Eq ă1) andE₁­ãÑ B.

LetKibe the kernelKrestricted toE_i.That is

Ki “ pKpx, yqqx,yPEi.

Letµ^‹_i be the (unique) QSD of Ki andΘ_i the associated extinction rate. Note that, by irre- ducibility ofK_i, and the Perron Frobenius Theorem,Θ_i is nothing but the spectral radius of Ki.

We considerµ^‹_i as an element ofPpEq by identifyingPpE_iqwith the set ofµ P PpEq sup- ported byE_i.

As previously noticed,H₁,H₂andH₃are always true. However, assumptionH₄might fail to hold. More precisely, we have the following result.

Proposition 3.3(Sharpness ofH₄). ConditionH₄ holds if and only ifΘ₁ ďΘ₂.In this case the unique QSD ofKisµ^‹₂and, under Hypothesis 2.1µ_nѵ^‹₂.

Proof. Fixx₀ PE₂ and letΨ“δ_x0.Hence,H₁,H₂ andH₃hold. By Lemma 3.1 there exists c ą 0 such thatΨpKⁿ1q “ Kⁿ1px0q ě cKⁿ1pxq for all x P E₂ andn ě 0. ThusH₄ is equivalent to

@xPE₁, Kⁿ1px₀q ěCpnqKⁿ1pxq (14) withCsatisfying (12). Letτ₁ “mintně0 :Y_n RE₁uandτ₂ “mintně0 : Y_n “ Bu.By Lemma 3.1 applied to each of the kernelKi, and from the relationΘⁿ_i “µiK_iⁿ1_Ei,we get that for allxPE_i

1

cΘⁿ_i ěPxpτi ąnq ěcΘⁿ_i (15) for somecą0.Thus for allxPE₁,

Pxpτ₂ąnq “ExrPpτ₂ ąn|F_τ1qs “Ex

“

PYτ1pτ₂ ąn´τ₁q‰ ď 1

cEx

“Θ^n´τ₂ ¹1_τ1ďn`1_τ1ąn‰ by (15). Thus,

Pxpτ₂ąnq ď 1 cΘⁿ₂

ÿn k“1

Θ^´k₂ pPxpτ₁ ąk´1q ´Pxpτ₁ ąkqq ` 1

cPxpτ₁ ąnq

“ 1

cΘⁿ₂pΘ^´1₂ `

n´1

ÿ

k“1

pΘ^´k´1₂ ´Θ^´k₂ qPxpτ1 ąkqq

“ 1

cΘ^n´1₂ p1` p1´Θ₂q

n´1

ÿ

k“1

Θ^´k₂ Pxpτ1ąkqq Then, by (15) again, we get

Pxpτ2 ąnq ďOpΘ^n´1₂ q “OpPx0pτ2 ąnqq whenΘ₁ ăΘ₂and

Pxpτ₂ąnq ďOpΘ^n´1₂ p1`nqq “OpP<sub>x0</sub>pτ<sub>2</sub> ąnqp1`nqq

when Θ₂ “ Θ₁.This proves that (14) holds with Cptq “ C when Θ₁ ă Θ₂ and Cptq “ C{p1`tqwhenΘ₁ “ Θ₂.If nowΘ₁ ą Θ₂,it follows from Theorem 3.4 below that another QSDµ^‹‰µ^‹₂exists, henceH₄fails.

ForΘ₁ ďΘ₂, µ^‹₂is a global attractor of the dynamics induced by (9), but whenΘ₁ exceeds Θ₂atranscritical bifurcationoccurs:µ^‹₂becomes a saddle point whose stable manifold isPpE₂q, while there is another linearly stable pointµ^‹whose basin of attraction isPpEqzPpE₂q.

This behavior will be shown in section 7 and combined with standard techniques from sto- chastic approximation, it will be used to prove the following result.

Theorem 3.4(Behavior of the algorithm without Assumption H₄). SupposeΘ₁ ą Θ₂.Then there is another QSDµ^‹ having full support (i.e µ^‹pxq ą 0for all x P E). Under Hypothesis 2.1,

(i) pµ_nqně0 converges almost surely toµ₈P tµ^‹₂, µ^‹u.

(ii) IfX0 PE₂, XnPE₂for allnandµ8 “µ^‹₂with probability one.

(iii) IfX₀ PE₁,the eventtµ₈ “µ^‹uhas positive probability.

(iv) Ifř

n

ś_n

i“1p1´γ_iq ă `8, the eventtDN PN:X_nPE₂ for allněNuhas positive probability, and on this eventµ8“µ^‹₂.

Example 3.5(Two points space). The previous results are in particular adapted to the case where E_i “ tiu, i“1,2and

K “

ˆ a 1´a 0 b

˙

witha, bP p0,1q. WriteµPPpEqasµ“ px,1´xq,0ďxď1.Then K_µ“

ˆ a 1´a

p1´bqx b` p1´bqp1´xq

˙

and the ODE (9) writes

x9 “ ´x` p1´bqx

p1´aq ` p1´bqx. (16) In this case, one can check that Θ₁ “ a andΘ₂ “ b, µ^‹₂ “ δ2 and when a ą b, µ^‹ “

a´b

1´bδ₁`^1´a_1´bδ₂. In Figure 1, for a fixed value ofb, we draw the phase portrait of the ODE (16) in terms ofaand especially the bifurcation which appears whenaąb.

FIGURE 1. Transcritical bifurcation associated to Equation (16); b “ 1{3, Continuous line:aÞѵ^‹₂p1q, dotted line: aÞѵ^‹p1q.

Remark 3.6(Open problem). Supposeγ_n“ ^A_n.Althoughµ^‹₂is a saddle point whenΘ₁ ąΘ₂, Theorem 3.4 shows that the eventµnѵ^‹₂has positive probability whenAą1.A challenging question would be to prove (or disprove) that this event has zero probability whenA ď1.This is reminiscent of the situation thoroughly analyzed for two-armed bandit problems in [29, 30].

Remark 3.7(Conditioned dynamics). Note that by mimicking the proof of Lemma 7.2 below, one is also able to compute the limit of the conditioned dynamics:

nÑ8lim PypYnP ¨|YnPEq “ lim

nÑ8

Kⁿpy,¨q Kⁿ1pyq “ν,

whereν “ µ^‹₂ ifΘ₁ ď Θ₂ ory P E₂ andν “ µ^‹ whenΘ₁ ą Θ₂ andy P E₁. Furthermore, at least for Example 3.5 above, it is worth noting that the convergence is not exponential when Θ₁ “Θ₂.

Remark 3.8(Fleming-Viot algorithm). Theorem 3.4 shows that, with positive probability, our algorithm asymptotically matches with the behavior of the dynamics conditioned to the non- absorption. Surprisingly, this is not the case for the discrete-time (or continuous-time) Fleming- Viot particle system (see [6, Section 3], for the definition) which always converges toµ^‹₂. Actu- ally, let us recall that this algorithm has two parameters: the (current) timetě0and the number of particlesN ě1. When the number of particles goes to infinity, it is known that the empirical measure (induced by the particles at a fixed time) converges to the laws conditioned to not be killed; see for instance [18, 43] in the continuous-time setting. However, if we keep constant the number of particles and let first the timettend to infinity then, one obtains the convergence toµ^‹₂ in place ofµ^‹. This comes from the fact that the state where all the particles are inE₂ is absorbing and accessible. In this case, the commutation of the limits established in [6, Section 3] fails. Finally, note that the study of the rate of convergence of Fleming-Viot processes in a two-points space is investigated in [17].

3.2. Approximation of QSD of diffusions. A potential application of this work is to generate a way to simulate QSD of continuous-time Markov dynamics. To this end, the natural idea is to apply the procedure to a discretized version (Euler scheme in the sequel) of the process. Here, we focus on the case of non-degenerate diffusionspξtqtě0 inR^dkilled when leaving a bounded connected open setD. More precisely, letpξ_tqtě0 be the unique solution to thed-dimensional SDE

dξ_t“bpξ_tqdt`σpξ_tqdW_t, ξ₀ PD,

wherebandσ are defined onR^dwith values inR^dandMd,drespectively. One assumes below that the diffusion is uniformly elliptic and thatbandσbelong toC²pR^dq(see Remark 3.10).

For a given stephą0, we denote bypξ_t^hqtě0, the stepwise constant Euler scheme defined by ξ₀ “yPD,

@nPN, ξ_{pn`1qh</sub><sup>h</sup> “ξ<sub>nh</sub><sup>h</sup> `hbpξ_nh^h q `σpξ<sub>nh</sub><sup>h</sup> qpW<sub>pn`1qh}´W_nhq,

and for all t P rnh,pn`1qhq, ξ^h_t “ ξ^h_nh. Under the ellipticity assumption on the diffusion, the Markov chain pY_n :“ ξ^h_nhqn satisfies the assumptions of Theorem 2.5 (with E “ D) (ins particular (13)) and thus admits a unique QSD that we denote byµ^‹_h.

This QSD can be approximated through the procedure defined above and the natural question is: doespµ^‹_hqhconverge toµ^‹whenh Ñ0, whereµ^‹denotes the unique QSD ofpξtqtě0killed when leavingD? A positive answer is given below.

Theorem 3.9 (Euler scheme approximation). Assume that pξtqtě0 is a uniformly elliptic dif- fusion and that D is a bounded domain (i.e. connected open set) with C³-boundary. Then, ppξtqtě0,BDqadmits a unique QSDµ^‹andpµ^‹_hqhą0converges weakly toµ^‹whenhÑ0.

Remark 3.10 (Smoothness assumptions). The uniqueness of µ^‹ is given by Theorem 5.5 of Chapter3in[39](see also[27, Theorem (C)]). Also note that in the proof of the above theorem, one makes use of some results of [26] about the discretization of killed diffusions. The C²- assumption onbandσis adapted to the setting of these papers but could be probably relaxed in our context.

We propose to illustrate the previous results by some simulations. We consider an Ornstein- Uhlenbeck process

dξt“ ´ξtdt`dBt, tě0

killed outside an intervalra, bsand thus compute the sequencepµ^h_nqně1with steph.

We will assume thata“0andb“3. In Figure 2, we represent on the left the approximated density ofµ^h_n(obtained by a convolution with a Gaussian kernel) for a fixed value ofhand dif- ferent values ofn. Then, on the right,nis fixedpn“10⁷qandhdecreases to0. Unfortunately,

FIGURE 2. Left: Approximated density of pµ^h_nq with h “ 0.01 and n “ 5.10⁴,10⁵,10⁶ (green, blue, red) Right: Comparison of µ^‹_h for h “ 0.05,0.01,0.001, (red, orange, blue) withµ^‹(red, dotted-line)

even though the convergence innseems to be fast, the convergence ofµ^‹_h towards µ^‹ is very slow: the discretization of the problem underestimates the probability to be killed between two discretization times. The slow convergence means in fact that this probability decreases slowly to0withh.

However, it is now well-known that, under some conditions on the domain and/or on the di- mension, it is possible to compute a sharp estimate of this probability. More precisely, letpξr_t^hqt

denote the refined continuous-time Euler schemeξr_nh^h “ξ_nh^h and for alltP rnh,pn`1qhq, ξr<sub>t</sub><sup>h</sup> “ξr<sub>nh</sub><sup>h</sup> ` pt´nhqbpξr_nh^h q `σpξr_nh^h qpWt´Wnhq.

It can be shown that L

´

pξr^h_tqtPrnh,pn`1qhs|ξr<sub>nh</sub><sup>h</sup> “x,ξr<sub>pn`1qh</sub>^h “y

¯

“L ˆ

x`t´nh

h py´xq `σpξr^h_nhqB^h_t

˙

where for a givenT ą 0, B^T denotes the Brownian Bridge on the intervalr0, Tsdefined by:

B^T_t “W_t´_T^tW_T. In dimension1, the law of the infimum and the supremum of the Brownian Bridge can be computed (see [26] for details and a discussion about higher dimension). One has for everyzěmaxpx, yq,

P

˜ sup

tPr0,Ts

ˆ

x` py´xqt

T `λB^T

˙ ďz

¸

“1´exp ˆ

´ 2

T λ²pz´xqpz´yq

˙ .

Thus, this means that at each stepn, ifξ_pn`1qhPD, one can compute, with the help of the above properties, a Bernoulli random variableV with parameter

p“PpDtP pnh,pn`1qhq, ξ<sup>h</sup><sub>t</sub> PD<sup>c</sup>|ξ<sub>nh</sub>“x, ξ<sub>pn`1qh</sub> “yq (IfV “1, the particle is killed).

This refined algorithm has been tested numerically and illustrated in Figure 3.

0 0.5 1 1.5 2 2.5 3

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

FIGURE 3. Approximation of µ^‹_h with the Brownian Bridge method forh “ 0.1(blue) compared with the reference density (red, dotted-line)

Remark 3.11. Here, the effect of the Brownian Bridge method is only considered from a numer- ical viewpoint. The theoretical consequences on the rate of convergence are outside of the scope of this paper. Also remark that in order to get only one asymptotic for the algorithm, it would be natural to replace the constant stephby a decreasing sequence as in[31, 37]. Once again, such a theoretical extension is left to a future work.

Outline of the proofs. In Section 4, we begin by some preliminaries: the starting point is to show that the QSD is a fixed point for the applicationµÞÑΠ_µ(onPpEq) whereΠ_µdenotes the invariant distribution ofK_µ(see Lemma 4.3 below). Then, in order to give a rigorous sense to the ODE (9), we prove that this application is Lipschitz continuous for the total variation norm (Proposition 4.5) by taking advantage of the exponential ergodicity of the transition kernelK_µ and the control of the exit timeτ (see Lemma 4.1 and Lemma 4.4 ). In Section 5, we define the solution of the ODE and prove its global asymptotic stability. In Section 6, we then show that (a scaled version of)pµ_nqně0is an asymptotic pseudo-trajectory for the ODE. The proofs of Theorems 2.5 and 2.6 are finally achieved at the beginning of Section 7. In this section, we also prove the main results of Section 3: Theorems 3.4 and 3.9. We end the paper by some possible extensions of our present work.

4. PRELIMINARIES

We begin the proof by a series of preliminary lemmas. The first one provides uniform esti- mates on the extinction time

τ “mintně0 :Yn“ Bu (17) wherepYnqně0is a Markov chain with transitionKs defined in (3).

Lemma 4.1(Expectation of the extinction time). AssumeH₁andH₂.Then (i) There existN PNandδ0 ą0such that for allxPE,Ks^Npx,tBuq ěδ0. (ii)

sup

xPEExrτs ă `8.

Proof. (i)ByH₁ the mapx ÞÑ Ks^Npx,Bq “1´K^N1pxqis continuous onE.It then suffices to show that there existsN PNsuch thatKs^Npx,Bq ą 0for allx PE. Suppose to the contrary that for allN P Nthere existsx_N P E such thatKs^Npx_N,Bq “ 0.HenceKs^kpx_N,Bq “ 0for allkďN.By compactness ofE, we can always assume (by replacingpx_Nqby a subsequence) thatx_N Ñ x^˚ P E.ThusKs^kpx_N,Bq Ý^NÑ8ÝÝÝÑ Ks^kpx^˚,Bq “ 0 for all k P N.This leads to a contradiction with assumptionH₂.

(ii)LetN andδ₀be like inpiq.By the Markov property, for allkPN^˚ Pxpτ ąkNq “Ex

”

PY_pk´1qNpτ ąNq1_τąpk´1qN ı

ď p1´δ0qPxpτ ą pk´1qNq. (18) Thus, for allkPN

Pxpτ ąkNq ď p1´δ0q^k and, consequently,

1

NExrτs ď 1 δ₀ `1.

Remark 4.2. Note that(18)leads in fact to the following statement: there existsλ ą 0such thatsup_xPEEre^λτs ă `8.

The following lemma is reminiscent of the approach developed in [24] for Markov chains on the positive integers killed at the origin and in [4] for diffusions killed on the boundary of a domain.

Lemma 4.3(Invariant distributions and QSD). AssumeH₁. Then,

(i) For everyµPPpEq,K_µis a Feller kernel and admits at least one invariant probability.

(ii) A probabilityµ^‹ is a QSD forKif and only if it is an invariant probability ofKµ^‹. (iii) Assume that for every µ, K_µ has a unique invariant probability Π_µ. Then µ ÞÑ Π_µ

is continuous in PpEq(i.e for the topology of weak convergence) and then there exists µ^‹ PPpEqsuch thatµ^‹ “Π_µ‹ or, equivalently, a QSDµ^‹forK.

Proof. (i)The Feller property is obvious underH₁ and it is well known that a Feller Markov chain on a compact space has an invariant probability (since any weak limit of the sequence p_n¹ř_n

k“1νK_µⁿqně0is an invariant probability).

(ii)Sinceδ“1´K1, for everyAPBpEq, we have

µ^‹pAq “ pµ^‹Kµ^‹qpAq ôµ^‹pAq “ pµ^‹KqpAq pµ^‹Kq1 .

But, by definitionµ^‹is a QSD if and only if the right-hand side is satisfied for everyAPBpEq.

(iii)Letpµ_nqně0 be a probability sequence converging to someµinPpEq.Replacingpµ_nqně0

by a subsequence, we can always assume, by compactness ofPpEq,thatpΠ_µnqně0converges to someν.For everyně0andf PCpE,Rq, we have

Π_µnpfq “Π_µnpK_µnfq “Π_µnpKfq `Π_µnpδqµ_npfq.

ByH₁, the mapsKf andδare continuous and hence by lettingnÑ 8, one obtains νpfq “νpKfq `νpδqµpfq,

namelyνis an invariant forK_µ. By uniquenessν “Π_µ. This proves the continuity of the map µÞÑΠ_µ.Now, sincePpEqis a convex compact subset of a locally convex topological space (the space of signed measures equipped with the weak* topology) every continuous mapping from PpEqinto itself has a fixed point by Leray-Schauder-Tychonoff fixed point theorem.

For allµPPpMqandtě0we letP_t^µdenote the Markov kernel onEdefined by

P_t^µpx,¨q:“e^´tÿ

n

tⁿ

n!K_µⁿpx,¨q. (19)

It is classical (and easy to verify) that

(a) pP_t^µqtě0is a semigroup (i.eP_t`s^µ f “P_t^µP_s^µf for allf PBpE,Rq);

(b) Every invariant probability forKµis invariant forP_t^µ; (c) P_t^µis Feller wheneverK_µis (in particular underH₁).

IfpX_n^µqně0is a Markov chain with transitionKµ,pP_t^µqtě0denotes the semi-group ofpX_N^µ_tqtě0

wherepNtqtě0is an independent Poisson process with intensity1.

For any finite signed measureν onM recall that thetotal variation normofν is defined as }ν}TV “ supt|νf| : f PBpE,Rq,}f}8ď1u (20)

“ ν<sup>`</sup>pEq `ν^´pEq

whereν “ν^`´ν^´is the Hahn Jordan decomposition ofν.Let us recall that ifP is a Markov kernel onM andα, βPPpEq,then

}αP ´βP}TVď }α´β}_TV (21)

since}P f}₈ď }f}₈.

Lemma 4.4(Uniform exponential ergodicity). AssumeH₁andH₂.Then there exists0ăεă1 such that for allα, β, µPPpEqandtě0

}αP_t^µ´βP_t^µ}TVď p1´εq^ttu}α´β}TV. In particular, ifΠ_µdenotes an invariant probability forKµ,

}αP_t^µ´Π_µ}TVď p1´εq^ttu}α´Π_µ}TV. As a consequence,K_µhas a unique invariant probability.

Proof. (i). SetP_µ “P₁^µ.Letδ₀ ą0andN PNbe given by Lemma 4.1(i). It easily seen by induction that for allkě1andf :E ÞÑ r0,8rmeasurable,

K_µ^kf ěµpfqK^k´1δ.

Thus,

P_µf ě 1 eµpfq

N

ÿ

k“1

1

k!K^k´1δě 1 eN!µpfq

N

ÿ

k“1

K^k´1δ

“ 1

eN!µpfqp1´K^N1q ěεµpfq (22) where

ε“ 1 eN!δ0. LetRµbe the kernel onE defined by

@xPE, P_µpx, .q “εµp.q ` p1´εqR_µpx, .q. (23) Inequality (22) makesRµa Markov kernel. Thus for allα, β PPpEq

}αP_µ´βP_µ}TV“ p1´εq}αR_µ´βR_µ}TV ď p1´εq}α´β}_TV, (where the last inequality follows from (21)) and, by induction,

}αP_µⁿ´βP_µⁿ}TVď p1´εqⁿ}α´β}_TV. Now, for alltě0writet“n`rwithnPNand0ďr ă1.Then,

}αP_t^µ´βP_t^µ}TV“ }αP_r^µP_µⁿ´βP_r^µP_µⁿ}TV

ď p1´εqⁿ}αP_r^µ´βP_r^µ}TVď p1´εqⁿ}α´β}_TV.

As mentioned before, ifΠ_µis an invariant probability forK_µ,Π_µis also an invariant probability forpP_t^µqtě0. The second inequality is thus obtained by settingβ “ Π_µand uniqueness of the invariant probability is a consequence of the convergence ofpαP_t^µqtě0towardsΠ_µ. 4.1. Explicit form forΠ_µ. Let us denote byAthe transition kernel onEdefined by

Apx, .q “ ÿ

ně0

Kⁿpx, .q and set

}A}₈ “supt}Af}₈ : f PBpE,Rq,}f}8 ď1u.

Remark that

}A}₈ “sup

xPE

Apx,Eq P r0,8s.

Proposition 4.5. AssumeH₁andH₂. Then:

(i) For allxPE,

1ďApx,Eq“A1pxq “Exrτs ď }A}8ă 8.

(ii) For allµPPpMq,

Π_µ“ µA

pµAqp1q. (24)

(iii) The mapµÞÑΠ_µis Lipschitz continuous for the total variation distance.

Proof. (i) The inequalityApx,Eq ě 1 is obvious. For the second one, we remark that for all xPE

Apx,Eq “ ÿ

ně0

Kⁿpx,Eq “ ÿ

ně0

Pxpτ ąnq “Exrτs ďsup

x Expτq ă 8 where the last inequality follows from Lemma 4.1.

(ii)For anyf PBpE,Rq, µAK_µpfq “µ

˜ ÿ

ně0

pK<sup>n`1</sup>f`Kⁿδµpfqq

¸

“ ÿ

ně0

µK<sup>n`1</sup>f`µpfqµpÿ

ně0

Kⁿpδqq.

Sinceř

ně0Kⁿδpxq “ ř

ně0

`K<sup>n</sup>px,Eq ´K<sup>n`1</sup>px,Eq˘

“Apx,Eq ´ pApx,Eq ´1q “ 1,it follows that

pµAqKµpfq “µpfq ` ÿ

ně1

µKⁿf “ pµAqpfq.

As a consequence,µAis an invariant measure and it remains to divide by its mass to obtain an invariant probability.

(iii)It follows from(i)that}µA}TVď }µ}_TV}A}8andµA1ě1.Thus, reducing the fraction, it easily follows from(ii)that}Π_µ´Π_ν}TV ď2}A}8}µ´ν}TV.

5. THE LIMITINGODE

As mentioned before, the idea of the proof of Theorem 2.5 is to show that the long time behav- ior ofpµ_nqně0 can be precisely related to the long term behavior of a deterministic dynamical systemPpEqinduced by the "ODE"

“µ9 “ ´µ`Π_µ.” (25)

The purpose of this section is to define rigorously this dynamical system and to investigate some of its asymptotic properties.

Throughout the section, hypothesesH₁ andH₂are implicitly assumed. Recall thatPpEqis a compact metric space equipped with a distance metrizing the weak* convergence.

Asemi-flowonPpEqis a continuous map

Φ :R^`ˆPpEq ÑPpEq, pt, µq ÞÑΦ_tpµq such that

Φ₀pµq “µandΦ_t`spµq “Φ_t˝Φ_spµq.

We call such a semi-flowinjectiveif each of the mapsΦ_tis injective.

Aweak solutionto (25) with initial conditionµPPpEq,is a continuous mapξ :R^`ÞÑPpEq such that

ξptqf “µf` ż_t

0

p´ξpsqf`Π_ξpsqfqds for allf PCpEqandtě0.

We shall now show that there exists an injective semi-flowΦonPpEqsuch that the trajectory tÑΦ_tpµqis the unique weak solution to (25) with initial conditionµ.

LetM_spEqbe the space of finite signed measures onEequipped with the total variation norm } ¨ }TV (defined by equation (20)). By a Riesz type theorem,M_spEq is a Banach space which can be identified with the dual space ofCpE,Rqequipped with the uniform norm (see e.g [21,