A penalized bandit algorithm

HAL Id: hal-00012187

https://hal.archives-ouvertes.fr/hal-00012187

Submitted on 18 Oct 2005

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

A penalized bandit algorithm

Damien Lamberton, Gilles Pagès

To cite this version:

Damien Lamberton, Gilles Pagès. A penalized bandit algorithm. Electronic Journal of Probability,

Institute of Mathematical Statistics (IMS), 2008, 13, 341-373 ; http://dx.doi.org/10.1214/EJP.v13-

489. �10.1214/EJP.v13-489�. �hal-00012187�

ccsd-00012187, version 1 - 18 Oct 2005

A penalized bandit algorithm ^∗

Damien Lamberton

^†

Gilles Pag` es

^‡

Abstract

We study a two armed-bandit algorithm with penalty. We show the convergence of the algorithm and establish the rate of convergence. For some choices of the parame- ters, we obtain a central limit theorem in which the limit distribution is characterized as the unique stationary distribution of a discontinuous Markov process.

Key words: Two-armed bandit algorithm, penalization, stochastic approximation, conver- gence rate, learning automata, asset allocation.

2001 AMS classification: 62L20, secondary 93C40, 91E40, 68T05, 91B32 .

Introduction

In a recent joint work with P. Tarr`es (see [12]), we studied the convergence of the so-called two armed bandit algorithm. The purpose of the present paper is to investigate a modified version of this algorithm, in which a penalization is introduced. In the terminology of learning theory (see [14, 15]), the algorithm studied in [12] was a Linear Reward-Inaction (LRI) scheme, whereas the one we want to introduce is a Linear Reward-Penalty (LRP ) procedure.

In our previous paper, the algorithm was introduced in a financial context as a pro- cedure for the optimal allocation of a fund between two traders who manage it. Imagine that the owner of a fund can share his wealth between two traders, say A and B , and that, every day, he can evaluate the results of one of the traders and, subsequently, modify the percentage of the fund managed by both traders. Denote by X

_n

the percentage managed by trader A at time n (X

_n

∈ [0, 1]). We assume that the owner selects the trader to be evaluated at random, in such a way that the probability that A is evaluated at time n is X

n

, in order to select preferably the trader in charge of the greater part of the fund. In the LRI scheme, if the evaluated trader performs well, its share is increased by a fraction

∗

This work has benefitted from the stay of both authors at the Isaac Newton Institute (Cambridge University) on the program Developments in Quantitative Finance.

†

Laboratoire d’analyse et de math´ematiques appliqu´ees, UMR 8050, Univ. Marne-la-Vall´ee, Cit´e Descartes, 5, Bld Descartes, Champs-sur-Marne, F-77454 Marne-la-Vall´ee Cedex 2, France.

damien.lamberton@univ-mlv.fr

‡

Laboratoire de probabilit´es et mod`eles al´eatoires, UMR 7599, Univ. Paris 6, case 188, 4, pl. Jussieu,

F-75252 Paris Cedex 5. gpa@ccr.jussieu.fr

γ

_n

∈ (0, 1) of the share of the other trader, and nothing happens if the evaluated trader performs badly. Therefore, the dynamics of the sequence (X

_n

)

_n≥0

can be modelled as follows:

X

_n+1

= X

_n

+ γ

_n+1

1

_{{Un+1≤Xn}∩An+1}

(1 − X

_n

) − 1

_{{Un+1>Xn}∩Bn+1}

X

_n

, X

₀

= x ∈ [0, 1], where (U

_n

)

_n≥1

is an iid sequence of uniform random variables on the interval [0, 1], A

_n

(resp. B

n

) is the event “trader A (resp. trader B) performs well at time n”. We assume P _(A

_n

_{) = p}

A

, P _(B

_n

_{) = p}

B

, for n ≥ 1, with p

_A

, p

_B

∈ (0, 1), and independence between these events and the sequence (U

_n

)

_n≥1

. The point is that the owner of the fund does not know the parameters p

_A

, p

_B

.

This recursive learning procedure has been designed in order to assign asymptotically the whole fund to the best trader. This means that, if say p

_A

> p

_B

, X

_n

converges to 1 with probability 1 provided X

0

∈ (0, 1) (if p

_A

< p

_B

, the limit is 0 with symmetric results).

However this “infallibility” property needs some very stringent assumptions on the reward parameter γ

_n

(see [12]). Furthermore, the rate of convergence of the procedure either toward its “target” 1 or its “trap” 0 is not ruled by a CLT with rate √ γ

n

like standard stochastic approximation algorithms (see [10]). It is shown in [11] that this rate is quite non-standard, strongly depends on the (unknown) values p

_A

and p

_B

and becomes very poor as these probabilities get close to each other.

In order to improve the efficiency of the algorithm, one may imagine to introduce a penalty when an evaluated trader has unsatisfactory performances. More precisely, if the evaluated trader at time n performs badly, its share is decreased by a penalty factor ρ

_n

γ

_n

. This leads to the following LRP – or “penalized two-armed bandit – procedure

X

_n+1

= X

_n

+ γ

_n+1

1

_{{Un+1≤Xn}∩An+1}

(1 − X

_n

) − 1

_{{Un+1>Xn}∩Bn+1}

X

_n

− γ

_n+1

ρ

_n+1

X

_n

1

_{{Un+1≤Xn}∩A}^c

n+1

− (1 − X

_n

)1

_{{Un+1>Xn}∩B}^c

n+1

, n ∈ N _, where the notation A

^c

is used for the complement of an event A. The precise assumptions on the reward rate γ

_n

and the penalty rate γ

_n

ρ

_n

will be given in the following sections.

The paper is organized as follows. In Section 1, we discuss the convergence of the sequence (X

_n

)

_n≥0

. First we show that, if ρ

_n

is a positive constant ρ, the sequence converges with probability one to a limit x

^∗_ρ

∈ (0, 1) satisfying x

^∗_ρ

>

¹₂

if and only if p

_A

> p

_B

, so that, although the algorithm manages to distinguish which trader is better, it does not assign the whole fund to the best trader. To get rid of this limitation, we consider a sequence (ρ

n

)

n≥1

which goes to zero so that the penalty rate becomes negligible with respect to the reward rate (γ

_n

ρ

_n

= o(γ

_n

)). This framework seems new in the learning theory literature.

Then, we are able to show that the algorithm is infallible i.e., if p

_A

> p

_B

, then lim

n→∞

X

_n

= 1

almost surely, under very light conditions on the reward rate γ

n

(and ρ

n

). From a stochastic

approximation viewpoint, this modification of the original procedure has the same mean

function and time scale (hence the same target and trap, see (5)) but it always keeps the

algorithm away from the trap without adding noise at these equilibria. In fact, it was

necessary not to add noise at these points in order to remain inside the domain [0, 1].

The other two sections are devoted to the rate of convergence. In Section 2, we show that under some conditions (including lim

n→∞

γ

_n

/ρ

_n

= 0) the sequence Y

_n

= (1 − X

_n

)/ρ

_n

converges in probability to (1 − p

_A

)/π, where π = p

_A

− p

_B

> 0. With additional assumptions, we prove that this convergence occurs with probability 1. In Section 3, we show that if the ratio γ

n

/ρ

n

goes to a positive limit as n goes to infinity, then (Y

n

)

n≥1

converges in a weak sense to a probability distribution ν. This distribution is identified as the unique stationary distribution of a discontinuous Markov process. This result is obtained by using weak functional methods applied to a re-scaling of the algorithm. This approach can be seen as an extension of the SDE method used to prove the CLT in a more standard framework of stochastic approximation (see [10]). Furthermore, we show that ν is absolutely continuous with continuous, possibly non-smooth, piecewise C

^∞

density. An interesting consequence of these results for practical applications is that, by choosing ρ

_n

and γ

_n

proportional to n

^−1/2

, one can achieve convergence at the rate 1/ √

n, without any a priori knowledge about the values of p

_A

and p

_B

. This is in contrast with the case of the LRI procedure, where the rate of convergence depends heavily on these parameters (see [11]) and becomes quite poor when they get close to each other.

Notation. Let (a

_n

)

_n≥0

and (b

_n

)

_n≥0

be two sequences of positive real numbers. The symbol a

n

∼ b

n

means a

n

= b

n

+ o(b

n

).

1 Convergence of the LRP algorithm

1.1 Some classical background on stochastic approximation

We will rely on the ODE lemma recalled below for a stochastic procedure (Z

_n

) taking its values in a given compact interval I .

Theorem 1 (a) Kushner & Clark’s ODE Lemma (see [9]): Let g : I → R _{such that} Id + g leaves I stable (

¹

). Then, consider the recursively defined stochastic approximation procedure defined on I by

Z

_n+1

= Z

_n

+ γ

_n+1

(g(Z

_n

) + ∆R

_n+1

), n ≥ 0, Z

₀

∈ I,

where (γ

n

)

n≥1

is a sequence of [0, 1]-valued real numbers satisfying γ

n

→ 0 and ^P

_n≥1

γ

n

= + ∞ . Set N (t) := min { n : γ

₁

+ · · · + γ

_n+1

> t } . If, for every T > 0,

N(t)≤n≤N

max

(t+T)

n

X

k=N(t)+1

γ

_k

∆R

_k

−→ 0 P _{-a.s. as t → + ∞ . (1)} Let z

^∗

be an attracting zero of g in I and G(z

^∗

) its attracting interval. Then, on the event

{ Z

n

visits infinitely often a compact subset of G(z

^∗

) } Z

n a.s.

−→ z

^∗

.

1

then for every γ ∈ [0, 1], Id + γg = γ(Id + g) + (1 − γ)Id still takes values in the convex set I

(b) The Hoeffding condition (see [1]): If (∆R

_n

)

_n≥0

is a sequence of L

^∞

-bounded martingale increments, if (γ

n

) is nonincreasing and ^X

n≥1

e

^−γnϑ

< + ∞ for every ϑ > 0, then Assumption (1) is satisfied.

Remark. The monotonous assumption on the sequence γ can be relaxed into γ

_n

→ 0 and sup

n,k≥1

γ

_n+k

γ

_n

< + ∞

1.2 Basic properties of the LRP algorithm

We first recall the definition of the algorithm. We are interested in the asymptotic behavior of the sequence (X

n

)

_n∈N

, where X

0

= x, with x ∈ (0, 1), and

X

_n+1

= X

_n

+ γ

_n+1

1

_{{Un+1≤Xn}∩An+1}

(1 − X

_n

) − 1

_{{Un+1>Xn}∩Bn+1}

X

_n

− γ

_n+1

ρ

_n+1

X

_n

1

_{{Un+1≤Xn}∩A}^c

n+1

− (1 − X

_n

)1

_{{Un+1>Xn}∩B}^c

n+1

, n ∈ N _. Throughout the paper, we assume that (γ

n

)

n≥1

is a non-increasing sequence of positive numbers satisfying γ

_n

< 1,

∞

X

n=1

γ

_n

= + ∞ and

∀ ϑ > 0, ^X

n

e

^−γnϑ

< ∞ ,

and that (ρ

n

)

n≥1

is a sequence of positive numbers satisfying γ

n

ρ

n

< 1; (U

n

)

n≥1

is a sequence of independent random variables which are uniformly distributed on the interval [0, 1], the events A

_n

, B

_n

satisfy

P _(A

_n

_{) = p}

A

, P _(B

_n

_{) = p}

B

, n ∈ N _,

where 0 < p

_B

≤ p

_A

< 1, and the sequences (U

n

)

n≥1

and (1

An

, 1

_Bn

)

n≥1

are independent.

The natural filtration of the sequence (U

_n

, 1

_An

, 1

_Bn

)

_n≥1

is denoted by ( F

n

)

_n≥0

and we set π = p

_A

− p

_B

.

With this notation, we have, for n ≥ 0,

X

_n+1

= X

_n

+ γ

_n+1

(πh(X

_n

) + ρ

_n+1

κ(X

_n

)) + γ

_n+1

∆M

_n+1

, (2) where the functions h and κ are defined by

h(x) = x(1 − x), κ(x) = − (1 − p

_A

)x

²

+ (1 − p

_B

)(1 − x)

²

, 0 ≤ x ≤ 1,

∆M

_n+1

= M

_n+1

− M

_n

, and the sequence (M

_n

)

_n≥0

is the martingale defined by M

₀

= 0 and

∆M

_n+1

= 1

_{{Un+1≤Xn}∩An+1}

(1 − X

_n

) − 1

_{{Un+1>Xn}∩Bn+1}

X

_n

− πh(X

_n

)

− ρ

_n+1

X

_n

1

_{{Un+1≤Xn}∩A}^c

n+1

− (1 − X

_n

)1

_{{Un+1>Xn}∩B}^c

n+1

+ κ(X

_n

) . (3)

Observe that the increments ∆M

_n+1

are bounded.

1.3 Constant penalty rate In this subsection, we assume

∀ n ≥ 1, ρ

n

= ρ, with 0 < ρ ≤ 1. We then have

X

n+1

= X

n

+ γ

n+1

(h

ρ

(X

n

) + ∆M

n+1

) , where

h

ρ

(x) = πh(x) + ρκ(x), 0 ≤ x ≤ 1.

Note that h

_ρ

(0) = ρ(1 − p

_B

) > 0 and h

_ρ

(1) = − ρ(1 − p

_A

) < 0, and that there exists a unique x

^∗_ρ

∈ (0, 1) such that h

_ρ

(x

^∗_ρ

) = 0. By a straightforward computation, we have

x

^∗_ρ

= π − 2ρ(1 − p

_B

) + ^p π

²

+ 4ρ

²

(1 − p

_B

)(1 − p

_A

)

2π(1 − ρ) if π 6 = 0 and ρ 6 = 1

= (1 − p

_A

)

(1 − p

_A

) + (1 − p

_B

) if π = 0 or ρ = 1.

In particular, x

^∗_ρ

= 1/2 if π = 0 regardless of the value of ρ. We also have h

ρ

(1/2) = π(1 + ρ)/4 ≥ 0, so that

x

^∗_ρ

> 1/2 if π > 0. (4)

Now, let x be a solution of the ODE dx/dt = h

_ρ

(x). If x(0) ∈ [0, x

^∗_ρ

], x is non-decreasing and lim

t→∞

x(t) = x

^∗_ρ

. If x(0) ∈ [x

^∗_ρ

, 1], x is non-increasing and lim

t→∞

x(t) = x

^∗_ρ

. It follows that the interval [0, 1] is a domain of attraction for x

^∗_ρ

. Consequently, using Kushner and Clark’s ODE Lemma (see Theorem 1), one reaches the following conclusion.

Proposition 1 Assume that ρ

_n

= ρ ∈ (0, 1], then

X

_n

−→

^a.s.

x

^∗_ρ

as n → ∞ .

The natural interpretation, given the above inequalities on x

^∗_ρ

, is that this algorithm never fails in pointing the best trader thanks to Inequality (4), but it never assigns the whole fund to this trader as the original LRI procedure did.

1.4 Convergence when the penalty rate goes to zero Proposition 2 Assume lim

n→∞

ρ

_n

= 0. The sequence (X

_n

)

_n∈N

is almost surely convergent and its limit X

_∞

satisfies X

_∞

∈ { 0, 1 } with probability 1.

Proof: We first write the algorithm in its canonical form

X

_n+1

= X

_n

+ γ

_n+1

(π h(X

_n

) + ∆R

_n+1

) with ∆R

_n

= ∆M

_n

+ ρ

_n

κ(X

_n−1

). (5)

It is straightforward to check that the ODE x ˙ = h(x) has two equilibrium points, 0 and

1, 1 being attractive with (0, 1] as an attracting interval and 0 is unstable.

Since the martingale increments ∆M

_n

are bounded, it follows from the assumptions on the sequence (γ

_n

)

_n≥1

and the Hoeffding condition (see Theorem 1(b)) that

N(t)≤n≤N

max

(t+T)

|

n

X

k=N(t)+1

γ

_k

∆M

_k

|

^P

-

a.s.

−→ 0 as t → + ∞

for every T > 0. On the other hand the function κ being bounded on [0, 1] and ρ

_n

converging to 0, we have, for every T > 0,

N(t)≤n≤N(t+T

max

)

|

n

X

k=N(t)+1

γ

_k

ρ

_k

κ(X

_k−1

) | ≤ k k k

[0,1]

(T +γ

_N(t+T)

) max

k≥N(t)+1

ρ

_k

−→ 0 as t → + ∞ . Finally, the sequence (∆R

_n

)

_n≥1

satisfies Assumption (1). Consequently, either X

_n

visits infinitely often an interval [ε, 1] for some ε > 0 and X

_n

converges toward 1, or X

_n

converges toward 0.

_♦

Remark 1 If π = 0, i.e. p

_A

= p

_B

, the algorithm reduces to

X

n+1

= X

n

+ γ

n+1

ρ

n+1

(1 − p

_A

)(1 − 2X

n

) + γ

n+1

∆M

n+1

.

The number 1/2 is the unique equilibrium of the ODE ˙ x = (1 − p

_A

)(1 − 2x), and the interval [0, 1] is a domain of attraction. Assuming ^P

^∞_n=1

ρ

_n

γ

_n

= + ∞ , and that the sequence (γ

_n

/ρ

_n

)

_n≥1

is non-increasing and satisfies

∀ ϑ > 0,

∞

X

n=1

exp

− ϑ ρ

_n

γ

_n

< + ∞ ,

it can be proved, using the Kushner-Clark ODE Lemma (Theorem 1), that lim

n→∞

X

_n

= 1/2 almost surely. As concerns the asymptotics of the algorithm when π = 0 and γ

_n

= g ρ

_n

(for which the above condition is not satisfied), we refer to the final remark of the paper.

From now on, we will assume that p

_A

> p

_B

. The next proposition shows that the penalized algorithm is infallible under very light assumptions on γ

_n

and ρ

_n

.

Proposition 3 (Infallibility) Assume lim

n→∞

ρ

_n

= 0. If the sequence (γ

_n

/ρ

_n

)

_n≥1

is bounded and ^P

_n

γ

_n

ρ

_n

= ∞ , and if π > 0, we have lim

n→∞

X

_n

= 1 almost surely.

Proof: We have from (2), since h ≥ 0 on the interval [0, 1], X

_n

≥ X

₀

+

n

X

j=1

γ

_j

ρ

_j

κ(X

_j−1

) +

n

X

j=1

γ

_j

∆M

_j

, n ≥ 1.

Since the jumps ∆M

j

are bounded, we have

n

X

j=1

γ

j

∆M

j

2

L²

≤ C

n

X

j=1

γ

_j²

≤ C sup

j∈N

(γ

j

/ρ

j

)

n

X

j=1

γ

j

ρ

j

,

for some positive constant C. Therefore, since ^P

_n

γ

_n

ρ

_n

= ∞ , L

²

- lim

n→∞

P

n

j=1

γ

_j

∆M

_j

P

n

j=1

γ

j

ρ

j

= 0 so that lim sup

n

P

n

j=1

γ

_j

∆M

_j

P

n

j=1

γ

j

ρ

j

≥ 0 a.s..

Now, on the set { X

_∞

= 0 } , we have

n→∞

lim

n

X

j=1

γ

_j

ρ

_j

κ(X

_j−1

)

n

X

j=1

γ

_j

ρ

_j

= κ(0) > 0.

Hence, it follows that, still on the set { X

_∞

= 0 } , lim sup

n→∞

X

_n

n

X

j=1

γ

_j

ρ

_j

> 0.

Therefore, we must have P _(X

_∞

_{= 0) = 0.}

_♦

The following Proposition will give a control on the conditional variance process of the martingale (M

n

)

_n∈N

which will be crucial to elucidate the rate of convergence of the algorithm.

Proposition 4 We have, for n ≥ 0, E _∆M

²

n+1

| F

n

≤ p

_A

(1 − X

_n

) + ρ

²_n+1

(1 − p

_B

).

Proof: We have

∆M

_n+1

= V

_n+1

− E _(V

_n+1

_{| F}

_n

_{) + W}

_n+1

₋ E _(W

_n+1

_{| F}

_n

_), with

V

_n+1

= 1

_{{Un+1≤Xn}∩An+1}

(1 − X

_n

) − 1

_{{Un+1>Xn}∩Bn+1}

X

_n

and

W

_n+1

= − ρ

_n+1

X

_n

1

_{{Un+1≤Xn}∩A}^c

n+1

− (1 − X

_n

)1

_{{Un+1>Xn}∩B}^c

n+1

.

Note that V

_n+1

W

_n+1

= 0, so that

E _∆M

_n+1²

_{| F}

_n

₌ E _(V

_n+1²

_{| F}

_n

_{) +} E _(W

_n+1²

_{| F}

_n

_{) − (} E _(V

_n+1

_{+ W}

_n+1

_{| F}

_n

₎₎

²

≤ E _(V

_n+1²

_{| F}

_n

_{) +} E _(W

_n+1²

_{| F}

_n

_).

Now, using p

_B

≤ p

_A

and X

_n

≤ 1, E _V

²

n+1

| F

n

= p

_A

X

_n

(1 − X

_n

)

²

+ p

_B

(1 − X

_n

)X

_n²

≤ p

_A

(1 − X

_n

) and E _(W

_n+1²

_{| F}

_n

_{) = ρ}

²_n+1

_X

_n³

_{(1 − p}

A

) + (1 − X

_n

)

³

(1 − p

_B

)

≤ ρ

²_n+1

(1 − p

_B

).

This proves the Proposition.

_♦

2 The rate of convergence: pointwise convergence

2.1 Convergence in probability Theorem 2 Assume

n→∞

lim ρ

_n

= 0, lim

n→∞

γ

_n

ρ

_n

= 0, ^X

n

ρ

_n

γ

_n

= ∞ , ρ

_n

− ρ

_n−1

= o(ρ

_n

γ

_n

). (6) Then, the sequence ((1 − X

_n

)/ρ

_n

)

_n≥1

converges to (1 − p

_A

)/π in probability.

Note that the assumptions of Theorem 2 are satisfied if γ

_n

= C/n

^a

and ρ

_n

= C

^′

/n

^r

, with C, C

^′

> 0, 0 < r < a and a + r < 1. In fact, we will see that for this choice of parameters, convergence holds with probability one (see Theorem 3).

Before proving Theorem 2, we introduce the notation Y

_n

= 1 − X

_n

ρ

_n

. We have, from (2)

1 − X

n+1

= 1 − X

n

− γ

n+1

πX

n

(1 − X

n

) − ρ

n+1

γ

n+1

κ(X

n

) − γ

n+1

∆M

n+1

1 − X

_n+1

ρ

_n+1

= 1 − X

_n

ρ

_n+1

− γ

_n+1

ρ

_n+1

πX

_n

(1 − X

_n

) − γ

_n+1

κ(X

_n

) − γ

_n+1

ρ

_n+1

∆M

_n+1

. Hence

Y

_n+1

= Y

_n

+ (1 − X

_n

) 1

ρ

_n+1

− 1

ρ

_n

− γ

_n+1

ρ

_n+1

πX

_n

− γ

_n+1

κ(X

_n

) − γ

_n+1

ρ

_n+1

∆M

_n+1

Y

_n+1

= Y

_n

(1 + γ

_n+1

ε

_n

− π

_n

γ

_n+1

X

_n

) − γ

_n+1

κ(X

_n

) − γ

_n+1

ρ

_n+1

∆M

_n+1

, where

ε

n

= ρ

_n

γ

_n+1

1 ρ

_n+1

− 1

ρ

_n

and π

n

= ρ

_n

ρ

_n+1

π.

It follows from the assumption ρ

_n

− ρ

_n−1

= o(ρ

_n

γ

_n

) that lim

n→∞

ε

_n

= 0 and lim

n→∞

π

_n

= π.

Lemma 1 Consider two positive numbers π

⁻

and π

⁺

with 0 < π

⁻

< π < π

⁺

< 1. Given l ∈ N _{, let}

ν

^l

= inf { n ≥ l | π

_n

X

_n

− ε

_n

> π

⁺

or π

_n

X

_n

− ε

_n

< π

⁻

} . We have

• lim

l→∞

P _(ν

^l

_{= ∞ ) = 1,}

• for n ≥ l, if θ

⁺_n

= ^Q

ⁿ_k=l+1

(1 − π

⁺

γ

_k

) and θ

⁻_n

= ^Q

ⁿ_k=l+1

(1 − π

⁻

γ

_k

), Y

_n∧ν^l

θ

_n∧ν⁻ l

≤ Y

_l

−

n∧ν^l

X

k=l+1

γ

_k

θ

_k⁻

κ(X

_k−1

) −

n∧ν^l

X

k=l+1

γ

_k

ρ

_k

θ

⁻_k

∆M

_k

(7)

and

Y

_n∧ν^l

θ

⁺_n∧νl

≥ Y

_l

−

n∧ν^l

X

k=l+1

γ

_k

θ

⁺_k

κ(X

_k−1

) −

n∧ν^l

X

k=l+1

γ

_k

ρ

_k

θ

⁺_k

∆M

_k

. (8) Moreover, with the notation || k ||

∞

= sup

0<x<1

| κ(x) | ,

sup

n≥l

E _Y

_n

₁

{ν^l=∞}

≤ E _Y

_l

₊ ^{|| k ||}

^∞

π

⁻

.

Remark 2 Note that, as the proof will show, Lemma 1 remains valid if the condition

n→∞

lim γ

n

/ρ

n

= 0 in (6) is replaced by the boundedness of the sequence (γ

n

/ρ

n

)

n≥1

. In particular, the last statement, which implies the tightness of the sequence (Y

_n

)

_n≥1

, will be used in Section 3.

Proof: Since lim

n→∞

(π

_n

X

_n

− ε

_n

) = π a.s., we clearly have lim

l→∞

P _(ν

^l

_{= ∞ ) = 1.}

On the other hand, for l ≤ n < ν

^l

, we have

Y

_n+1

≤ Y

_n

(1 − γ

_n+1

π

⁻

) − γ

_n+1

κ(X

_n

) − γ

n+1

ρ

_n+1

∆M

_n+1

and

Y

_n+1

≥ Y

_n

(1 − γ

_n+1

π

⁺

) − γ

_n+1

κ(X

_n

) − γ

_n+1

ρ

_n+1

∆M

_n+1

,

so that, with the notation θ

_n⁺

= ^Q

ⁿ_k=l+1

(1 − π

⁺

γ

_k

) and θ

_n⁻

= ^Q

ⁿ_k=l+1

(1 − π

⁻

γ

_k

), Y

_n+1

θ

⁻_n+1

≤ Y

_n

θ

n⁻

− γ

_n+1

θ

_n+1⁻

κ(X

_n

) − γ

_n+1

ρ

_n+1

θ

⁻_n+1

∆M

_n+1

and Y

n+1

θ

_n+1⁺

≥ Y

n

θ

n⁺

− γ

n+1

θ

⁺_n+1

κ(X

n

) − γ

n+1

ρ

n+1

θ

_n+1⁺

∆M

n+1

. By summing up these inequalities, we get (7) and (8).

By taking expectations in (7), we get E ^Y

^n∧νl

θ

_n∧ν⁻ l

≤ E _Y

_l

_{+ || k ||}

_∞

E

n∧ν^l

X

k=l+1

γ

_k

θ

_k⁻

= E _Y

_l

₊ ^{|| k ||}

^∞

π

⁻

E

n∧ν^l

X

k=l+1

1 θ

_k⁻

− 1

θ

⁻_k−1

!

≤ E _Y

_l

₊ ^{|| k ||}

^∞

π

⁻

1 θ

n⁻

. We then have

E _(Y

_n

₁

_{νl=∞}

) = θ

⁻_n

E ^Y

^n∧νl

θ

⁻_n∧νl

1

_{νl=∞}

!

≤ θ

⁻_n

E ^Y

^n∧νl

θ

_n∧ν⁻ l

≤ θ

⁻_n

E _Y

_l

₊ ^{|| k ||}

^∞

π

⁻

1 θ

n⁻

≤ E _Y

_l

₊ ^{|| k ||}

^∞

π

⁻

.

♦

Lemma 2 Let (θ

_n

)

_n∈N

be a sequence of positive numbers such that θ

_n

= ^Q

ⁿ_k=1

(1 − pγ

_k

) for some p ∈ (0, 1). The sequence θ

_n

^P

ⁿ_{k=1 θ}^γk

kρk

∆M

_k

n∈N

converges to 0 in probability.

Proof: It suffices to show convergence to 0 in probability for the associated conditional variances T

_n

, defined by

T

_n

= θ

_n²

n

X

k=1

γ

_k²

θ

²_k

ρ

²_k

E _∆M

_k²

_{| F}

_k−1

_. We know from Proposition 4 that

E _∆M

_k²

_{| F}

_k−1

_{≤ p}

A

(1 − X

_k−1

) + ρ

²_k

(1 − p

_B

)

= p

_A

ρ

_k−1

Y

_k−1

+ ρ

²_k

(1 − p

_B

).

Therefore, T

_n

≤ p

_A

T

n⁽¹⁾

+ (1 − p

_B

)T

n⁽²⁾

, where T

_n⁽¹⁾

= θ

²_n

n

X

k=1

γ

_k²

θ

_k²

ρ

²_k

ρ

_k−1

Y

_k−1

and

T

_n⁽²⁾

= θ

²_n

n

X

k=1

γ

_k²

θ

²_k

. We first prove that lim

n→∞

T

_n⁽²⁾

= 0. Note that, since pγ

_k

≤ 1, 1

θ

²_k

− 1

θ

²_k−1

= 2pγ

_k

− p

²

γ

_k²

θ

_k²

≥ p γ

_k

θ

_k²

. (9)

Therefore,

T

_n⁽²⁾

≤ θ

_n²

p

n

X

k=1

γ

_k

1 θ

_k²

− 1

θ

²_k−1

! , and lim

n→∞

T

_n⁽²⁾

= 0 follows from Cesaro’s lemma.

We now deal with T

n⁽¹⁾

. First note that the assumption ρ

n

− ρ

n−1

= o(ρ

n

γ

n

) implies

n→∞

lim ρ

_n

/ρ

_n−1

= 1, so that, the sequence (γ

_n

)

_n≥1

being non-increasing with limit 0, we only need to prove that lim

n→∞

T ¯

_n⁽¹⁾

= 0 in probability, where T ¯

_n⁽¹⁾

= θ

²_n

n

X

k=1

γ

_k²

θ

_k²

ρ

_k

Y

_k

.

Now, with the notation of Lemma 1, we have, for n ≥ l > 1 and ε > 0, P T ¯

_n⁽¹⁾

≥ ε ≤ P _(ν

^l

_{< ∞ ) +} P _θ

_n²

n

X

k=1

γ

_k²

θ

_k²

ρ

_k

Y

_k

1

_{ν^l_=∞}

≥ ε

!

≤ P _(ν

^l

_{< ∞ ) +} ¹ ε θ

_n²

n

X

k=1

γ

_k²

θ

²_k

ρ

_k

E _Y

_k

₁

_{νl=∞}

.

Using Lemma 1, lim

n→∞

γ

_n

/ρ

_n

= 0 and (9), we have

n→∞

lim θ

²_n

n

X

k=1

γ

_k²

θ

_k²

ρ

_k

E _Y

_k

₁

{ν^l=∞}

= 0.

We also know that lim

l→∞

P _(ν

^l

_{< ∞} ) = 0. Hence,

n→∞

lim P T ¯

_n⁽¹⁾

≥ ε = 0.

_♦

Proof of Theorem 2: First note that if θ

_n

= ^Q

ⁿ_k=1

(1 − pγ

_k

), with 0 < p < 1, we have

n

X

k=1

γ

_k

θ

_k

κ(X

_k−1

) = 1 p

n

X

k=1

1 θ

_k

− 1

θ

_k−1

κ(X

_k−1

).

Hence, using lim

n→∞

X

_n

= 1 and κ(1) = − (1 − p

_A

),

n→∞

lim θ

_n

n

X

k=1

γ

_k

θ

_k

κ(X

_k−1

) = − 1 − p

_A

p .

Going back to (7) and (8) and using Lemma 2 with p = π

⁺

and π

⁻

, and the fact that

l→∞

lim

P _(ν

^l

_{= ∞} ) = 1, we have, for all ε > 0, lim

n→∞

P _(Y

_n

_≥ ^{1 − p}

^A

π

⁻

+ ε) = lim

n→∞

P _(Y

_n

_≤ 1 − p

_A

π

⁺

− ε) = 0, and since π

⁺

and π

⁻

can be made arbitrarily close to π, the Theorem is

proved.

_♦

2.2 Almost sure convergence

Theorem 3 In addition to (6), we assume that for all β ∈ [0, 1],

γ

_n

ρ

^β_n

− γ

_n−1

ρ

^β_n−1

= o(γ

_n²

ρ

^β_n

), (10) and that, for some η > 0, we have

∀ C > 0, ^X

n

exp − C ρ

^1+η_n

γ

n

!

< ∞ . (11)

Then, with probability 1,

n→∞

lim

1 − X

_n

ρ

_n

= 1 − p

_A

π .

Note that the assumptions of Theorem 3 are satisfied if γ

_n

= Cn

^−a

and ρ

_n

= C

^′

n

^−r

, with C, C

^′

> 0, 0 < r < a and a + r < 1.

The proof of Theorem 3 is based on the following lemma, which will be proved later.

Lemma 3 Under the assumptions of Theorem 3, let α ∈ [0, 1] and let (θ

_n

)

_n∈≥1

be a sequence of positive numbers such that θ

_n

= ^Q

ⁿ_k=1

(1 − pγ

_k

), for some p ∈ (0, 1). On the set { sup

_n

(ρ

^α_n

Y

_n

) < ∞} , we have

n→∞

lim θ

_n

ρ

α−η 2 −1 n

n

X

k=1

γ

_k

θ

_k

∆M

_k

= 0 a.s., where η satisfies (11).

Proof of Theorem 3: We start from the following form of (2):

1 − X

_n+1

= (1 − X

_n

)(1 − γ

_n+1

πX

_n

) − ρ

_n+1

γ

_n+1

κ(X

_n

) − γ

_n+1

∆M

_n+1

. We know that lim

n→∞

X

_n

= 1 a.s.. Therefore, given π

⁺

and π

⁻

, with 0 < π

⁻

< π < π

⁺

< 1, there exists l ∈ N such that, for n ≥ l,

1 − X

_n+1

≤ (1 − X

_n

)(1 − γ

_n+1

π

⁻

) − ρ

_n+1

γ

_n+1

κ(X

_n

) − γ

_n+1

∆M

_n+1

and

1 − X

_n+1

≥ (1 − X

_n

)(1 − γ

_n+1

π

⁺

) − ρ

_n+1

γ

_n+1

κ(X

_n

) − γ

_n+1

∆M

_n+1

, so that, with the notation θ

_n⁺

= ^Q

ⁿ_k=l+1

(1 − π

⁺

γ

_k

) and θ

_n⁻

= ^Q

ⁿ_k=l+1

(1 − π

⁻

γ

_k

),

1 − X

_n+1

θ

⁻_n+1

≤ 1 − X

_n

θ

n⁻

− ρ

_n+1

γ

_n+1

θ

_n+1⁻

κ(X

_n

) − γ

_n+1

θ

⁻_n+1

∆M

_n+1

and 1 − X

_n+1

θ

_n+1⁺

≥ 1 − X

_n

θ

⁺n

− ρ

_n+1

γ

_n+1

θ

⁺_n+1

κ(X

_n

) − γ

_n+1

θ

_n+1⁺

∆M

_n+1

. By summing up these inequalities, we get, for n ≥ l + 1,

1 − X

_n

θ

⁻n

≤ (1 − X

_l

) −

n

X

k=l+1

ρ

_k

γ

_k

θ

_k⁻

κ(X

_k−1

) −

n

X

k=l+1

γ

_k

θ

_k⁻

∆M

_k

and

1 − X

_n

θ

⁺n

≥ (1 − X

_l

) −

n

X

k=l+1

ρ

_k

γ

_k

θ

_k⁺

κ(X

_k−1

) −

n

X

k=l+1

γ

_k

θ

⁺_k

∆M

_k

. Hence

Y

_n

≤ θ

⁻_n

ρ

_n

(1 − X

_l

) − θ

_n⁻

ρ

_n

n

X

k=l+1

ρ

_k

γ

_k

θ

⁻_k

κ(X

_k−1

) − θ

⁻_n

ρ

_n

n

X

k=l+1

γ

_k

θ

_k⁻

∆M

_k

(12) and

Y

_n

≥ θ

_n⁺

ρ

_n

(1 − X

_l

) − θ

⁺_n

ρ

_n

n

X

k=l+1

ρ

_k

γ

_k

θ

_k⁺

κ(X

_k−1

) − θ

_n⁺

ρ

_n

n

X

k=l+1

γ

_k

θ

_k⁺

∆M

_k

. (13) We have, with probability 1, lim

n→∞

κ(X

_n

) = κ(1) = − (1 − p

_A

), and, since ^P

^∞_n=1

ρ

_n

γ

_n

= + ∞ ,

n

X

k=l+1

ρ

_k

γ

_k

θ

⁻_k

κ(X

_k−1

) ∼ − (1 − p

_A

)

n

X

k=l+1

ρ

_k

γ

_k

θ

⁻_k

. (14)

On the other hand,

n

X

k=l+1

ρ

_k

γ

_k

θ

_k⁻

= 1 π

⁻

n

X

k=l+1

ρ

_k

1 θ

_k⁻

− 1

θ

⁻_k−1

!

= 1

π

⁻





n

X

k=l+1

(ρ

_k−1

− ρ

_k

) 1

θ

_k−1⁻

+ ρ

_n

θ

n⁻

− ρ

_l

θ

⁻_l





∼ 1 π

⁻

ρ

_n

θ

n⁻

, (15)

where we have used the condition ρ

_k

− ρ

_k−1

=o(ρ

_k

γ

_k

). We deduce from (14) and (15) that

n→∞

lim θ

_n⁻

ρ

_n

n

X

k=1

ρ

_k

γ

_k

θ

_k⁻

κ(X

_k−1

) = − 1 − p

_A

π

⁻

and, also, that lim

n→∞

θ

_n⁻

ρ

n

= 0. By a similar argument, we get lim

n→∞

θ

_n⁺

ρ

n

= 0 and

n→∞

lim θ

_n⁺

ρ

_n

n

X

k=1

ρ

_k

γ

_k

θ

_k⁺

κ(X

_k−1

) = − 1 − p

_A

π

⁺

It follows from Lemma 3, that given α ∈ [0, 1], we have, on the set E

_α

:= { sup

_n

(ρ

^α_n

Y

_n

) < ∞} ,

n→∞

lim ρ

α−η 2 −1 n

θ

^±_n

n

X

k=1

γ

_k

θ

^±_k

∆M

_k

= 0.

Together with (12) and (13) this implies

• lim

n→∞

Y

_n

= (1 − p

_A

)/π a.s., if

^α−η₂

≤ 0,

• lim

n→∞

Y

_n

ρ

α−η

n2

= 0 a.s., if

^α−η₂

> 0.

We obviously have P _(E

_α

_{) = 1 for α} = 1. We deduce from the previous argument that if P _(E

_α

_{) = 1 and}

^α−η

2

> 0, then P _(E

_α′

) = 1, with α

^′

=

^α−η₂

− 1. Set α

0

= 1 and α

_k+1

=

^αk₂^−η

− 1. If α

₀

≤ η, we have lim

n→∞

Y

_n

= (1 − p

_A

)/π a.s. on E

_α0

. If α

₀

> η, let j be the largest integer such that α

_j

> η (note that j exists because lim

k→∞

α

_k

< 0). We have P _(E

_α

j+1

) = 1, and, on E

_αj+1

, lim

n→∞

Y

_n

= (1 − p

_A

)/π a.s., because

^αj₂^−η

≤ 0.

_♦

We now turn to the proof of Lemma 3 which is based on the following classical martin- gale inequality (see [13], remark 1, p.14 for a proof in the case of i.i.d. random variables:

the extension to bounded martingale increments is straightforward).

Lemma 4 (Bernstein’s inequality for bounded martingale increments) Let (Z

_i

)

_1≤i≤n

be a

finite sequence of square integrable random variables, adapted to the filtration ( F

i

)

_1≤i≤n

,

such that

1. E _(Z

_i

_{| F}

_i−1

_{) = 0, i} = 1, . . . , n, 2. E _(Z

²

i

| F

i−1

) ≤ σ

_i²

, i = 1, . . . , n, 3. | Z

_i

| ≤ ∆

_n

, i = 1, . . . , n,

where σ

₁²

, . . . , σ

_n²

, ∆

_n

are deterministic positive constants.

Then, the following inequality holds:

∀ λ > 0, P

n

X

i=1

Z

_i

≥ λ

!

≤ 2 exp



 − λ

²

2 b

²_n

+ λ

^∆₃ⁿ



 ,

with b

²_n

= ^P

ⁿ_i=1

σ

_i²

.

We will also need the following technical result.

Lemma 5 Let (θ

n

)

n≥1

be a sequence of positive numbers such that θ

n

= ^Q

ⁿ_k=1

(1 − pγ

_k

), for some p ∈ (0, 1) and let (ξ

_n

)

_n≥1

be a sequence of non-negative numbers satisfying

γ

_n

ξ

_n

− γ

_n−1

ξ

_n−1

= o(γ

_n²

ξ

_n

).

We have

n

X

k=1

γ

_k²

ξ

_k

θ

_k²

∼ γ

_n

ξ

_n

2pθ

_n²

.

Proof: First observe that the condition γ

_n

ξ

_n

− γ

_n−1

ξ

_n−1

= o(γ

_n²

ξ

_n

) implies γ

_n

ξ

_n

∼ γ

_n−1

ξ

_n−1

and that, given ε > 0, we have, for n large enough,

γ

n

ξ

n

− γ

n−1

ξ

n−1

≥ − εγ

_n²

ξ

n

≥ − εγ

_n−1

γ

_n

ξ

_n

,

where we have used the fact that the sequence (γ

_n

) is non-increasing. Since γ

_n

ξ

_n

∼ γ

_n−1

ξ

_n−1

, we have, for n large enough, say n ≥ n

₀

,

γ

n

ξ

n

≥ γ

n−1

ξ

n−1

(1 − 2εγ

n−1

).

Therefore, for n > n

0

,

γ

_n

ξ

_n

≥ γ

_n0

ξ

_n0

n

Y

k=n0+1

(1 − 2εγ

_k−1

).

From this, we easily deduce that lim

n→∞

γ

_n

ξ

_n

/θ

_n

= ∞ and that ^P

_n

γ

_n²

ξ

_n

/θ

_n²

= ∞ . Now, from

1 θ

²_k

− 1

θ

²_k−1

= 2γ

_k

p − γ

_k²

p

²

θ

_k²

, we deduce (recall that lim

n→∞

γ

n

= 0) γ

_k²

ξ

_k

θ

_k²

∼ γ

_k

ξ

_k

2p

1 θ

_k²

− 1

θ

_k−1²

!

,

and, since ^P

_n

γ

_n²

ξ

_n

/θ

_n²

= ∞ ,

n

X

k=1

γ

_k²

ξ

_k

θ

²_k

∼ 1 2p

n

X

k=1

γ

_k

ξ

_k

1 θ

_k²

− 1

θ

_k−1²

!

= 1

2p

γ

n

ξ

n

θ

²_n

+

n

X

k=1

(γ

_k−1

ξ

_k−1

− γ

_k

ξ

_k

) 1 θ

_k−1²

!

= γ

_n

ξ

_n

2pθ

²_n

+ o

n

X

k=1

γ

_k²

ξ

_k

θ

²_k

! ,

where, for the first equality, we have assumed ξ

₀

= 0, and, for the last one, we have used

again γ

_n

ξ

_n

− γ

_n−1

ξ

_n−1

= o(γ

_n²

ξ

_n

).

_♦

Proof of Lemma 3: Given µ > 0, let

ν

_µ

= inf { n ≥ 0 | ρ

^α_n

Y

_n

> µ } . Note that { sup

_n

ρ

^α_n

Y

_n

< ∞} = ^S

_µ>0

{ ν

_µ

= ∞} .

On the set { ν

_µ

= ∞} , we have

n

X

k=1

γ

_k

θ

_k

∆M

_k

=

n

X

k=1

γ

_k

θ

_k

1

_{k≤νµ}

∆M

_k

. We now apply Lemma 4 with Z

_i

=

^γ_θⁱ

i

1

_{i≤νµ}

∆M

_i

. We have, using Proposition 4, E _(Z

_i²

_{| F}

_i−1

_{) =} ^γ

i2

θ

_i²

1

_{i≤νµ}

E _(∆M

_i²

_{| F}

_i−1

₎

≤ γ

_i²

θ

_i²

1

_{i≤νµ}

p

_A

ρ

i−1

Y

i−1

+ ρ

²_i

(1 − p

_B

)

≤ γ

_i²

θ

_i²

p

_A

ρ

^1−α_i−1

µ + ρ

²_i

(1 − p

_B

) ,

where we have used the fact that, on { i ≤ ν

_µ

} , ρ

^α_i−1

Y

_i−1

≤ µ. Since lim

n→∞

ρ

_n

= 0 and

n→∞

lim ρ

_n

/ρ

_n−1

= 1 (which follows from ρ

_n

− ρ

_n−1

= o(γ

_n

ρ

_n

)), we have E _(Z

_i²

_{| F}

_i−1

_{) ≤ σ}

_i²

_,

with σ

_i²

= C

_µ^γi2ρ_θ^1−αi2 i

, for some C

_µ

> 0, depending only on µ. Using Lemma 5, we have

n

X

i=1

σ

_i²

∼ C

_µ

γ

n

ρ

^1−α_n

2pθ

²_n

.

On the other hand, we have, because the jumps ∆M

_i

are bounded,

| Z

_i

| ≤ C γ

_i

θ

i

,

for some C > 0. Note that

_γ ^γk/θk

k−1/θk−1

=

_γ ^γk

k−1(1−pγk)

, and, since γ

_k

− γ

_k−1

= o(γ

_k²

) (take β = 0 in (10)), we have, for k large enough, γ

_k

− γ

_k−1

≥ − pγ

_k

γ

_k−1

, so that γ

_k

/γ

_k−1

≥ 1 − pγ

_k

, and the sequence (γ

_n

/θ

_n

) is non-increasing for n large enough. Therefore, we have

sup

1≤i≤n

| Z

_i

| ≤ ∆

_n

,

with ∆

n

= Cγ

n

/θ

n

for some C > 0. Now, applying Lemma 4 with λ =λ

0

ρ

¹⁻

α−η

n 2

/θ

n

, we get P _θ

_n

n

X

k=1

γ

_k

θ

_k

1

_{k≤νµ}

∆M

_k

≥ λ

0

ρ

¹⁻

α−η

n 2

!

≤ 2 exp



 − λ

²₀

ρ

^2−α+η_n

2θ

²_n

b

²_n

+ 2λ

₀

θ

_n

ρ

¹⁻

α−η

n 2 ∆n

3





≤ 2 exp



 − C

₁

ρ

^2−α+η_n

C

2

γ

n

ρ

^1−αn

+ C

3

γ

n

ρ

¹⁻

α−η

n 2





≤ 2 exp − C

₄

ρ

^1+η_n

γ

n

! ,

where the positive constants C

₁

, C

₂

, C

₃

and C

₄

depend on λ

₀

and µ, but not on n.

Using (11) and the Borel-Cantelli lemma, we conclude that, on { ν

_µ

= ∞} , we have, for n large enough,

θ

_n

n

X

k=1

γ

_k

θ

_k

∆M

_k

< λ

₀

ρ

¹⁻

α−η

n 2

, a.s.,

and, since λ

₀

is arbitrary, this completes the proof of the Lemma.

_♦

3 Weak convergence of the normalized algorithm

Throughout this section, we assume (in addition to the initial conditions on the sequence (γ

_n

)

_n∈N

)

γ

_n²

− γ

_n−1²

= o(γ

_n²

) and γ

n

ρ

_n

= g + o(γ

_n

), (16) where g is a positive constant. Note that a possible choice is γ

_n

= ag/ √

n and ρ

_n

= a/ √ n, with a > 0.

Under these conditions, we have ρ

_n

− ρ

_n−1

= o(γ

_n²

), and we can write, as in the beginning of Section 2,

Y

_n+1

= Y

_n

(1 + γ

_n+1

ε

_n

− π

_n

γ

_n+1

X

_n

) − γ

_n+1

κ(X

_n

) − γ

_n+1

ρ

n+1

∆M

_n+1

, (17) where lim

n→∞

ε

n

= 0 and lim

n→∞

π

n

= π. As observed in Remark 2, we know that, under the

assumptions (16), the sequence (Y

_n

)

_n≥1

is tight. We will prove that it is convergent in

distribution.

Theorem 4 Under conditions (16), the sequence (Y

_n

= (1 − X

_n

)/ρ

_n

)

_n∈N

converges weakly to the unique stationary distribution of the Markov process on [0, + ∞ ) with generator L defined by

Lf (y) = p

_B

y f (y + g) − f (y)

g + (1 − p

_A

− p

_A

y)f

^′

(y), y ≥ 0, (18) for f continuously differentiable and compactly supported in [0, + ∞ ).

The method for proving Theorem 4 is based on the classical functional approach to central limit theorems for stochastic algorithms (see Bouton [2], Kushner [10], Duflo [6]). The long time behavior of the sequence (Y

_n

) will be elucidated through the study of a sequence of continuous-time processes Y

⁽ⁿ⁾

= (Y

_t⁽ⁿ⁾

)

t≥0

, which will be proved to converge weakly to the Markov process with generator L. We will show that ν has a unique stationary distribution, and that this is the weak limit of the sequence (Y

_n

)

_n∈N

.

The sequence Y

⁽ⁿ⁾

is defined as follows. Given n ∈ N _{, and t ≥ 0, set}

Y

_t⁽ⁿ⁾

= Y

_N(n,t)

, (19)

where

N(n, t) = min (

m ≥ n |

m

X

k=n

γ

_k+1

> t )

,

so that N (n, 0) = n, for t ∈ [0, γ

n+1

), and, for m ≥ n + 1, N (n, t) = m if and only if P

m

k=n+1

γ

_k

≤ t < ^P

^m+1_k=n+1

γ

_k

.

Theorem 5 Under the assumptions of Theorem 4, the sequence of continuous time pro- cesses (Y

⁽ⁿ⁾

)

_n∈N

converges weakly (in the sense of Skorokhod) to a Markov process with generator L.

The proof of Theorem 5 is done in two steps: in section 3.1, we prove tightness, in section 3.2, we characterize the limit by a martingale problem.

3.1 Tightness

It follows from (17) that the process Y

⁽ⁿ⁾

admits the following decomposition:

Y

_t⁽ⁿ⁾

= Y

_n

+ B

_t⁽ⁿ⁾

+ M

_t⁽ⁿ⁾

, (20) with

B

⁽ⁿ⁾_t

= −

N(n,t)

X

k=n+1

γ

_k

[κ(X

_k−1

) + (π

_k−1

X

_k−1

− ε

_k−1

)Y

_k−1

] and

M

_t⁽ⁿ⁾

= −

N(n,t)

X

k=n+1

γ

_k

ρ

_k

∆M

_k

.