The renormalization transformation for two-type branching models

(1)

www.imstat.org/aihp 2008, Vol. 44, No. 6, 1038–1077

DOI: 10.1214/07-AIHP143

The renormalization transformation for two-type branching models

D. A. Dawson

^a

, A. Greven

^b

, F. den Hollander

^c,d

Rongfeng Sun

^d,e

, and J. M. Swart

^f

aSchool of Mathematics and Statistics, Carleton University, Ottawa K1S 5B6, Canada. E-mail: [email protected] bMathematisches Institut, Universität Erlangen–Nürnberg, Bismarckstraße 1 1/2, D-91054 Erlangen, Germany.

E-mail: [email protected]

cMathematical Institute, Leiden University, P.O. Box 9512, 2300 RA Leiden, the Netherlands. E-mail: [email protected] dMA 7-5, Fakultät II – Institut für Mathematik, TU Berlin, Straße des 17. Juni 136, 10623 Berlin. E-mail: [email protected]

eEURANDOM, P.O. Box 513, 5600 MB Eindhoven, the Netherlands

fÚTIA, Pod vodárenskou vˇeží 4, 18208 Praha 8, Czech Republic. E-mail: [email protected] Received 20 October 2006; revised 16 May 2007; accepted 5 September 2007

Abstract. This paper studies countable systems of linearly and hierarchically interacting diffusions taking values in the positive quadrant. These systems arise in population dynamics for two types of individuals migrating between and interacting within colonies. Their large-scale space–time behavior can be studied by means of a renormalization program. This program, which has been carried out successfully in a number of other cases (mostly one-dimensional), is based on the construction and the analysis of a nonlinear renormalization transformation, acting on the diffusion function for the components of the system and connecting the evolution of successive block averages on successive time scales. We identify a general class of diffusion functions on the positive quadrant for which this renormalization transformation is well defined and, subject to a conjecture on its boundary behavior, can be iterated. Within certain subclasses, we identify the fixed points for the transformation and investigate their domains of attraction.

These domains of attraction constitute the universality classes of the system under space–time scaling.

Résumé. Cet article étudie des systèmes dénombrables de diffusions en interaction hiérarchiques et linéaires vivant dans le quadrant positif. De tels systèmes apparaissent dans la dynamique d’individus de deux types qui migrent tout en interagissant dans des colonies. Le comportement à grande échelle et temps long peut être étudié en utilisant le programme de renormalisation. Ce programme, qui a permis de résoudre d’autres cas (principalement uni-dimensionnels) est basé sur la construction et l’analyse d’une transformation de renormalisation non linéaire, agissant sur la fonction de diffusion des composants du système et connectant l’évolution de blocs moyennés sur le temps à différentes échelles. Nous identifions une classe générale de fonctions de diffusion dans le quadrant positif pour lequel la transformation de renormalisation est bien définie et qui, sous une conjecture de comportement aux bords, peut-être itérée. À l’intérieur de certaines sous-classes, nous identifiens les points fixes de la transformation et étudions leurs domaines d’attraction. Ces domaines d’attraction constitutent les classes d’universalité du système après changement d’échelle dans le temps et l’espace.

MSC:60J60; 60J70; 60K35

Keywords:Interacting diffusions; Space–time renormalization; Two-type populations; Independent branching; Catalytic branching; Mutually catalytic branching; Universality

(2)

1. Introduction

1.1. Model and background

We are interested in the following system of coupled stochastic differential equations (SDE):

dXη,i(t )=

ξ∈ΩN

a_N(ξ, η)

X_ξ,i(t )−X_η,i(t ) dt+

2gi

X_η(t )

dBη,i(t ), η∈Ω_N, i=1,2. (1.1)

Herea_N(·,·)is the transition rate kernel of a random walk onΩ_N, the hierarchical group (or lattice) of orderN (see (1.3)),{ X_η}η∈ΩN withX_η=(Xη,1, Xη,2)is a family of diffusions taking values in[0,∞)²,g=(g1, g2)is a pair of diffusion functions on[0,∞)², and{ B_η}η∈ΩN withB_η=(Bη,1, Bη,2)is a family of independent standard Brownian motions onR². As the initial condition, we take

Xη(0)= θ=(θ1, θ2)∈ [0,∞)² ∀η∈ΩN. (1.2) Equation (1.1) arises as the continuum limit of discrete models inpopulation dynamics. In these models, individuals live in colonies labeled by the hierarchical groupΩN. Each colonyη∈ΩNconsists of two types of individuals, whose total masses are represented by the vectorXη. Individualsmigratebetween colonies according to the migration kernel a_N(·,·). At each colony, each individual undergoesbranchingat a rate that depends on the total masses of the two types of individuals present at that colony. The system in (1.1) arises in the so-called “small-mass–fast-branching”

limit, where the number of individuals in each colony tends to infinity, the mass of each individual tends to zero, and the effective branching rate grows proportionally to the number of individuals in each colony. The drift term in (1.1) arises from the migration, which is the only source of interactionbetweencolonies. The diffusion term in (1.1) arises from the branching, whereg_i(x)/x_i is the state-dependent branching rate of theith type, which incorporates the interaction between individualswithina colony. For more background, see, e.g. [7,9,16,27], Chapters 9 and 10 in [19].

The goal of the present paper is to study the universality classes of the large-scale space–time behavior of (1.1). It turns out that, for the specific form of the migration kernela_N(·,·)given by (1.5) and in the limit asN→ ∞, (1.1) is susceptible to a renormalization analysis. Therenormalization programfor hierarchically interacting diffusions was introduced by Dawson and Greven [10,11] for diffusions taking values in[0,1]. It has since been extended to several other state spaces (see [22,23] for an overview). We will give more detailed references in Section 1.3. First we outline the main ingredients of the renormalization program.

1.2. Renormalization program

The lattice in (1.1) is thehierarchical groupof orderN, which is defined as Ω_N=

η=(η_i)_i_∈N∈ {0,1, . . . , N−1}^N:

i∈N

η_i<∞ , (1.3)

with coordinatewise addition moduloN. Define a shiftφ:Ω_N→Ω_Nby(φη)_i:=η_i₊₁(i∈N). OnΩ_N, thehierar- chical distanceis defined as

d(η, ξ )=min

k∈N0=N∪ {0}: φ^kη=φ^kξ

, (1.4)

which is anultrametric, i.e.,d(η, ξ )≤d(η, ζ )∨d(ξ, ζ )for allη, ξ, ζ ∈Ω_N. We choose the random walk transition rate kernel in such a way thata_N(ξ, η)depends only on the hierarchical distance betweenξ andη. In view of what follows, we writea_N in the form

aN(ξ, η)=

k≥d(ξ,η)

ck−1N¹⁻^2k, ξ, η∈ΩN, ξ=η, (1.5)

(3)

where(c_n)_n_∈N₀ is a sequence of positive constants. Formula (1.5) says that the random walk associated witha_N(·,·) jumps with rateck−1/N^k⁻¹fromηto an arbitrary site in thek-block{ξ∈Ωn: φ^kξ=φ^kη}aroundη.

The key objects in the renormalization analysis are thek-block averages:

Y_η,i^[^k^](t )= 1 N^k

ξ∈Ω_N φ^kξ=η

X_ξ,i(t ), η∈Ω_N, i=1,2, k∈N0. (1.6)

Using (1.5), we may rewrite (1.1) as dXη,i(t )=

k≥1

c_k₋₁ N^k⁻¹

Y^[^k^]

φ^kη,i(t )−Xη,i(t ) dt+

2gi

Xη(t )

dBη,i(t ), η∈ΩN, i=1,2, (1.7)

where each componentX_ηfeels a drift towards the successive averages ofk-blocks containingη. It can be seen that the evolution of the 1-block averages is described in law by the SDE

dY_η,i^[¹^](t N )=

k≥1

c_k N^k⁻¹

Y^[^k⁺¹^]

φ^kη,i(t N )−Y_η,i^[¹^](t N ) dt

+ 2 N

ξ∈Ω_N φξ=η

g_iX_ξ(t N )

dBη,i(t ), η∈Ω_N, i=1,2, (1.8)

whereB_η=(Bη,1, Bη,2)is a family of independent standard two-dimensional Brownian motions. Note that in the limitN → ∞, we expect both the drift and the diffusion term in (1.8) to be of order one, which means thatY_η^[¹^] evolves on the time scalet N.

Let us next seeheuristicallywhat happens if we letN→ ∞, the so-calledhierarchical mean-field limit. If we let N→ ∞in (1.7), then the only drift term that survives is

c0

Y_φη,i^[¹^] (t )−X_η,i(t ) dt.

Furthermore,Y_φη^[¹^](t )→ X₍_·₎(0)≡ θ for allt≥0, because Y_φη^[¹^] evolves on the time scalet N. Therefore the system { X_η(t )}η∈Ω_N converges in law to anindependentsystem of diffusions, each satisfying the autonomous SDE

dZi(t )=c0(θ_i−Z_i)dt+ 2gi

Z(t )

dBi(t ), i=1,2. (1.9)

This kind of behavior is frequently referred to as the “McKean–Vlasov limit” and “propagation of chaos.”

With the above fact in mind, we move one step up in the hierarchy. SinceXξ(t )evolves on the time scalet, for each fixedt the family

X_ξ(t N )

ξ∈ΩN

φξ=η

(1.10) decouples and converges almost instantly to the equilibrium distribution of (1.9) with the drift towardsθreplaced by a drift towards the first block averageY_η^[¹^](t N ). Thus, we expect that

1 N

ξ∈Ω_N φξ=η

g_iX_ξ(t N )

∼

[0,∞)²

Γ^c⁰^,g

Yη^[1](t N )(dx)g _i(x) asN→ ∞for fixedt, η∈Ω_N, i=1,2, (1.11)

whereΓ^c⁰^,g

θ denotes theequilibrium distributionof (1.9). Thus, if we set (F_c₀g)_i(θ )=

[0,∞)²

Γ^c⁰^,g

θ (dx)g_i(x), i=1,2,θ∈ [0,∞)², (1.12)

(4)

then by (1.11), for largeN, the SDE (1.8) for the 1-block averagesY_η^[¹^]takes exactly the same form as the SDE (1.7) for the single components, provided that we rescale time by a factorN and replace the single component diffusion functionsg_i by(F_c₀g)_i (i=1,2). Here,F_c₀ plays the role of arenormalization transformationacting on the pair of diffusion functionsg=(g1, g2).

We caniteratethe above procedure. The upshot of this is that, asN→ ∞, thek-block averagesYη^[^k^]evolve on the time scalet N^kaccording to the SDE

dZ_i^[k](t )=c_k

θ_i−Z^[k]_i (t ) dt+

2 F^[^k^]g

i

Z^[^k^](t )

dBi(t ), i=1,2, (1.13)

with diffusion functionsF^[^k^]g=(F^[^k^]g₁, F^[^k^]g₂)given by

F^[^k^]g=Fc_k−1◦ · · · ◦Fc₀g, k∈N0. (1.14) In fact, putting the successive iterates together and observing the sequence of block averages

Y^[k]

φ^kη

sN^k ,Y^[k−¹^]

φ^k−1η

sN^k

, . . . ,Y_η^[⁰^] sN^k

(1.15) on the time scalesN^k, asN→ ∞, we expect this sequence to converge in distribution to abackward Markov chain

M( −k),M( −k+1), . . . ,M(0)

, (1.16)

the so-calledinteraction chain, where

(1) The starting positionM( −k)is distributed as the weak solution of (1.13) at timeswith initial conditionZ^[^k^](0)= θ;

(2) for 0≤j ≤k−1, the transition probability kernel fromM( −j−1)toM( −j )is given by PM( −j )∈dy| M(−j−1)= x

=Γ_x^c^j^,F^[j]^g(dy), (1.17)

whereΓ_x^c^j^,F^[j]^g(·)denotes the equilibrium distribution of (1.13) withkreplaced byj. The distribution of M(−k) depends on s because Y^[^k^]

φ^kη(sN^k)evolves on the time scalesN^k, while the transition probability kernel fromM( −j−1)toM( −j )for 0≤j≤k−1 is independent oftbecause, conditioned onY^[^j⁺¹^]

φ^j+¹η, Y^[^j^]

φ^jηequilibrates almost instantly on the time scalesN^k. Note that(F^[^k^]g)_i(θ )=E[g_i(M(0)) | M(−k)= θ], whereE denotes expectation with respect to the interaction chain.

With these heuristics in mind, therenormalization programconsists of the following two steps:

(I) Stochastic part:Show that for all scalesk∈N, in the hierarchical mean-field limitN→ ∞, the block average in (1.6) converges in law to the solution of the SDE in (1.13), and the sequence of block averages in (1.15) converges in law to the interaction chain in (1.16).

(II) Analytic part: Analyze the renormalization transformationFcand the iteratesF^[n],n∈N0.

Assuming that the stochastic part of the renormalization program can be completed, the large-scale space–time behavior of (1.1) in the limitN→ ∞is characterized by the behavior ofF^[ⁿ^]asn→ ∞, in particular, by itsfixed shapesand theiruniversality classes.

Here, by a fixed shape we mean a pair of diffusion functionsg=(g1, g2)such thatFcg=λg for somec, λ >0.

We speak of adowngoing fixed shape, fixed pointorupgoing fixed shapedepending on whetherλ <1,=1, or>1.

Note that since the factorλcan always be absorbed in time-scaling, such fixed shapes correspond to models that are mapped into themselves after a suitable rescaling of space and time. Indeed, if we setc_k=cλ^k (k≥0), then such a fixed shape satisfies F^[^k^]g=λ^kgbecause the SDE associated with(c_k, F^[^k^]g)is simply a time change of the SDE associated with (c, g), which induces the same renormalization transformation. For the interacting model in (1.7), this means that thek-block averages evolve on the time scalet N^kλ^k according to the diffusion functiong. We note that our definition of a fixed shape deviates from the definition used in some earlier work, e.g. [21]. What is called a

(5)

fixed shape there is, in our terminology, ajoint fixed shapefor allc >0, i.e., agsuch that for allc >0 there exists a λ=λ(c)withF_cg=λg.

By a universality class, we mean a setGof diffusion functions with the property that, given(ck)k∈N0, for eachg∈G there exist scaling constants(s_n)_n_∈N such thats_nF^[ⁿ^]gconverges to the same limit (possibly up to a multiplicative constant). Typically, the limit will be a fixed shape or anasymptotic fixed shape(for the latter, see [21]). Note that each joint fixed shape gives rise to a universality class, namely all models within a given universality class exhibit the same large-scale space–time behavior.

Apart from being relevant in the study of large-scale space–time behavior, fixed shapes also give rise to continuum models, by taking the so-calledhierarchical mean-field continuum limit, which is a spatial continuum limit of the hierarchical latticeΩ_N withN→ ∞. These continuum models also exhibit universality on small space–time scales, which is governed by the same renormalization transformationF_cand its iteratesF^[ⁿ^],n∈N0. For more details, see [7] and [14].

The large-scale space–time behavior of (1.1) depends both on the diffusion functiongand on the potential-theoretic properties of the random walk with transition rate kernel (1.5). Based on earlier work, we expect nontrivial universality classes to arise only when

n∈N0c⁻_n¹= ∞, which is the “necessary and sufficient” condition for the random walk with transition rate kernelaN(·,·)onΩN to be recurrent (except for a side condition that becomes irrelevant in the limitN→ ∞; see [27]). For linear systems such as (1.1), the recurrence of the random walk is usually associated withclustering; see e.g. [8,11,30]. In our context, clustering means that the solution of (1.1) converges in law to a mixture of distributions, each of which is concentrated on the configurationXη= x,η∈ΩN, for somex∈ [0,∞)² withg₁(x) =g₂(x) =0. The choice of(c_n)_n_∈N₀ determines the pattern of cluster formation, such as whether only small clusters appear, or only large clusters appear, or clusters of all scales appear. The latter is known asdiffusive clustering(see e.g. [11,20]).

With the above facts in mind, theanalytic partof the renormalization program can be more precisely formulated as follows.

1. Find classes of diffusion functions on which the renormalization transformationsF_cand their iteratesF^[ⁿ^],n∈N0, are well defined.

2. Determine all the (asymptotic) fixed shapes.

3. Determine the universality classes of diffusion functions that, for given(c_n)_n_∈N₀ and after appropriate rescaling, converge to these (asymptotic) fixed shapes, and determine the associated scaling constants.

1.3. Literature

Thefull renormalization program has been successfully carried out for hierarchically interacting diffusions taking values in:

(1) the compact interval[0,1][2,10,11], where the Wright–Fisher diffusion is the unique fixed shape and is globally attracting with a scaling that is independent of the diffusion function;

(2) the halfline[0,∞)[3,12], where the Feller branching diffusion is the unique fixed point and is globally attracting with a scaling that depends on the asymptotic behavior of the diffusion function at infinity.

For higher-dimensional diffusions, theanalytic parthas been carried out for:

(3) isotropic diffusions taking values in a compact convex subset ofR^d [24,31], where the diffusion function with constant curvature is the unique fixed shape and is globally attracting with a scaling that is independent of the diffusion function;

(4) a class of probability-measure-valued diffusions [13,15], where the Fleming–Viot process is the unique fixed shape and is globally attracting with a scaling that is independent of the diffusion function;

(5) a class of catalytic Wright–Fisher diffusions taking values in[0,1]²[21], where the diffusion function for the first component is an autonomous Wright–Fisher diffusion and the diffusion function for the second component is an autonomous Wright–Fisher diffusion function multiplied by acatalyzing functiondepending only on the first component. The renormalization transformation effectively acts on the catalyzing function. There are four attracting shapes for the catalyzing function, depending on whether the initial catalyzing function is zero or strictly positive at the boundary points of[0,1], and these attracting shapes are globally attracting with a scaling that is independent of the catalyzing function.

(6)

Thestochastic partfor higher-dimensional diffusions has only been completed for interacting Fleming–Viot processes [13] and for mutually catalytic branching diffusions taking values in[0,∞)²[7].

All previous studies deal with diffusions that have certain simplifying properties. In the one-dimensional cases (1) and (2), as well as in the two-dimensional case (5), the equilibrium of (1.9) is reversible. As a result, many explicit calculations can be performed that are crucial for the analysis. For certain diffusions with compact state space, which includes the cases (1), (3) and (4), there is a common underlying structure (called “invariant harmonics,” see [30]) that allows the determination of the unique fixed shape and its domain of attraction. In all cases where the state space is compact, the scaling needed for convergence to an attracting shape depends only on(c_n)_n_∈N₀, not on the diffusion functiong. This is different in case (2), where the state space is not compact. In all cases except case (5), the fixed shapes turn out to be joint fixed shapes for allc >0.

The goal of the present paper is to carry out theanalytic partof the renormalization program for ageneralclass of branching diffusions taking values in[0,∞)². The multi-dimensionality and the noncompactness of the state space pose significant challenges. Due to the multidimensionality, the well-definedness of the renormalization transformation is nontrivial. The structure of the fixed points/shapes turns out to be rather rich. In fact, we will prove that, under certain restrictions, the class of fixed points is a4-parameter family of diffusions with independent branching, catalytic branching and mutually catalytic branching as the extremal fixed points, and they are joint fixed points ofFcfor all c >0. Moreover, we will prove that all diffusion functions that are comparable to these fixed points in an appropriate sense fall in their domains of attraction.

1.4. Outline

The rest of the paper is organized as follows. In Section 2 we formulate our main results, which come with varying degrees of restrictions on the diffusion functions. Section 3 contains the proof of the ergodicity of the SDE (1.9), and basic properties of the renormalization transformation. Section 4 proves the identification of fixed points/shapes.

Sections 5 and 6 identify the domains of attraction for the fixed points. In Appendices A and B we collect some technical results needed for the proofs.

2. Main results

In Section 2.1, we formulate a key class of diffusion functionsC, for which the SDE (1.9) has a unique weak solution.

Section 2.2 contains a theorem on the ergodicity of the SDE (1.9), defines the renormalization transformation, formu- lates a subclassH0+⊂Con which the renormalization transformation is well defined and, subject to a conjecture on the preservation of certain boundary properties, can be iterated. Section 2.3 gives the definition of certain generalized fixed points/shapes, and identifies some special fixed points/shapes. Section 2.4 contains results on the identification of fixed points/shapes inH0+ under additional regularity assumptions. Section 2.5 contains our main result on the domains of attraction to the fixed points under further assumptions. Lastly, Section 2.6 provides a brief discussion of these results and lists some future challenges.

2.1. Key class and uniqueness for the autonomous SDE

The renormalization transformationFc is based on (1.9), which is the SDE for the vectorX(t ) =(X1(t ), X2(t ))∈ [0,∞)²written out as

dX1(t )=c

θ₁−X₁(t ) dt+

2g1

X₁(t ), X₂(t ) dB1(t ), dX2(t )=c

θ2−X2(t ) dt+

2g2

X1(t ), X2(t )

dB2(t ), (2.1)

wherec >0,θ=(θ1, θ2)∈ [0,∞)², andB(t ) =(B1(t ), B2(t ))are independent standard Brownian motions onR². The corresponding generator is

L^c,g

θ f (x) =c

2 i=1

(θ_i−x_i) ∂

∂x_if (x) + 2 i=1

g_i(x) ∂²

∂x²_i f (x), f∈C_c²

[0,∞)²

. (2.2)

(7)

Note that, due to the absence of mixed partial derivatives,L^c,g

θ can be interpreted as the generator of a two-type branching diffusionwith state-dependent branching ratesg_i(x)/x _i (i=1,2).

Abbreviate

A₁= [0,∞)× {0}, A₂= {0} × [0,∞). (2.3) We will say that a functionf: [0,∞)²→ [0,∞)hasboundary property

(∂1) if lim

x→y

f (x)

x₁ =γ (y) ∀y∈A1∪A2withγ continuous and >0 onA1∪A2, (∂₂) if lim

x→y

f (x)

x2 =γ (y) ∀y∈A₁∪A₂withγ continuous and >0 onA₁∪A₂, (∂12) if lim

x→y

f (x)

x₁x₂ =γ (y) ∀y∈A1∪A2withγcontinuous and >0 onA1∪A2. (2.4) Throughout the paper, the pairg=(g1, g2)will be assumed to be in the following class.

Definition 2.1 (ClassC). LetCbe the class of functionsg(x) =(g1(x), g 2(x)) satisfying:

(i) Fori=1,2,gi is continuous on[0,∞)²and>0on(0,∞)². (ii) Fori=1,2,gi satisfies boundary property(∂i)or(∂12).

Note that for(g₁, g₂)∈Cwe can writeg_i(x) =x_iγ_i(x) org_i(x) =x₁x₂γ_i(x) for some positive continuous function γi on[0,∞)², depending on whethergi satisfies boundary property(∂i)or(∂12). Note also thatg1andg2vanish on A2, respectively,A1, which is necessary to guarantee that the diffusion stays within[0,∞)². Thus, if we denote the effective boundaryofgby

∂g=

x∈ [0,∞)²: g1(x) =g2(x) =0

, (2.5)

then∂gcan be either of the following:

A₁∩A₂, A₁, A₂, A₁∪A₂. (2.6)

These boundary constraints allow for the system (2.1) to be treated as a perturbation of either of the following diffusions:

(1) Independent branching:(g1, g2)=(b1x1, b2x2),b1, b2>0, ∂g=A1∩A2.

(2) Catalytic branching: either (g₁, g₂) =(b₁x₁, c₂x₁x₂), b₁, c₂ >0, ∂g =A₂; or (g₁, g₂) =(c₁x₁x₂, b₂x₂), c₁, b₂>0, ∂g=A₁.

(3) Mutually catalytic branching:(g₁, g₂)=(c₁x₁x₂, c₂x₁x₂),c₁, c₂>0, ∂g=A₁∪A₂.

Such a perturbation is behind the following result of [1] and [6], which provides the starting point of our analysis. The latter paper improves results in [17], where Hölder continuity is assumed rather than continuity.

Theorem 2.2 (Well-posedness of martingale problem [1,6]). For allc >0, g∈C,θ∈ [0,∞)²andx∈ [0,∞)², with the possible exception of the case whenx=(0,0),θ∈(0,∞)²,and eitherg₁org₂satisfies boundary property (∂12),the martingale problem associated with the generator in(2.2)has a unique solution with starting positionx.

As a consequence of Theorem 2.2, the SDE (2.1) has a unique weak solution for allθ∈ [0,∞)²andx∈ [0,∞)², with the possible exception of the case whenx=(0,0),θ∈(0,∞)², and eitherg₁org₂satisfies boundary property (∂12). For each fixedθ∈ [0,∞)², the SDE (2.1) defines a Feller process satisfying the strong Markov property (see e.g. Theorem 4.4.2 in [19] and Corollary 11.1.5 in [29]).

(8)

Remark 1. Whenθ∈(0,∞)²,g∈C,g₁andg₂satisfy(∂₁),resp.(∂₂),the well-posedness of the martingale problem was established in[1]for all initial conditionsx∈ [0,∞)².Whenθ∈(0,∞)²,g∈C,andg₁, g₂both satisfy(∂₁₂), the well-posedness is established in [6] for all initial condition x∈ [0,∞)²\{(0,0)}. Both [1] and [6] use local perturbation arguments and the results are not restricted to linear drift as considered here.Since the perturbation arguments are local,this implies that well-posedness also holds for mixed boundaries, i.e., g₁ satisfies (∂₁)and g₂ satisfies (∂₁₂), or vice versa. When eitherg₁ or g₂ satisfies (∂₁₂),Lemma 35 of [17]shows that,for all x∈ [0,∞)²\(0,0),with probability1 the unique weak solution of(2.1)with initial conditionx never hits(0,0),and hence we can restrict the state space to[0,∞)²\{(0,0)}.Whenθ∈∂[0,∞)²,the local analysis of[1]and[6]still applies until the diffusion first hits the absorbing boundary,at which time the diffusion becomes one-dimensional, a situation for which the well-posedness of the martingale problem is standard.

Remark 2. The proof given in[1]requires the drift to be strictly positive in each component on∂[0,∞)².However,as pointed out in[5],it is sufficient that the inward normal component of the drift is strictly positive on∂[0,∞)²,which holds in our setting whenθ∈(0,∞)².

Remark 3. It would be considerably more difficult to deduce from Theorem2.2the well-posedness of the martingale problem for the system(1.1),for which one would need to restrict the state space.To deduce the Feller property,one would need to restrict the state space even further and impose growth conditions on the diffusion functiong,typically g1(x) +g2(x) =O(x₁²+x₂²)(see,e.g., [7,28]).We will not resolve these issues here,since they belong to the stochastic part of the renormalization program,which remains open.

2.2. Equilibrium distribution and renormalization transformation

Our first result shows that (2.1) has a unique equilibrium for the class C. The proof will be given in Section 3.1.

HenceforthLdenotes law.

Theorem 2.3 (Equilibrium distribution). For allg∈C, θ∈ [0,∞)² and c >0, (2.1)has a unique equilibrium distributionΓ^c,g

θ ,which is continuous inθwith respect to weak convergence of probability measures,and LX(t )

t⇒→∞Γ^c,g

θ ∀ X(0)∈ [0,∞)². (2.7)

The convergence in (2.7) is crucial for thestochastic partof the renormalization program (not considered here), while the uniqueness of the equilibrium is crucial for the definition of therenormalization transformation, which we now define.

Definition 2.4 (Renormalization transformation). The renormalization transformationF_c,acting ong∈C,is defined as

(F_cg)_i(θ )=

[0,∞)²

g_i(x)Γ ^c,g

θ (dx), θ∈ [0,∞)², c >0, i=1,2. (2.8) Henceforth we will denote expectation with respect toΓ^c,g

θ byE^c,g_θ .

Without restrictions on the growth ofgat infinity, it is possible thatFcgis infinite. We therefore need to consider a tempered subclass ofC.

Definition 2.5 (ClassH0⁺).

(i) Fora≥0,letHa⊂Cbe the class of allg∈Csatisfying g₁(x₁, x₂)+g₂(x₁, x₂)≤C(1+x₁)(1+x₂)+a

x₁²+x₂²

, (x₁, x₂)∈ [0,∞)², (2.9) for some0< C=C(g) <∞.

(9)

(ii) Let H0⁺=

a>0

Ha. (2.10)

Note thatH0⁺ is much larger thanH0. In particular,H0⁺ includes diffusion functions that along the axes grow faster than linear but slower than quadratic.

Our second result shows thatF_cis well defined on the classHawhen 0≤a < c, preserves the effective boundary, and preserves the growth bound in (2.9) though with a different coefficient. The proof will be given in Section 3.2.

Theorem 2.6 (Finiteness, continuity, preservation of∂gand growth bound). Forc >0and0≤a < c,ifg∈Ha, thenFcgis finite and continuous on[0,∞)²,∂Fcg=∂g,andFcgsatisfies(2.9)withareplaced by_c₋^c_aa.

To proceed with our analysis, we need the following:

Conjecture 2.7 (Preservation of boundary properties). Letg∈H0⁺. (i) Fori=1,2,ifg_i satisfies(∂_i),then so does(F_cg)_i for allc >0.

(ii) Fori=1,2,ifg_i satisfies(∂12),then so does(F_cg)_ifor allc >0.

In Section 3.3 we will explain why this conjecture is plausible. Combining Theorem 2.6 with Conjecture 2.7, we get:

Corollary 2.8 (Preservation of classH0⁺). For allc >0,the classH0⁺is preserved underF_c,i.e.,F_cg∈H0⁺ for allg∈H0⁺.

The latter is a key property, because it allows us toiterateF_conH0+and investigate the orbitF^[ⁿ^]g=F_c_n₋₁◦ · · · ◦ Fc₀g,n∈N0. We will not need Conjecture 2.7 or Corollary 2.8 until we study the iteratesF^[ⁿ^]in Section 2.5.

The subquadratic growth bound imposed byH0⁺cannot be relaxed: we will see in Corollary 2.11 thatFccannot be iterated indefinitely onHafor anya >0.

2.3. Definition and examples of fixed points and fixed shapes

We next give the definition of fixed points and fixed shapes ofFc. Generalizing our definition given in the Introduction, we allow for the case whereF_cg=λgwithλnot a constant but a diagonal matrix. Thesegeneralized fixed shapesdo not give rise to universality classes as defined in Section 1.2, but they may be relevant for studying finer properties of the orbit(F^[ⁿ^]g)_n_∈N₀.

Definition 2.9 (Generalized fixed shapes and points). The pairg=(g₁, g₂)∈Hawitha∈ [0, c)is called a general- ized fixed shape ofF_cif

Fc(g1, g2)=(λ1g1, λ2g2) for someλ1, λ2>0. (2.11) Ifλ₁=λ₂,thengis called a fixed shape,and ifλ₁=λ₂=1,thengis called a fixed point ofF_c.

Our third result identifies a family of fixed points and (generalized) fixed shapes ofF_c. The proof is nontrivial because of integrability issues, and will be given in Section 3.2.

Theorem 2.10 (Examples of fixed points and fixed shapes).

(i) The pair

(g₁, g₂)=(b₁x₁+c₁x₁x₂, b₂x₂+c₂x₁x₂) (2.12)

is a fixed point ofFcinH0⁺ for allc >0and allb1, b2, c1, c2≥0with(b1+c1)(b2+c2) >0.

(10)

(ii) The pair (g1, g2)=

a1x₁²+b1x1+c1x1x2, a2x₂²+b2x2+c2x1x2

(2.13)

is a generalized fixed shape ofF_cinHa1∨a2 for allc >0, 0< a₁, a₂< candb₁, b₂, c₁, c₂≥0.The corresponding scaling constants are

λ1= c c−a1

, λ2= c

c−a2

. (2.14)

Diffusion functions of the form in (2.12) aremixturesof independent branching, catalytic branching and mutually catalytic branching (recall Section 2.1), all of which are in the classH0+. We will see in Theorem 2.15 that, under additional regularity conditions, such mixtures are theonlyfixed points ofF_c. Diffusion functions of the form in (2.13) are mixtures of these fixed points and the Anderson branching diffusion(g1, g2)=(a1x₁², a2x₂²). The latter do not fall in the classH0+.

The following corollary of Theorem 2.10 shows thatF_cg cannot be defined for allg∈Ha witha≥c, and F_c cannot be iterated indefinitely onHafor anya >0. The proof will be given in Section 3.2.

Corollary 2.11 (Divergence of iterated fixed shapes). Let gi(x) =αix²_i +βixi +γix1x2 with αi>0 and βi, γ_i≥0, i=1,2. Let (c_n)_n_∈N₀ be the positive sequence that definesF^[ⁿ^] (see(1.14)). Letn₀=min{n∈N: (α₁∨ α2)_n₋₁

i=0c⁻_i ¹≥1}.Then F^[ⁿ^]g

1, F^[ⁿ^]g

2

=

1 1−α1

_n₋₁

i=0c⁻_i ¹

g₁, 1

1−α2

_n₋₁

i=0c⁻_i ¹ g₂

, 0≤n < n₀, (2.15)

while(F^[ⁿ⁰^]g)1+(F^[ⁿ⁰^]g)2≡ ∞on(0,∞)². 2.4. Identification of fixed points and fixed shapes

Our fourth result rules out generalized fixed shapes inH0⁺ with an upgoing component. The proof will be given in Section 4.3.

Theorem 2.12 (No fixed shapes inH0+ with an upgoing component). Forc >0,there is nog∈H0⁺ such that either(F_cg)1=λ1g1withλ1>1or(F_cg)2=λ2g2withλ2>1.

Our fifth result does the same for generalized fixed shapes with a downgoing component, but only under mild additional regularity conditions. The proof will be given in Section 4.3. Below, in line with general topological notation, lim inf_x_→₍_∞_,_∞₎denotes the infimum of all limits along sequences tending to(∞,∞).

Theorem 2.13 (Sufficient conditions for no downgoing fixed shapes inH0⁺). Letc >0.

(i) There is nog∈H0+such thatFc(g1, g2)=(λ1g1, λ2g2)with0< λ1, λ2<1and lim inf

x→(∞,∞)

g₁(x)

x₁² +g₂(x) x₂²

=0. (2.16)

(ii) There is nog∈H0+such that(Fcg)1=λ1g1for some0< λ1<1andgsatisfies any of the following condi- tions:

• g₁>0 onA₁\ (0,0)

, (2.17)

• lim inf

x→(∞,∞)

g1(x) x1x2

>0. (2.18)

(11)

A similar result holds with the indices1and2interchanged.

Remark. Conditions(2.16)and(2.18)are complementary.Note that one particular case not covered by conditions (2.16)–(2.18)is wheng₁vanishes on both axes,g₁(x) =o(x1x₂)asx→(∞,∞),andg₂(x) =x₁x₂.In that case we cannot rule out the possibility ofg₁being a downgoing fixed shape.

In Theorem 2.10 we identified a 4-parameter family of fixed points. To show that these are the only fixed points, we need to impose strong additional regularity conditions.

Abbreviate R_∞=

(0,∞), (∞,0), (∞,∞)

(2.19) and

h(∞,0)(x)=x1, h(0,∞)(x)=x2, h(∞,∞)(x)=x1x2. (2.20) Definition 2.14 (ClassH^r₀). LetH^r₀be the set ofg∈H0satisfying

(i) inf

x∈[s,∞)²

g_i(x) > 0 ∀s >0, i=1,2, (2.21)

(ii) lim

x→z

g_i(x)

h_z(x) =λ_i,_z∈ [0,∞) ∀z∈R_∞, i=1,2. (2.22)

Note that H₀^r ⊂H0⊂H0⁺. Also note that, because g1 vanishes on A2 and g2 on A1, necessarily λ1,(0,∞)= λ_2,(∞,0)=0.

Our sixth result is the following. The proof will be given in Section 4.1.

Theorem 2.15 (Identification of fixed points inH^r₀). Letc >0andg=(g1, g2)∈H^r₀.IfFc(g1, g2)=(g1, g2),then g₁(x) =λ_1,(_∞_,0)x₁+λ_1,(_∞_,_∞₎x₁x₂,

g2(x)=λ2,(0,∞)x2+λ2,(∞,∞)x1x2, (2.23)

whereλ_i,_z,z∈R_∞,are defined in(2.22).

2.5. Domain of attraction of fixed points

Our seventh and final result is on the domain of attraction of the iterated mapsF^[ⁿ^]=F_c_n−₁◦ · · · ◦F_c₀,n∈N0, for a fixed positive sequence(c_n)_n_∈N₀. We show that, provided infn∈N0c_n>0 and

n∈N0c⁻_n¹= ∞, all diffusion functions that are comparable to a mixture of the fixed points fall into its domain of attraction. In Section 5, we will give the proof for the special casec_n≡c, while in Section 6, we prove the result for varyingc_n.

Theorem 2.16 (Domain of attraction of fixed points). Let(cn)_n∈N₀ be a sequence such thatinf_n∈N₀cn>0 and n∈N0c⁻_n¹= ∞.Letg∈H₀^r be such that

gi(x) ≥αixi+βix1x2, αi, βi≥0, αi+βi>0, i=1,2. (2.24) Then

nlim→∞

F^[n]g

i(θ )=

z∈R_∞

λ_i,_zh_z(θ ) ∀θ∈ [0,∞)², i=1,2, (2.25)

whereh_z,λ_i,_z,z∈R_∞,are defined in(2.20)and(2.22).