Generalization of Selberg’s 16 3 theorem and affine sieve

c 2011 by Institut Mittag-Leffler. All rights reserved

Generalization of Selberg’s ₁₆ ³ theorem and affine sieve

by

Jean Bourgain

Institute for Advanced Study Princeton, NJ, U.S.A.

Alex Gamburd

University of California at Santa Cruz Santa Cruz, CA, U.S.A.

Peter Sarnak

Institute for Advanced Study Princeton, NJ, U.S.A.

and Princeton University Princeton, NJ, U.S.A.

Dedicated to the memory of Atle Selberg.

1. Introduction

A celebrated theorem of Selberg [33] states that for congruence subgroups of SL₂(Z) there are no exceptional eigenvalues below ₁₆³. We prove a generalization of Selberg’s theorem for infinite index “congruence” subgroups of SL₂(Z). Consequently we obtain sharp upper bounds in the affine linear sieve, where in contrast to [3] we use an archimedean norm to order the elements.

Let Λ be a finitely generated non-elementary subgroup of SL2(Z). Let XΛ=Λ\H be the corresponding hyperbolic surface (which is of infinite volume if Λ is of infinite index in SL(2,Z)). Letδ(Λ) denote the Hausdorff dimension of the limit set of Λ. The generalization of Selberg’s theorem splits into two cases: δ(Λ)>¹₂ and 0<δ(Λ)6¹2.

In the case when δ(Λ)>¹₂ the discrete spectrum of the Laplace–Beltrami operator onL²(XΛ) consists of a finite number of points in

0,¹₄

(see [18]). We denote them by 06λ0(Λ)< λ1(Λ)6...6λmax(Λ)<¹₄.

The first author was supported in part by the NSF. The second author was supported in part by DARPA, NSF, and the Sloan Foundation. The third author was supported in part by the NSF.

The assumptionδ(Λ)>¹₂ is equivalent toλ0(Λ)<¹₄, and in this case δ(1−δ)=λ0[25].

The following extension of Selberg’s theorem is proved in§2.

Theorem1.1. Let Λbe a finitely generated subgroup of SL(2,Z)withδ(Λ)>¹₂. For q>1let Λ(q)be the “congruence” subgroup

{x∈Λ :x≡I modq}.

There is ε=ε(Λ)>0 such that

λ1(Λ(q))>λ0(Λ(q))+ε, for all square-free q>1 (note that λ0(Λ(q))=λ0(Λ)).

In [10] an explicit and stronger version of Theorem 1.1 is proven under the assump- tion thatδ(Λ)>⁵₆. See [30] for the sharpest known bounds towards Selberg’s ¹₄conjecture as well as bounds towards the Ramanujan conjectures for more general groups.

Theorem 1.1 is a consequence of Theorem 1.2 in [3] and the following result, which is of independent interest.

Theorem1.2. LetΛ=hSibe a finitely generated subgroup of SL(2,R)withδ(Λ)>¹₂. Let {Nj}j be a family of finite-index normal subgroups of Λ. Then the following are equivalent:

(i) the Cayley graphs G(Λ/Nj, S)form a family of expanders;

(ii) there is ε=ε(Λ)>0such that λ1(Λ/Nj)>λ0(Λ/Nj)+ε.

The argument in§2 establishes that (i)⇒(ii). The implication (ii)⇒(i) is proved us- ing Fell’s continuity of induction in [10,§7]. Theorem 1.2 generalizes results of Brooks [4]

and Burger [5], [6] who proved it in the case of co-compact Λ.

Combining Theorem 1.1 with Lax–Phillips theory of asymptotic distribution of lat- tice points [18], we obtain the following result, which is the crucial ingredient in the execution of the affine linear sieve in the archimedean norm.

Theorem 1.3. Let Λ be a finitely generated subgroup of SL(2,Z) with δ(Λ)>¹₂. Assume that q is square-free and (q, q0)=1,where q0 is provided by the strong approx- imation theorem [20]. There is ε1>0 depending on Λ such that for any g∈SL2(q) we have

|{γ∈Λ :kγk6T andγ≡g modq}|= cΛT^2δ

|SL2(q)|+O(q³T^2δ−ε1).

We now turn to the discussion of the caseδ(X)6¹2. In this case there is no discreteL² spectrum and its natural replacement is furnished by the resonances ofX, which are given

as the poles of the meromorphic continuation of the resolventRX(s)=(∆X−s(1−s))⁻¹. By the result of Patterson [25] and Sullivan [35],RX(s) is analytic for Res>δ. Mazzeo and Melrose [21] proved thatR_X(s) has a meromorphic continuation to the entire plane.

In [26] Patterson proved thatR_X(s) has a simple pole ats=δand no further poles on the line Res=δ. His proof is based on ideas from ergodic theory related to the Ruelle zeta function. Using further development of these ideas due to Dolgopyat [7], Naud [22] has recently established thatR_X(s) is holomorphic (with the exception of a simple pole at s=δ) for Res>δ−ε, withεdepending onX. The following result, giving a resonance-free region for congruence resolvent, is proved in§11.

Theorem1.4. Let Λbe a finitely generated subgroup of SL(2,Z)with δ(Λ)6¹2. For q>1 square-free let Λ(q)be the “congruence” subgroup

{x∈Λ :x≡I modq}.

Let X(q)=Λ(q)\H. There is ε=ε(Λ)>0 such that R_X(q)(s) is holomorphic (with the exception of a simple pole at s=δ)for

Res > δ−εmin

1, 1

log(1+|Ims|)

.

Whenδ6¹2 we cannot apply the expansion property [3] directly, instead, to prove Theorem 1.4 we use a dynamical treatment and invoke a generalization of the underlying result on measure convolution (“L²-flattening lemma”): see Lemmas 7.1 and 7.2 in §7.

It is likely that by combining our methods with the extension of Dolgopyat’s result [7]

to vector-valued functions, analyticity ofR_X(q)(s) can be established for Res>δ−ε, in complete analogy(¹) with Theorem 1.1.

Using methods of Lalley [16] we obtain the following analogue of Theorem 1.3, which is sufficient for sieving applications.

Theorem1.5. Let Λbe a finitely generated subgroup of SL(2,Z)with 0<δ(Λ)6¹2. Assume that q is square-free and (q, q₀)=1, where q₀ is provided by the strong approx- imation theorem [20]. There are ε₁>0 and C >0 depending on Λ such that for any g∈SL2(q) we have

|{γ∈Λ :kγk6T andγ≡g modq}|= cΛT^2δ

|SL2(q)|(1+O(T^{−1/log logT}))+O(q^CT^2δ−ε1).

(¹) The analogy between Theorems 1.1 and 1.4 becomes clearer when their assertions are expressed in terms of the Selberg zeta function [32]. If Λ is a finitely generated subgroups of SL2(R) the Selberg zeta functionZX(s) associated withX=Λ\His known to be an entire function, whose non-trivial zeros are given by the resonances and the finite point spectrum [11], [27]. Consequently, Theorem 1.1 is equivalent to the assertion that whenδ(Λ)>¹₂ there isε(Λ)>0 such thatZ_X(q)(s) is analytic and non-vanishing on the set{s:Res>δ−ε}, except ats=δwhich is a simple zero, while Theorem 1.4 is equivalent to the assertion that whenδ(Λ)6¹₂ there isε(Λ)>0 such thatZX(q)(s) is analytic and non-vanishing on the set{s:Res>δ−εmin{1,1/log(1+|Ims|)}}, except ats=δwhich is a simple zero.

We turn to applications to the affine linear sieve [3]. Consider the standard action on the 2×2 integer matrices by multiplication on the left, and take the orbitOofI (the identity matrix) under Λ. Set

|x|= ²

X

j,k=1

x²_jk 1/2

,

where

x=

x11 x12

x21 x22

.

SetNΛ(T)=|{x∈Λ:|x|6T}|and letδ(Λ) be the Hausdorff dimension of the limit set of an orbit Λz⊂H∪{∞}∪R, where H is the hyperbolic plane, z∈H and Λ acts by linear fractional transformations. By the results of Lax–Phillips [18] and Lalley [16], we have that N_Λ(T)∼cΛT^2δ(Λ), as T!∞. Let f∈Q[x_jk] be integral on O and assume that it is weakly primitive for O, that is gcd{f(x):x∈O}=1. Iff is not weakly primitive then f /N is, whereN=gcdf(O), and we can represent any weakly primitivef as g/N with g∈Z[x_jk] andN=gcd(O).

The coordinate ringQ[x_jk]/(det(x_jk)−1) is a unique factorization domain [29] and we can factorf into t=t(f) irreduciblesf₁f₂... f_tin this ring. Set

π_Λ,f(T) =|{x∈Λ :|x|6T andf_j(x) is prime forj= 1, ..., t}|.

Forf∈Z[xjk] weakly primitive witht(f) irreducible factors, our conjectured asymp- totics is of the form

πΛ,f(T)∼c(Λ, f)NΛ(T)

(logT)^t(f) , asT!∞,

where c(Λ, f) can be expressed as a product of local densities, see [9] and [31] for an example of explicit computation and numerical experiments. In §13 we establish the following sharp upper bound forπ_Λ,f(T).

Theorem 1.6. Let Λ be a subgroup of SL(2,Z) which is Zariski dense in SL2 and let f∈Z[xjk]be weakly primitive with t(f)irreducible factors. Then

πΛ,f(T) NΛ(T) (logT)^t(f).

We also obtain the following lower bound for the number of pointsx∈Λ for which f has at most a fixed number of prime factors.

Theorem 1.7. Let Λ be a subgroup of SL(2,Z) which is Zariski dense in SL₂ and let f∈Z[x_jk] be weakly primitive with t(f) irreducible factors. Then there is an r<∞, which can be given explicitly in terms of ε(Λ) in Theorems 1.1and 1.4,such that

|{x∈Λ :|x|6T andf(x)has at mostr prime factors}| NΛ(T) (logT)^t(f).

2. Generalization of Selberg’s ₁₆³ theorem when δ >¹₂

Being a subgroup of finite index in Λ(1), Λ(q) has the same bottom of the spectrum, λ0(Λ(q)\H)=λ0(Λ(1)\H). As in [10,§2], we have that for q large enough Λ(1)/Λ(q)∼= SL2(Z/qZ). LetS={A1, ..., Ak}, and letSq be the natural projection ofS moduloq.

Theorem 1.7 in [3] implies that if Λ=hSiis non-elementary, then G_q=G(SL₂(Z/qZ), S_q)

is a family of expanders. Consider the space H(q) of vector-valued functions F on F=F(1), the fundamental domain of Λ=Λ(1), satisfying

F(γz) =Rq(γ)F(z)

forγ∈Λ(1)/Λ(q)∼=SL2(Z/qZ), whereR_q(γ) is the regular representation of SL₂(Z/qZ).

We denote by h ·,· i the inner product on this space and by k · k the associated norm.

Letϕ₀ denote the eigenfunction corresponding to λ₀. LetH₀(q) denote the subspace of functions inH(q) orthogonal to ϕ₀⊗Id. The assertion of Theorem 1.1 is equivalent to existence ofc>0 such that

R

Fk∇Fk²dµ R

FkFk²dµ >λ0+c for allF∈H₀(q).

Applying [3, Theorem 1.7] for each z∈F(1) implies that there is ε>0, depending only onS, such that for allF∈H0(q), we have

kF(γz)−F(z)k>εkF(z)k for some γ∈S. (2.1) Letf=kFk, and decompose it as

f=aϕ0(z)+b(z), where

Z

F

ϕ0(z)b(z)dµ(z) = 0 (2.2)

and

Z

F

|f|²dµ=a²+ Z

F

|b|²dµ= 1.

Write

F(z) = (F1(z), ..., FN(z)),

whereN=|SL2(Z/qZ)|. Since

∇ ^N

X

j=1

|F_j(z)|^{2 1/2}

=









 ^N

X

j=1

Fj(z)∇Fj(z) ^N

X

j=1

|Fj(z)|² −1/2

, if

N

X

j=1

|Fj(z)|²6= 0,

0, otherwise,

we have

k∇Fk²(z)>

∇kFk

2(z) =|∇f|²(z).

Consequently we obtain R

Fk∇Fk²dµ R

FkFk²dµ >

R

F

∇kFk

2dµ R

FkFk²dµ = R

F|∇f|²dµ R

F|f|²dµ = R

Fh∆f, fidµ R

F|f|²dµ

= Z

F

haλ0ϕ0+∆b, aϕ0+bidµ=a²λ0+h∆b, bi

(2.2)

>a²λ0+λ1

Z

F

|b|²dµ

>λ₀+(λ₁−λ0) Z

F

|b|²dµ.

By a theorem of Lax and Phillips [18], there are only finitely many discrete eigen- values of Λ in

0,¹₄

, consequently,

λ₁−λ0>c₁>0.

Therefore, as soon asR

F|b|²dµ>ε₁>0, we have that R

F|∇F|²dµ R

F|F|²dµ >λ₀+c₁ε₁. Now consider the case R

F|b|²dµ=0. We may assume a=1 and writeF(z)=u(z)ϕ₀(z), withu(z)=(u₁(z), ..., u_N(z)), whereN=|SL2(Z/qZ)|. Now

ku(z)k=

N

X

j=1

|uj|²(z) = 1 implies that

N

X

j=1

u_j∂u_j

∂x =

N

X

j=1

u_j∂u_j

∂x = 0, and since

∂(ϕ₀u_j)

∂x =u_j∂ϕ₀

∂x +ϕ₀∂u_j

∂x and ∂(ϕ₀u_j)

∂y =u_j∂ϕ₀

∂y +ϕ₀∂u_j

∂y ,

we have that k∇ϕ0uk²=

∂ϕ₀

∂x 2 N

X

j=1

u²_j+ϕ²₀

N

X

j=1

∂u_j

∂x 2

+ ∂ϕ₀

∂y 2 N

X

j=1

u²_j+ϕ²₀

N

X

j=1

∂u_j

∂y 2

+2ϕ0

∂ϕ0

∂x

N

X

j=1

uj

∂uj

∂x +2ϕ0

∂ϕ0

∂y

N

X

j=1

uj

∂uj

∂y

=|∇ϕ0|²+ϕ²₀k∇uk². Consequently,

R

Fk∇Fk²dµ R

FkFk²dµ = R

F(|∇ϕ₀|²+ϕ²₀k∇uk²)dµ R

F|ϕ0|²dµ

= R

F|∇ϕ₀|²dµ R

F|ϕ0|²dµ + R

Fϕ²₀k∇uk²dµ R

F|ϕ0|²dµ >λ₀+ R

Fϕ²₀k∇uk²dµ R

F|ϕ0|²dµ . Our aim now is to show that

R

Fϕ²₀k∇uk²dµ R

F|ϕ0|²dµ >c2>0. (2.3) To that end, we assume that

R

Fϕ²₀k∇uk²dµ R

F|ϕ0|²dµ < (2.4)

and will obtain a contradiction for sufficiently small(j below are of the forma_jfor suitable constantsa_j). Consider the fundamental domain F=Λ\H. Its boundary, ∂F, consists of finitely many geodesic arcs{l_k}_k splitting into pairs l_j, l⁰_j in such a way that there isγ_j∈S so thatl_j=γ_jl_j⁰, whereγ_j are distinct and generate Λ. Further, we have a decomposition of the form

F=K∪ [

k∈Cu

cusp_k∪[

j∈Fl

flarej,

where the following hold:

(1) K is relatively compact inH;

(2) Cu is a set of cusps of F. Each cusp_k is isometric to a standard cuspidal fundamental domainP(Yk) of the form

P(Y) ={z=x+iy: 0< x <1 andy > Y}, based on a horocycle

hY ={x+iy:y=Y};

(3) Fl is a set of flares of F. Each flarej(α) is isometric to a standard hyperbolic fundamental domain F(α) of the form

F(α) ={z: 1<|z|< e^β and 0<argz < α}, whereα<¹₂π.

Sinceϕ₀∈L²(F), we have that Z

K

|ϕ0|²dµ>c3

Z

F

|ϕ0|²dµ for somec3>0, and therefore (2.4) implies that

R

Kϕ²₀k∇uk²dµ R

K|ϕ0|²dµ 61. (2.5)

We recall the definition of Fermi coordinates. Letηbe the geodesic in the hyperbolic plane parameterized with the unit speed in the form

t7−!η(t)∈H, t∈R.

ThenηseparatesHinto two half-planes: a left-hand side and a right-hand side ofη. For eachp∈Hwe have the directed distance%fromptoη. There exists a uniquetsuch that the perpendicular frompto η meetsη atη(t). Now (% , t) is a pair of Fermi coordinates ofpwith respect toη. In these coordinates the metric tensor is

ds²=d%²+cosh²% dt². (2.6)

Introduce Fermi coordinates based on the bounding geodesics lj, and use them to foliateK. By compactness, using (2.5), we can findz∈Kandδ >0 such that

Z

B(z,δ)

|ϕ0|²dµ > c4>0, and, for allj=1, ..., k,

R

T_j(δ)ϕ²₀k∇uk²dµ R

Tj(δ)|ϕ0|²dµ <2, (2.7) whereTj(δ) is a tube lying inK and containingB(z, δ) along the perpendicular tolj.

Each tubeTj is of the form [−δ, δ]×[%1,j, %2,j] in the appropriate Fermi coordinates.

Rewriting (2.7) in Fermi coordinates (2.6), we have Z

Tj(δ)

ϕ²₀k∇uk²dµ>

Z δ

−δ

Z %_2,j

%1,j

ϕ²₀

∂uj

∂%

2

cosh% d% dt.

LetL denote the maximal length of the tubesTj. Using the fact that if

|u(%1)−u(%2)|>C and %₁−%26L then

Z %₂

%1

|u⁰(%)|²d% >C² L , since

C²6 Z %2

%₁

u⁰(%)d%

2

6 Z %2

%₁

1d%

Z %2

%₁

|u⁰(%)|²d%

,

we obtain that (2.7) implies that for allj=1, ..., kwe have Z

B(z,δ)

ϕ²₀ku(γ_jz)−u(z)kdµ(z)<3

Z

B(z,δ)

ϕ²₀dµ(z). (2.8) On the other hand, sinceF(z)=u(z)ϕ0(z) andϕ0(γz)=ϕ0(z) for allγ∈SL2(Z/qZ), (2.1) implies that there isε(S)>0 independent ofq, such that

ku(γz)−u(z)k> ε(S) for someγ∈S. (2.9) Applying the mean-value theorem, we see that (2.8) implies a contradiction with (2.9) onceis small enough depending onε(S). Consequently we have proved the validity of (2.3) and the proof of Theorem 1.1 is complete.

The adaption of the preceding argument to proving the implication (i)⇒(ii) of The- orem 1.2 is straightforward, as is the generalization of this result to higher-dimensional hyperbolic spaces: The theorem of Lax and Phillips, of which we made crucial use in the first part of the argument, holds for geometrically finite subgroups of SO(n,1) with Hausdorff dimension of the limit set greater than ¹₂n. The second part of the argu- ment proceeds as above by restricting to a compact part of the fundamental domain and foliating it using Fermi coordinates. In particular, by combining theH³ analogue of Theorem 1.2 with [36] and Theorem 6.3 in [3], we obtain the following theorem which has applications to integral Apollonian packings [9], [15], [31].

Theorem 2.1. Let Λ be a geometrically finite subgroup of SL₂(Z[i])

with δ(Λ)>1and such that the traces of elements of Λgenerate the field Q(i). There is ε=ε(Λ)>0 such that

λ₁(Λ(A))>λ₀(Λ(A))+ε as Avaries over square-free ideals in Z[i].

3. Counting lattice points for δ >¹₂

Recall that the Poincar´e upper half-plane model is the following subset of the complex planeC:

H={z=x+iy∈C:y >0}, with the hyperbolic metric

ds²=dx²+dy² y² . The distance function onHis explicitly given by

%(z, w) = log|z−w|+|z−w|

|z−w|−|z−w|. (3.1)

We will use the expression

cosh%(z, w) = 1+2u(z, w), where

u(z, w) = |z−w|²

4 ImzImw. (3.2)

The ringM₂(R) of 2×2 real matrices is a vector space with inner product given by hg, hi= trace(gh^t).

One easily checks thatkgk=hg, gi^1/2 is a norm inM2(R) and that kgk²=a²+b²+c²+d² forg=

a b c d

.

By takingz=w=iin (3.2), we obtain that

kgk²=a²+b²+c²+d²= 4u(gi, i)+2. (3.3) Now the result of Lax and Phillips [18] is as follows. Let

NΛ(T;z, w) = #{γ∈Λ :%(z, γw)6T}.

Suppose that δ >¹₂ and write λj=δj(1−δj), δ0=δ. Denoting the eigenfunctions corre- sponding toλj byϕj, we have

N(T;z, w)−X

j

c_jϕ_j(z)ϕ_j(w)e^δjT

=O(T^5/6e^{T /2}).

Turning to congruence subgroups Λ(q) we have, using the methods of [18], that

N_Λ(q)(T;z, w)−X

j

cjϕj,q(z)ϕj,q(w)e^δj,qT

=O(q³T^5/6e^{T /2}), (3.4)

where the implied constant is independent ofq.

The base eigenfunction for Λ(q), normalized to haveL² norm 1, is given by ϕ0,q= 1

|SL2(Z/qZ)|ϕ0,1.

Combining Theorem 1.1 with (3.4), (3.1) and (3.3), we obtain ε1>0 depending on Λ such that for anyg∈SL2(q) we have

|{γ∈Λ :kγk6T andγ≡g modq}|= c_ΛT^2δ

|SL2(q)|+O(q³T^2δ−ε1), establishing Theorem 1.3.

4. Shifts and thermodynamic formalism

Whenδ6¹2 theL² spectral theory of Lax and Phillips [18] is not available and we use a symbolic dynamics approach, in particular the work of Lalley [16]. In this section we review the key necessary notions and results pertaining to shifts of finite type.

A shift of finite type is defined as follows. Let A be an irreducible, aperiodic l×l matrix of zeros and ones, called the transition matrix. Define Σ to be the space of all sequences taking values in the alphabet{1,2, ..., l}with transitions allowed byA, that is

Σ =

x∈

∞

Y

n=0

{1, ..., l}:A(xn, xn+1) = 1 for alln

.

The space Σ is compact and metrizable in the product topology. Define theforward shift σ: Σ!Σ by (σx)_n=x_n+1 forn>0.

LetC(Σ) be the space of continuous, complex-valued functions on Σ. Forf∈C(Σ) and 0<%<1 define

var_nf= sup{|f(x)−f(y)|:x_j=y_j for 06j6n},

|f|%= sup

n>0

varn(f)

%ⁿ ,

F%={f∈C(Σ) :|f|%<∞}.

The elements of F% are called H¨older continuous functions. The space F%, when endowed with the norm k · k%=| · |%+k · k∞is a Banach space.

Forf, g∈C(Σ) define the transfer operatorLfg∈C(Σ) by Lfg(x) = X

y:σy=x

e^f(y)g(y).

For each %∈(0,1) and f∈F%, Lf:F%!F% is a continuous linear operator. If f is real- valued thenLf is positive.

Denoting

Snf=f+f σ+...+f σⁿ⁻¹, we have

Lⁿ_fg(x) = X

y:σⁿy=x

e^Snf(y)g(y).

The following result is due to Ruelle, a proof can be found in [24] or [28].

Theorem 4.1. Let f∈F% be a real-valued function.

(1) There is a simple eigenvalue λf>0 of Lf:F%!F% with strictly positive eigen- function h_f.

(2) The rest of the spectrum of Lf is contained in {z∈C:|z|6λ_f−ε}for some ε>0.

(3) There is a Borel probability measure ν_f on Σ such that L^∗_fν_f=λ_fν_f. (4) If h_f is normalized so that R

Σh_fdν_f=1then for every g∈C(Σ),

nlim!∞

λ⁻ⁿ_f Lⁿ_fg−

Z

Σ

g dνf

hf

_∞

= 0.

(5) There exist constants C1 and ε1 such that for all g∈F% and for all n,

λ⁻ⁿ_f Lⁿ_fg−

Z

Σ

g dν_f

h_f

_%6C₁(1−ε1)ⁿkgk%. Thepressure functional is defined by

P(f) = sup

ν

Z

Σ

f dν+H_ν(σ),

where the supremum is taken over the set ofσ-invariant probability measures andHν(σ) is the measure-theoretic entropy ofσwith respect toν. We have (see [24] or [28])

P(f) = logλf.

A measureµis called theequilibrium state or theGibbs measure withpotential f if Z

Σ

f dµ+Hµ(σ) =P(f).

Forf∈F% the Gibbs measureµ_f is the uniqueσ-invariant probability measure on Σ for which there exist constants 0<C₁6C₂<∞such that

C16µf{y∈Σ :yj=xj for 06j < n}

λ⁻ⁿ_f e^Snf(x) 6C2.

As will become clear in the next section, the analyticity properties of the map z7!Lzf, z∈C, will play a crucial role in the proof. For a fixed, real-valued function f∈F%, such thatSmf is strictly positive for somem, the quantities Lzf, λzf, hzf and νzf will be abbreviated byLz, λz,hz andνz, respectively.

5. Resolvent of transfer operator and lattice count problem

Let Λ be a Fuchsian group with no parabolic elements (this condition is automatically sat- isfied in the caseδ(Λ)6¹2, see [18]), generated bykelementsg1, ..., gk⊂SL2(Z). We iden- tify Λ with Σ_∗, defined as the set of finite sequences in the alphabet{g1, g₁⁻¹, ..., gk, g⁻¹_k } (l=2k) with admissible transitions. According to Series [34], this may be done so as to obtain a shift of finite type.

Letw∈H, and suppose it is not a fixed point for anyγ∈Λ. Letd_H denote hyperbolic distance. Forx=x₁, ..., x_m∈Σ_∗(=Λ) define

τ(x) =dH(i, x1... xmw)−dH(i, x2... xmw). (5.1) The left shiftσ on Σ corresponds to the Nielsen map (see [34]) F:L!L, where L denotes the limit set of Λ.

Recalling that

Snτ=τ+τ σ+...+τ σⁿ⁻¹, we have

Snτ(x) =dH(i, x1... xnxn+1... w)−dH(i, xn+1... w).

Fora∈R,x∈Σ∗ andφ: Σ_∗!R, let N(a, x) =

∞

X

n=0

X

y:σⁿy=x

φ(y)1_{Snτ(y)6a}. (5.2)

Clearly

N(a, x) = X

y∈Λ

d_H(i,yxw)−dH(i,xw)6a

φ(yx), (5.3)

where in the summationy is restricted so as to makeyx admissible.

In particular, forφ=1,

N(a, x) =|{γ∈Λ :d_H(i, γxw)−d_H(i, xw)6a}|.

Returning to (5.2), one has the renewal equation (cf. [16, equation (2.2)]) N(a, x) = X

x⁰:σ(x⁰)=x

N(a−τ(x⁰), x⁰)+φ(x)1_{a>0}. (5.4)

The link with the transfer operator comes by taking the Laplace transform of (5.4).

Defining, for Rez <−C,

F(z, x) = Z ∞

−∞

e^azN(a, x)da, (5.5)

equation (5.4) gives the relation F(z, x) = X

x⁰:σ(x⁰)=x

e^zτ(x0)F(z, x⁰)+φ(x) z .

Thus we have

(I−Lz)F(z, x) =φ(x)

z . (5.6)

This leads us to the study of the resolvent (I−Lz)⁻¹. Before stating the results of Lalley [16] and Naud [22] for Lz|_F%(Σ) (Theorem 5.1 below) we recall the following reinterpretation of the Hausdorff dimension of the limit set in terms of the pressure functional (see [24] or [28]). Because τ is eventually positive, the variational principle implies (see [24] or [28]) that the pressure functional P(−sτ) is strictly decreasing and has a unique positive zero. DefineδbyP(−δτ)=0, that is,

λ_−δ= 1.

Theorem 5.1. (1) There is ε>0 such that for Rez <−δ+ε,z /∈U (U being a suit- able neighborhood of −δ),we have

kLⁿ_z|_F%(Σ)k%.|Imz|²e^−εn. (5.7) (2) For z∈U decompose on F%(Σ),

Lz=λz(νz⊗hz)+L⁰⁰_z, (5.8)

where z7!λz,z7!hz and z7!νz are holomorphic extensions to U satisfying Lzhz=λzhz, L^∗_zνz=λzνz and

Z

Σ

hzdνz= 1.

Then

k(L⁰⁰_z)ⁿk%< e^−εn for z∈U . (5.9) Part (1) follows from the discussion in Lalley [16, p. 25] (in the case whenτ is not a lattice) and Theorem 2.3 in Naud [22] to provide (5.7) when|Imz|is large. Naud’s work build crucially on the approach of Dolgopyat [7]. Note that Lalley does not give explicit estimates onkLⁿ_zkfor Rez=−δand Imz!∞and certainly no bound of the strength of Theorem 2.3 in Naud [22].

Part (2) is Proposition 7.2 in [16].

6. Lattice count in congruence subgroups forδ6¹₂

In this section we modify the setup discussed in the preceding two sections to the setting of congruence subgroups of Λ—the modification is analogous to the one preformed in the caseδ >¹₂.

Fix the modulus qsuch that πq(Λ)=SL2(q). Instead of considering functions on Σ (as in Lalley [16]), we consider functions on Σ×SL2(q).

Forf∈C(Σ×SL2(q)) define kfk∞= max

x

X

g∈SL2(q)

|f(x;g)|² 1/2

,

Varnf= sup

X

g∈SL2(q)

|f(x;g)−f(y;g)|^{2 1/2}

:xj=yj for 06j6n

,

|f|%= sup

n

Varn(f)

%ⁿ .

LetF%=F%(Σ×SL2(q)) denote the space of %-Lipschitz continuous functions with the norm

k · k%=k · k∞+| · |%.

Let τ: Σ∗!R be given by (5.1) and consider the “congruence transfer operator”

Mz=Mzτ onF%(Σ×SL2(q)):

M_zf(x;g) =

l

X

j=1

e^zτ(j,x)f((j, x);g_jg),

wherez∈Cand the summation is restricted so as to make (j, x) admissible.

Thus ourMz differs from the one considered in [16] in that it acts on functions on Σ×SL2(q) rather than on functions on Σ: the reason behind this difference is the same as in the proof of the spectral gap whenδ(Λ)>¹₂.

We have

M²_zf(x;g) =

l

X

i1=1

e^zτ(i1,x)M_zf((i₁, x);g_i1g)

=

l

X

i1,i2=1

e^{z(τ(i1,x)+τ(i2,i1,x))}f((i₂, i₁, x);g_i2g_i1g), and in general for thenth iterate we have

Mⁿ_zf(x;g) =

l

X

i₁,...,i_n=1

e^{z(τ(in,...,i1,x)+τ(in−1,...,i1},x)+...+τ(i1,x))f((in, ..., i1, x);gi_n... gi₁g), (6.1) where again the summation is restricted to admissible words.

From Ruelle’s theorem (Theorem 4.1) it follows that

l

X

i₁,...,i_n=1

e^{Rez(τ(in,...,i1,x)+τ(in−1,...,i1},x)+...+τ(i₁,x))

∼λⁿ_Rez (6.2) for large n.

Letϕbe a function on SL2(q). Returning to (5.2) and (5.3), we let

N(a, x) = X

y∈Λ

dH(0,yxw)−dH(0,xw)6a

ϕ(πq(y))

andF(z, x) be its Laplace transform, defined by (5.5).

Then

F(z, x) =f((z, x);πq(x)), (6.3) wheref((z, x);g) satisfies

(1−Mz)f=1⊗ϕ

z , (6.4)

andMzis the congruence transfer operator introduced above (note that obviously 1⊗ϕ is in F%(Σ×SL2(q))).

Our aim is to evaluate

N(a) = X

y∈Λ d_H(0,yw)−dH(0,w)6a

ϕ(πq(y)), (6.5)

which gives the sum ofϕon the mod-q reduction of the hyperbolic ball {y∈Λ :dH(0, yw)−dH(0, w)6a}.

Our goal now is to obtain the appropriate extension of Theorem 5.1 to the setting of congruence subgroups. As is to be expected, it is at this point that the expansion property will play a crucial role.

7. Expansion and L² flattening

Letµbe a symmetric measure onG=SL₂(p) (for the sake of exposition we first consider the simpler case of primep) and consider the convolution mapT:L²(G)!L²(G), given by ϕ7!µ∗ϕ. Decomposing the regular representation ofGinto irreducible representations, it follows from the result of Frobenius [8] that each eigenvalue λ of the convolution restricted to L²₀(G) occurs with multiplicity at least ¹₂(p−1). Trace calculation yields therefore

|G| kµk²₂=X

x∈G

hT²δx, δxi>¹2(p−1)λ². Hence

|λ|6 r 2

p−1kµk₂|G|^1/2. (7.1)

Recall also theL²-flattening lemma proven in [2]. Letµ∈P(G), a probability mea- sure onG, satisfy

kµk∞< p^−τ (7.2)

for someτ >0 and also

µ(aG₁)< p^−τ (7.3)

for all cosets of proper subgroupsG1 ofG. Given>0 there isl=l(τ, x)∈Z+such that kµ^(l)k2<|G|^−1/2+. (7.4) Setµ⁰(x)=µ(x⁻¹). Sinceµ∗µ⁰ also satisfies (7.2) and (7.3), we have by (7.4) that

k(µ⁰∗µ)^(l)k2<|G|^−1/2+.

Consider the convolution operatorT ϕ=µ⁰∗µ∗ϕand letλbe an eigenvalue ofTonL²₀(G).

Henceλ^lis an eigenvalue ofT^lonL²₀(G) and applying (7.1) withµreplaced by (µ⁰∗µ)^(l) implies that

|λ|^l6 r 2

p−1k(µ⁰∗µ)^(l)k₂|G|^1/2<

r 2

p−1|G|< p^−1/4

if we take<¹₄. Consequently,

|λ|< p^−1/4l.

This means that ifϕ∈L²₀(G), then

kµ⁰∗µ∗ϕk26p^−1/4lkϕk2, and hence

kµ∗ϕk26p^−1/8lkϕk2.

We proved that if µ∈P(G) satisfies (7.2) and (7.3) for someτ >0, then kµ∗ϕk26p^−τ0kϕk2, ϕ∈L²₀(G),

for someτ⁰>0. Therefore, ifµ∈M+(G), a positive real-valued measure onG, satisfies

kµk_∞< p^−τkµk1 (7.5)

and

µ(aG1)< p^−τkµk1 (7.6)

for cosets of proper subgroupsG₁, then

kµ∗ϕk26p^−τ0kµk kϕk2, ϕ∈L²₀(G). (7.7) We make next the following observation. Assume thatB >0 andµ∈M+(G) satisfies kµk16B,

kµk∞< p^−τB, (7.8)

and

µ(aG1)< p^−τB (7.9)

for cosets of proper subgroupsG₁. Then

kµ∗ϕk2< p^−τ0Bkϕk2 forϕ∈L²₀(G). (7.10) Indeed, if kµk1>p^{−τ /2}B, then (7.5) and (7.6) hold with τ replaced by ¹₂τ and (7.10) follows from (7.7). Ifkµk16p^{−τ /2}B then obviously

kµ∗ϕk26kµk1kϕk2< p^{τ /2}Bkϕk2. More generally, letµ∈M_C(G) satisfy kµk₁6B,

kµk_∞< p^−τB

and

|µ|(aG1)< p^−τB for cosets of proper subgroupsG₁. Decompose

µ=|µRe|−(|µRe|−µRe)+i(|µIm|−(|µIm|−µIm)).

Each of the measuresν=|µRe|,|µRe|−µRe,|µIm|and|µIm|−µIm obviously satisfies (7.8) and (7.9). Hence

kν∗ϕk₂< p^−τ0Bkϕk₂. Thus we obtain the following result.

Lemma7.1. Given >0,there is⁰>0such that if µ∈M_C(G)andB>kµk1satisfy kµk_∞< p⁻B and |µ|(aG1)< p⁻B

for cosets of proper subgroups G₁ of G,then

kµ∗ϕk26Cp⁻⁰Bkϕk2 for ϕ∈L²₀(G).

Here pis assumed to be sufficiently large.

We have a similar result for G=SL₂(q) with q square-free (see [3] and [36]). We make the following decomposition of the spaceL²(SL₂(q)). For q₁|q, defineE_q1 as the subspace of functions defined modq₁and orthogonal to all functions defined modq₂ for someq₂|q1,q₂6=q1. Hence

L²(SL₂(q)) =R⊕M

q₁|q

E_q1, (7.11)

which is, in fact, the generalized Fourier–Walsh decomposition corresponding to the product representation

SL2(q)∼=Y

p|q

SL2(p).

LetP1 (resp.Pq₁) be the projection operator on the constant functions (resp.Eq₁).

Lemma 7.2. Let q be square-free and G=SL₂(q). For µ∈M_C(G) and q₁|q define

|||π_q1(µ)|||_∞ to be the maximum weight of |µ| over cosets of subgroups of SL₂(q₁) that have proper projection in each divisor of q₁. Given >0,there is ⁰>0 such that if µ satisfies kµk16B and

|||πq₁(µ)|||∞< q⁻B for q1|q,q1> q^1/10, (7.12) then

kµ∗ϕk26Cq⁻⁰Bkϕk2 for ϕ∈Eq. (7.13) Remark. The assumptionϕ∈Eq is important in (7.13). The restrictionq1>q^1/10in (7.12) has to do with the fact that an irreducible representation of SL2(q) which does not factor through SL2(q⁰) for someq⁰|qhas dimension at leastQ

p|q 1 2(p−1).

8. Bounds for congruence transfer operator

Our goal is to obtain a bound for powers of transfer operatorkM^m_zk% for the family of congruence subgroups. Herezdenotes a complex number with Rez <0 which is bounded in absolute value. Recall that Mz acts on functions on Σ×G, so in order to apply Lemma 7.2 we need to decouple the variables. Returning to (6.1), fixm6r<nsuch that m=n−r∼logqto be specified (we assume qlarge enough). Write

Mⁿ_zf(x;g)

=

l

X

i₁,...,i_n=1

e^{z(τ(in,...,i1,x)+τ(in−1,...,i1},x)+...+τ(i₁,x))f((i_n, ..., i_n−r+1,0);g_in... g_i1g) +O(λⁿ_Rez|f|%%^r),

where the error term refers to theL^∞_l2(G)(Σ)-norm.

Fix then the matrices corresponding to indicesin, ..., in−r+1 and consider the func- tionϕonGdefined by

ϕ(g) =f((i_n, ..., i_n−r+1,0);g_in... g_in−r+1g).

We assume thatf(x;·)∈E_q for eachx. Henceϕ∈E_q. Our aim is to apply Lemma 7.2 with

µ= X

i1,...,in−r

e^{z(τ(in,...,i1},x)+...+τ(i1,x))δg_in−r...g_i1.

Thus, by (6.2), we have

kµk₁.λ^m_Reze^{Rez(τ(in,...,in−r+1},0)+...+τ(in−r+1,0))e^{|Rez| |τ|%/(1−%)}≡B,

where we used the inequality

|τ(in, ..., i_n−r+1, i_n−r, ..., i1, x)−τ(in, ..., i_n−r+1, x)|6|τ|%%^r.

We now boundkµk∞, which amounts to estimating

S:= X

g_im...gi1=g

e^{Rez(τ(in,...,i1},x)+...+τ(i₁,x)),

wheregis fixed. (We use here the fact that the relationg_im... g_i1=gmodqis equivalent to g_im... g_i1=g because of the restriction onm.) Also because of the index restriction on the transition matrix A, the condition g_im... g_i1=g specifies (i_m, ..., i₁)∈Σ so that estimatingS amounts to bounding

T:=e^{Rez(τ(in,...,i1},x)+...+τ(i1,x))

.Bλ^−m_Reze^{Rez(τ(im,...,i1},x)+...+τ(i1,x))

for fixed (in, ..., i1, x)∈Σ. Thus

B⁻¹T=λ^−m_RezM^m_Rezδ_{(im,...,i1,x)}(x)6λ^−m_RezkM^m_Rezφk_∞ withφbeing any non-negative function on Σ satisfyingφ(im, ..., i1, x)=1.

By Ruelle’s theorem (Theorem 4.1),

λ^−m_RezM^mφ−

Z

Σ

φ dν

h

_%.(1−ε1)^mkφk%, implying that

kλ^−m_RezM^mφk∞6c Z

Σ

φ dν+(1−ε1)^mkφk%

.

We may now chooseφsuitably, so as to obtain an estimate λ^−m_RezkM^mφk_∞.e^−cm.

Hence,

S, T.q⁻B. (8.1)

More generally, we also need to evaluate|||π_q1(µ)|||_∞ forq₁|q. It turns out that the issue reduces to the previous one, using the following observation (cf. [2]). Let H <G andπ_p(H)<SL₂(p) proper for eachp|q₁. Then we can assume the second commutator of π_p(H) to be trivial ifp|q₁, and hence the second commutator ofHto be trivial (modq₁).

Takem₁<m,m₁∼logq₁, so as to ensure that words of length 2m₁have norm less thanq₁. Using properties of the free group (see [2]), it follows from the preceding that the number of (im₁, ..., i1)∈Σ such that gi_m1... gi₁∈aH is bounded by O(m^C₁) for some constant C.

Hence we may invoke the estimate onT withmreplaced bym1, to obtain also that

|||πq₁(µ)|||∞< q₁⁻B.

Applying Lemma 7.2, it follows that kµ∗ϕk₂6q⁻⁰k˜µk₁kϕk₂

6q⁻⁰λ^m_Reze^{Rez(τ(in,...,in−r+1},0)+...+τ(in−r+1,0))e^{|Rez| |τ|%/(1−%)}kfk_∞,

or

l

X

i₁,...,i_n−r=1

e^{z(τ(in,...,i1},x)+...+τ(i1,x))f((in, ..., in−r+1,0);gi_n... gi₁g) _l2(G) 6q⁻⁰λ^m_Reze^{Rez(τ(in,...,in−r+1},0)+...+τ(in−r+1,0))e^{|Rez| |τ|%/(1−%)}kfk_∞.

(8.2)

Summing (8.2) overin, ..., in−r+1 implies by (6.2) again that

l

X

i1,...,in=1

e^{z(τ(in,...,i1,x)+τ(in−1,...,i1},x)+...+τ(i1,x))f((in, ..., i_n−r+1,0);gi_n... gi₁g) _l2(G) .q⁻⁰λ^m+rkfk_∞.

Therefore it follows that, ifnlogq, then

kMⁿ_zfkL^∞_{l2 (G)}(Σ)6λⁿ_Rez(q⁻⁰kfk∞+%^r|f|%)6λⁿ_Rezq⁻⁰(kfk∞+%^n/2|f|%) (8.3) for

f∈ F_%⁰=F%∩CEq(Σ),

where Eq was defined just before (7.11). Note that in (8.3) there is no restriction on Imz.

We also need to estimate|Mⁿ_zf|%.

Letx, y∈Σ be such that xi=yi for 06i<l. Estimate

|Mⁿ_zf(x;g)−Mⁿ_zf(y;g)|

6 X

i1,...,in

e^{Rez(τ(in,...,i1},x)+...+τ(i₁,x))

×|f((in, ..., i1, x);gi_n... gi₁g)−f((in, ..., i1, y);gi_n... gi₁g)|

+

X

i₁,...,i_n

(e^{z(τ(in,...,i1},x)+...+τ(i₁,x))

−e^{z(τ(in,...,i1},y)+...+τ(i₁,y)))

×f((i_n, ..., i₁, y);g_in... g_i1g)

=:U+V.

Clearly for the first term we have

U.λⁿ_Rez|f|%%^n+l. (8.4)

To estimateV we repeat the argument leading to (8.3). Thus we boundV as follows

X

i₁,...,i_n

(e^{z(τ(in,...,i1},x)+...+τ(i1,x))−e^{z(τ(in,...,i1},y)+...+τ(i1,y)))

×f((in, ..., i_n−r+1,0);g_in... g_i1g) +%^r|f|%

X

i1,...,in

|e^{z(τ(in,...,i1},x)+...+τ(i1,x))−e^{z(τ(in,...,i1},y)+...+τ(i1,y))|=:H+K.