HAL Id: hal-00680801
https://hal.archives-ouvertes.fr/hal-00680801
Preprint submitted on 20 Mar 2012
HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Density and localization of resonances for convex co-compact hyperbolic surfaces
Frédéric Naud
To cite this version:
Frédéric Naud. Density and localization of resonances for convex co-compact hyperbolic surfaces.
2012. �hal-00680801�
FOR CONVEX CO-COMPACT HYPERBOLIC SURFACES
FR´ED´ERIC NAUD
Abstract. Let X = Γ\H2 be a convex co-compact hyperbolic surface. We show that the density of resonances of the Laplacian
∆X in strips {σ ≤ Re(s) ≤ δ} with |Im(s)| ≤ T is less than O(T1+δ−ε(σ)) with ε(σ) > 0 as long as σ > δ2. This improves the previous fractal Weyl upper bound of Zworski [28] and is in agreement with the conjecture of [13] on the essential spectral gap.
1. Introduction and results
In this work, we will focus on the distribution of resonances of the Laplacian for a class of hyperbolic Riemann surface of infinite volume.
The spectral theory of these objects can be viewed as a relevant pic- ture for more realistic physical models such as obstacle or potential scattering, but is also interesting in itself because of its connection with counting problems and number theory (see for example the recent work of Bourgain-Gamburd-Sarnak [4], or Bourgain-Kontorovich [5]).
Resonances replace the missing eigenvalues in non-compact situations and their precise distribution and localization in the complex plane is still a widely open subject. Let us be more specific. Let H2 denote the usual hyperbolic plane with its standard metric with curvature −1, and letX = Γ\H2 be a convex co-compact hyperbolic surface (see next
§ for more details). Here Γ is a discrete group of isometries ofH2, con- vex co-compact (that is to say finitely generated, without non trivial parabolic elements or elliptic elements). In this paper we will assume that Γ is non-elementary which is equivalent to say that X is not a hyperbolic cylinder. The limit set Λ(Γ) is commonly defined as
Λ(Γ) := Γ.z∩∂H2,
where Γ.z denotes the orbit ofz ∈H2 under the action of Γ. This limit set actually does not depend on the choice ofzand is a Cantor set in the above setting. We will denote by δ(Γ) the Hausdorff dimension of Λ(Γ).
Let ∆X denote the hyperbolic Laplacian on the surfaceX. The spectral theory onL2(X) has been described by Lax and Phillips [14] :[1/4,+∞) is the continuous spectrum, has no embedded eigenvalues. The rest of
Key words and phrases. Laplacian on hyperbolic surfaces, Resonances, Selberg’s Zeta function, Ruelle Transfer operators, Topological Pressure.
1
the spectrum is made of a (possibly empty) finite set of eigenvalues, starting atδ(1−δ). The fact that the bottom of the spectrum is related to the dimension δ was first pointed out by Patterson [19] for convex co-compact groups. This result was later extented for geometrically finite groups by Sullivan [26, 25].
By the preceding description of the spectrum, the resolvent RX(s) = (∆X −s(1−s))−1 :L2(X)→L2(X),
is therefore well defined and analytic on the half-plane {Re(s) > 12} except at a possible finite set of poles corresponding to the finite point spectrum. Resonances are then defined as poles of the meromorphic continuation of
RX(s) :C0∞(X)→C∞(X)
to the whole complex plane. The set of poles is denoted by RX. This continuation is usually performed via the analytic Fredholm theorem after the construction of an adequate parametrix. The first result of this kind in the more general setting of asymptotically hyperbolic manifolds is due to Mazzeo and Melrose [16]. A more precise parametrix for surfaces was constructed by Guillop´e and Zworski [12, 11]. Note that by the above construction and choice of spectral parameter s=σ+it, the resonance set (including possible eigenvalues) RX is included in the half-plane {Re(s) ≤ δ}. If δ > 12, then except for a finite set of eigenvalues, RX ⊂ {Re(s)< 12}.
One of the basic problems of the theory is to localize resonances with the largest real part, which are resonances who play a key role in var- ious asymptotic problems, including hyperbolic lattice point counting and wave asymptotics. Another central and related question is the ex- istence of a fractal Weyl lawwhen counting resonances in strips. More precisely, in the papers by Zworski and Guillop´e-Lin-Zworski, [10, 28]
they prove the following. For all σ ≤δ set
N(σ, T) := #{z ∈ RX : σ ≤Re(z)≤δ and 0≤Im(z)≤T}, then for all σ, one can find Cσ such that for all T ≥0, one has
N(σ, T)≤Cσ 1 +T1+δ .
The first upper bound of this type involving a ”fractal” dimension is due to Sj¨ostrand [23] for potential scattering, see also [24]. In [10], the authors give some numerical evidence that the above estimate may be optimal provided σ is small enough, but no rigorous results so far have confirmed this conjecture, and the best existing lower bound is a sublinear omega estimate, see [9]. On the other hand, it is conjectured
in [13] that for all ǫ >0, there are only finitely many resonancesin the
half-plane
Re(s)≥ δ 2 +ǫ
,
hence suggesting that one has to take σ ≤ δ/2 to observe the optimal rate of growth of N(σ, T) as T →+∞. In this paper, we show the fol- lowing which is in agreement with the previous conjecture and provides a more precise picture in term of the distribution of resonances.
Theorem 1.1. Let Γ be a non-elementary convex co-compact group as above. Then for all σ ≥ δ2, one can find τ(σ) ≥ 0 such that as T →+∞,
N(σ, T) =O T1+τ(σ) ,
where τ(δ2) = δ, τ(σ) < δ for all δ2 < σ. Moreover, one can find σ0 > δ2 such that the map σ7→τ(σ)is real-analytic, convex and strictly decreasing on [δ2, σ0]. In particular, its derivative at σ = δ2 satisfies τ′(δ2)<0.
The main achievement of that paper is that τ(σ)< δ for all σ > δ2 and says that the O(T1+δ) upper bound is non-optimal whenever σ > 2δ. From a physics point of view, this result is in agreement with the idea that the density of resonances has to ”peak” in a strip close to ”half of the classical escape rate” which is exactly δ2 in our setting. See the paper [15] for some numerical study that support this conjecture, especially figure 2. We also refer the reader to [18] for a comprehensive survey on questions related to fractal Weyl laws and spectral gaps for various open (chaotic) quantum systems. An ”explicit” expression for τ(σ) is provided in the last section (see formula (6)), involving the topological pressure of the Bowen-Series map. The above result is of course relevant for σclose enough to the critical value δ2 since we know from a previous work of the author [17] that there always exists a spectral gap i.e. one can find ǫ >0 such that
RX ∩ {Re(s)≥δ−ǫ}={δ}.
Unfortunately, this ǫ is hardly explicit. Of course if δ > 12, then this spectral gap is already given by Lax-Phillips theory, but the above theorem is still meaningful since δ2 < 12.
This result can also be compared with the spectral deviations ob- tained by N. Anantharaman [1] for the resonances of the damped wave equation on negativey curved manifolds, except that in our case no Weyl law is known rigorously. The techniques we rely on to prove Theorem 1.1 are therefore very different although they also involve some ergodic theory and thermodynamical formalism. We would like to point out that Theorem 1.1 has a natural interpretation in terms of Selberg zeta function. Let us denote by P the set of primitive closed
geodesics on X = Γ\H2 and given C ∈ P we denote by l(C) its length.
Selberg zeta functionZΓ(s) is usually defined through the infinite prod- uct (Re(s) is taken large enough)
ZΓ(s) := Y
(C,k)∈P×N
1−e−(s+k)l(C) .
It is known since the work of Patterson-Perry [20] that this function extends analytically to C and the non-trivial (non topological) zeros of ZΓ(s) are the resonances. Therefore Theorem 1.1 can be read as a statement on zeros of Selberg’s zeta function, and this is actually how we prove it. Let us make a few comments on the organization of the paper. In the next section we recall the transfer operator approach for Selberg’s zeta function. In section §3 we prove the necessary a priori bounds to control the growth of a (modified ) Selberg zeta function in strips, generalizing slightly the method of [10]. In §4 we use a classical lemma of Littlewood to relate the counting function for resonances to a ”mean square estimate”. The idea of §6, which is the core of that paper, is to exhibit some cancellations in the quadratic sums to beat the pointwise bound of [10]. To achieve this goal some lower bounds on the derivatives of ”off-diagonal phases” are required and proved in
§5. The technique we use in §6 bears some similarity with the work of Dolgopyat [8, 17] and can be viewed as an ”averaged” Dolgopyat estimate. We would like to mention that the ideas presented here should extend without major difficulties to higher dimensional Schottky groups, actually only §5 needs to be significantly modified.
2. Transfer operator and Selberg’s zeta function We use the notations of §1. Let H2 denote the Poincar´e upper half- plane H2 ={x+iy ∈C : y >0} endowed with its standard metric of constant −1 curvature
ds2 = dx2+dy2
y .
The group of (positive) isometries of H2 is naturally isomorphic to PSL2(R) through the action of 2×2 matrices viewed as M¨obius trans- forms
z 7→ az+b
cz+d, ad−bc= 1.
A Fuchsian Schottky group is a discrete subgroup of PSL2(R) built as follows. Let D1, . . . ,Dp,Dp+1, . . . ,D2p be 2p Euclidean open discs in C orthogonal to the line R ≃ ∂H2. We assume that for all i 6= j, Di∩ Dj =∅. Letγ1, . . . , γp ∈PSL2(R) be pisometries such that for all i= 1, . . . , p, we have
γi(Di) =Cb \ Dp+i,
where Cb :=C∪ {∞}stands for the Riemann sphere.
The discrete group Γ generated byγ1, . . . , γp and their inverses is called aclassical Schottky group. Ifp >1 then Γ is said to be non elementary.
It is always a free, geometrically finite, discrete group and if in addition we require that for alli6=j,Di∩Dj =∅, then Γ is a convex co-compact group i.e. the quotient Riemann surface
X = Γ\H2
is an infinite volume geometrically finite hyperbolic surface with no cusps. The converse is true, up to an isometry, all convex co-compact hyperbolic surfaces can be uniformized by a group as above, see [7].
Let Γ ⊂PSL2(R) be a Fuchsian Schottky group as defined earlier:
Γ =hγ1, . . . , γp;γ1−1, . . . , γp−1i,
where γi(Di) =Cb\ Dp+i. We also set fori= 1, . . . , p,γp+i :=γi−1. For allj = 1, . . . ,2p, let H2(Dj) denote the Bergman space of holomorphic functions defined by
H2(Dj) :=
(
f :Dj →C : f holomorphic and Z
Dj
|f|2dm <+∞ )
, here m stands for the usual Lebesgue measure. Each function space H2(Dj) is a Hilbert space when endowed with the obvious norm. We set
H2 :=
M2p j=1
H2(Dj).
The Ruelle transfer operator is a bounded linear operator Ls : H2 → H2 defined by (z ∈ Di, s∈C )
(Ls(f))i(z) :=X
j6=i
(γj′(z))sfj+p(γj(z)),
with the notation f = (f1, . . . , f2p), and j+pis understood mod 2p.
We have to say a few words about the complex powers here: we have γj′(Di)⊂C\R− for i6=j so γj′(z)s is understood as
γj′(z)s :=esL(γj′(z)),
where Lis a complex logarithm onC\R− which coincides onR+\ {0} with the usual logarithm. For example, one can take
L(z) = Z z
1
dζ ζ .
This operator Ls acts as a compact, trace class operator on H2 and we refer the reader to [3, 10] for a proof. The Fredholm determinant associated to this family is the Selberg zeta function defined above: for all s∈C, we have
ZΓ(s) = det(I−Ls).
This remarkable formula that dates back to the work of Pollicott [21]
in the co-compact case, is the starting point of our analysis. Unfortu- nately, the spaceH2 as defined above is not enough for our analysis and we will need to use as in [10] a family of Hilbert spaces H2(h) depend- ing on a small scale parameter 0< h≤h0. We recall the construction taken from [10], see also [3] chapter 15. Let 0< h and set
Λ(h) := Λ + (−h,+h),
then for all h small enough, Λ(h) is a bounded subset of R whose connected components have length at most Ch where C > 0 is inde- pendent of h, see [3] Lemma 15.12. Let Iℓ(h) denote these connected components, with ℓ= 1, . . . , N(h). The existence of a finite Patterson- Sullivan measure µ supported by Λ(Γ) plus Sullivan Shadow Lemma (see [3], chapter 14) show that
A−1hδN(h)≤X
ℓ
µ(Iℓ(h)) =µ(Λ(Γ)),
for some uniform A > 0, hence the number N(h) of connected com- ponents is O h−δ
. Given 1 ≤ ℓ ≤ N(h), let Dℓ(h) be the unique euclidean open disc in C orthogonal to R such that
Dℓ(h)∩R=Iℓ(h).
Now set
H2(h) :=
N(h)M
ℓ=1
H2(Dℓ(h)).
We will see in the next section that for n ≥ n0 (with n0 independent of h) that the operators Ln
s act as compact trace class operators on H2(h). Moreover, the Fredholm determinant
ZΓ(n)(s) := det(I−Ln
s )
is a multiple of the Selberg zeta function and we will count resonances using this determinant instead of the original zeta function. The goal of the next section is to provide a proof of the following fact, which is a slight modification of the argument of Guillop´e-Lin-Zworski in [10].
Proposition 2.1. For all σ0 ≤δ, there exists C >0 such that for all σ0 ≤ Re(s)≤δ and n≥n0, we have for |Im(s)| large,
log|ZΓ(n)(s)| ≤C|Im(s)|δenP(σ0), where P(σ) is the topological pressure at σ.
We refer the reader to the next§for a definition of topological pressure.
3. Basic pointwise estimates
The goal of this section is to prove the previous Proposition. We first introduce some notations. We recall that γ1, . . . , γp are generators of the Schottky group Γ. Considering a finite sequence α with
α= (α1, . . . , αn)∈ {1, . . . ,2p}n, we set
γα:=γα1 ◦. . .◦γαn.
We then denote by Wn the set of admissible sequences of length n by Wn:={α∈ {1, . . . ,2p}n : ∀ i= 1, . . . , n−1, αi+1 6=αi+p mod 2p}. We point out that if α ∈ Wn, then γα is a reduced word in the free group Γ. For all j = 1, . . . ,2p, we define Wj
n by Wj
n :={α∈Wn : αn6=j}. If α ∈ W j
n, then γα maps Dj into Dα1+p. Given the above notations and f ∈H2, we have for all z ∈ Dj and n∈N,
Ln
s (f)(z) = X
α∈Wnj
(γα′(z))sf(γα(z)).
We will need throughout the paper some distortion estimates for these maps γα. More precisely we have for all j = 1, . . . ,2p,
• (Uniform hyperbolicity). One can findC >0 and 0 < θ < θ <1 such that for all n, j and α ∈Wj
n, C−1θn≤sup
Dj
|γα′| ≤Cθn.
• (Bounded distortion). There existsM1 >0 such that for alln, j and all α∈Wj
n,
sup
Dj
γα′′
γα′
≤M1.
The second estimate, called bounded distortion, will be used constantly throughout the paper. In particular it implies that for all z1, z2 ∈ Dj, for all α∈Wj
n, we have
e−|z1−z2|M1 ≤ |γα′(z1)|
|γα′(z2)| ≤e|z1−z2|M1.
These estimates are rather standard facts in the classical ergodic theory of uniformly expanding Markov maps. For a proof of the first estimate, we refer the reader to [3] for example. Bounded distortion follows from hyperbolicity and elementary computations. Another critical tool in our analysis is the Topological pressure and Bowen’s formula. Let Ij :=Dj ∩R. The Bowen-Series map T :∪2pi=1Ii →R∪ {∞}is defined
by T(x) =γi(x) ifx∈Ii. The non-wandering set of this map is exactly the limit set Λ(Γ) of the group:
Λ(Γ) =
+∞\
n=1
T−n(∪2pi=1Ii).
The limit set is T-invariant and given a continuous map ϕ: Λ(Γ)→R, the topological pressure P(ϕ) can be defined through the variational formula:
P(ϕ) = sup
µ
hµ(T) + Z
Λ
ϕdµ
,
where the supremum is taken over all T-invariant probability measures on Λ, and hµ(T) stands for the measure-theoretic entropy. A cele- brated result of Bowen [6] says that if one considers the map σ 7→
P(−σlog|T′|) then this map is convex, strictly decreasing and van- ishes exactly at σ = δ(Γ), the Hausdorff dimension of the limit set.
An alternative way to compute the topological pressure is to look at weighted sums on periodic orbits i.e. we have
(1) eP(ϕ) = lim
n→+∞
X
Tnx=x
eϕ(n)(x)
!1/n
,
with the notation ϕ(n)(x) = ϕ(x) +ϕ(T x) +. . .+ϕ(Tn−1x). We will seldom use the two previous formulas in our analysis but will rather use the following upper bound. For simplicity, we will use the notation P(σ) in place of P(−σlog|T′|).
Lemma 3.1. For all σ0 < M in R, one can find C0 >0 such that for all n≥1 and M ≥σ ≥σ0, we have
(2)
X2p j=1
X
α∈Wnj
sup
Ij
|γα′|σ
≤C0enP(σ0).
Proof. For all σ ≥ σ0 and x ∈ Ij, we have by bounded distortion property
X
α∈Wnj
sup
Ij
|γ′α|σ ≤CM X
α∈Wnj
(γα′(x))σ0 =CMLn
σ0(1)(x).
The Ruelle-Perron-Frobenius theorem [2] applied to Lσ
0 : C1(I) → C1(I) (I =∪jIj) says that this operator is quasi-compact, has spectral radius eP(σ0) and eP(σ0) is the only eigenvalue on the circle
{|z|=eP(σ0)}.
Moreover, eP(σ0) is a simple eigenvalue whose spectral projection is given by
f 7→ P(f) :=ϕ0
Z
I
f dµ0,
where µ0 is a probability measure andϕ0 is a positiveC1 density onI. Using Holomorphic functional calculus, we can therefore decompose
Ln
σ0 =enP(σ0)P +Nn,
where N has spectral radius at most θ0eP(σ0) for some θ0 < 1. It is now clear that we have
X
α∈Wnj
sup
Ij
|γα′|σ ≤C0supϕ0enP(σ0)+C1θe0nenP(σ0),
for some constants C0, C1 and θ0 <θe0 <1 and the proof is done.
We can now go back to the action of Ln
s :H2(h)→H2(h).
Set Ej(h) := {1 ≤ ℓ ≤ N(h) : Dℓ(h) ⊂ Dj}. Given α ∈ Wj
n and ℓ ∈ Ej(h), we have provided n is large enough,
γα(Dℓ(h))⊂ Dℓ′(h)
for someℓ′ ∈ {1, . . . , N(h)}. Indeed, since the limit set Λ is Γ-invariant, γα maps Dj(h) into a disc of size at most Chθn which contains an element of Λ. Taking n large enough (uniformly onh), this disc has to be inside ∪ℓDℓ(h). Therefore Ln
s leaves H2(h) invariant as long as n is large. We will actually need something more quantitative which is proved in [10, 3]:
Lemma 3.2. There existsn0, κ >0such that for all n≥n0, α∈Wj
n, we have for all ℓ∈Ej(h), γα(Dℓ(h))⊂ Dℓ′(h) with
dist(γα(Dℓ(h)), ∂Dℓ′(h))≥κh.
To estimate the pointwise growth of ZΓ(n)(s) = det(I−Ln
s ) we will use some singular values estimates on the space H2(h) with h=|Im(s)|−1. Notice that this zeta function does not depend on h. Indeed, provided Re(s)> δ, we have the formula
(3) det(I−Ln
s ) = exp − X∞ k=1
1
kTr(Lnk
s )
! ,
and the trace can be computed (using a fixed point analysis, see [3]):
Tr(Lp
s) = X
Tpx=x
(Tp)′(x)−s 1−((TP)′(x))−1,
the above sums over periodic orbits of the Bowen-Series are clearly independent of the choice of function space, hence of h which will be adjusted to optimize our estimates. First we need to recall some facts on singular values of compact operators and our basic reference is the book of Simon [22]. If H is a complex Hilbert space and T : H → H is a compact operator, the singular values µk(T) are the eigenvalues
(ordered decreasingly) of the self-adjoint positive operator √
T∗T. If we have
X∞ k=1
µk(T)<+∞,
then T is said to be trace class. If T is trace class, then the eigenvalue sequence
|λ1(T)| ≥ |λ2(T)| ≥. . .≥ |λk(T)| is summable and the trace Tr(T) is defined by
Tr(T) = X∞
k=1
λk(T), while the determinant det(I+T) is given by
det(I +T) :=
Y∞ k=1
(1 +λk(T)). Weyl’s inequalities show that we have
Y∞ k=1
(1 +|λk(T)|)≤ Y∞ k=1
(1 +µk(T)). In this paper we will use the following remark:
Lemma 3.3. Let T :H → H be a trace class operator and(eℓ)ℓ∈N is a Hilbert basis of H, then we have
log|det(I+T)| ≤ X∞
ℓ=0
kT(eℓ)k. Proof. By Weyl’s inequality, we write
log|det(I+T)| ≤ X∞ k=0
log(1 +µk(T))≤ X∞ k=0
µk(T) = Tr(√ T∗T).
But the classical Lidskii theorem says that Tr(√
T∗T) = X∞
ℓ=0
h√
T∗T eℓ, eℓi ≤ X∞
ℓ=0
k√
T∗T eℓk, by Schwartz inequality, but we have k√
T∗T eℓk=kT eℓk.
We can now give a proof of Proposition 2.1. First we need an explicit basis of H2(h). Setting forℓ ∈ {1, . . . , N(h)},
Dℓ(h) :=D(cℓ, rℓ)
with cℓ ∈ R and h ≤rℓ < Ch, we denote by eℓk the function in H2(h) defined for z ∈ Dj(h) by
eℓk(z) =
( 0 if j 6=ℓ qk+1
π 1 rj
z−cj
rj
k
if j =ℓ.
It is straightworfard to check that (eℓk)k∈N,1≤ℓ≤N(h) is a Hilbert basis of H2(h). We now use Lemma 3.3 and write
log|ZΓ(n)(s)| ≤ X∞ k=0
N(h)X
ℓ=1
kLn
s (eℓk)kH2(h). Using Schwartz inequality, we get (N(h) =O(h−δ))
(4) log|ZΓ(n)(s)| ≤Ch−2δ X∞
k=0
N(h)X
ℓ=1
kLn
s (eℓk)k2H2(h)
1/2
. On the other hand we have
NX(h) ℓ=1
kLn
s(eℓk)k2H2(h)= X2p
j=1
X
ℓ′∈Ej(h)
Z
Dℓ′(h) N(h)X
ℓ=1
|Ln
s (eℓk)|2dm,
wherem is the usual Lebesgue measure onC. Givenℓ′ ∈Ej(h), we can write
(5) Z
Dℓ′(h) N(h)X
ℓ=1
|Ln
s (eℓk)|2dm≤ X
α,β∈Wnj
Z
Dℓ′(h)|(γα′)s||(γβ′)s|Fα,β(k)dm, where
Fα,β(k)(z) :=
NX(h) ℓ=1
|eℓk◦γα(z)||eℓk◦γβ(z)|. We now need to prove the following remark.
Lemma 3.4. Setting Ωj(h) := ∪ℓ∈Ej(h)Dℓ(h), we have for all α, β ∈ W j
n,
sup
Ωj(h)|Fα,β(k)| ≤Ch−2ρk, with C and 0< ρ < 1 uniform.
Proof. First remark that given z ∈ Ωj(h), we have either Fα,β(k)(z) = 0 or
Fα,β(k)(z) =|eℓk0 ◦γα(z)||eℓk0◦γβ(z)|,
for some ℓ0 ∈ {1, . . . , N(h)}. Then combine Lemma 3.2 with the ex- plicit formula for eℓk0 to obtain the result.
Going back to estimate (5), and using the fact (use bounded distortion) that we set |Im(s)|=h−1
sup
z∈Dℓ′(h)|(γα′)s| ≤Ckγα′kRe(s)∞,j , we have reached
N(h)X
ℓ=1
kLn
s (eℓk)k2H2(h) ≤Ch−δρk
X2p
j=1
X
α∈Wnj
sup
Ij
|γα′|σ
2
≤Ch−δρke2nP(σ).
The proof is done by inserting the above estimate in formula (4) and summing over k.
4. Applying Littlewood’s Lemma
We now show how to reduce the proof of Theorem 1.1 to a mean square estimate on transfer operators. To this end we apply a result of Littlewood taken from the classics. More precisely, we prove the following. Define M(σ, T) by
M(σ, T) := #{s∈ RX : σ≤Re(s)≤δ and T /2≤Im(s)≤T}. Proposition 4.1. Let σ0 < σ < δ, then for all T large and n(T) = [νlogT] with ν >0 small enough, we have
M(σ, T)≤C1
Z T T /2
log|ZΓ(n(T))(σ0+it)|dt+C2T.
Proof. We start by recalling a version of Littlewood’s Lemma which suits our needs. Let σ0 < 1 and T > 0. Let f be a function which is holomorphic on a neighborhood of the rectangle
RT,σ0 := [σ0,1] +i[T /2, T],
and assume thatf does not vanish on the segment 1+i[T /2, T]. Denote by ZT,σ0 the set of zeros of f onRT,σ0. Then we have the formula
2π X
z∈ZT ,σ0
(Re(z)−σ0) = Z T
T /2
log|f(σ0+it)|dt− Z T
T /2
log|f(1 +it)|dt
+ Z 1
σ0
Arg(f(σ+iT))dσ− Z 1
σ0
Arg(f(σ+iT /2))dσ.
The function Arg(f) is the imaginary part of a determination of a complex logarithm logf which is defined by taking upper limits of a holomorphic logarithm on a suitable simply connected domain (see Titchmarsh [27], section 9.9, for more details). The resulting function Arg(f) is well defined on RT,σ0 \ Z and is discontinuous on a finite union of segments.
Fix σ0 < σ < δ <1. Then since RX is a subset of the set of zeros of ZΓ(n)(s), applying the above formula to ZΓ(n)(s) we get
2π(σ−σ0)M(σ, T)≤ Z T
T /2
log|ZΓ(n(T))(σ0+it)|dt
+O Z T
T /2
log|ZΓ(n(T))(1 +it)|dt
+O
sup
σ0≤σ≤1|Arg(ZΓ(n(T))(σ+iT))|
+O
sup
σ0≤σ≤1|Arg(ZΓ(n(T))(σ+iT /2))|
.
Remark that combining (1) and the trace formula (3) shows (P(1)<0) that ZΓn(T)(s) is uniformly bounded on {Re(s) = 1}, hence
Z T T /2
log|ZΓ(n(T))(1 +it)|dt=O(T).
To control Arg(ZΓ(n)(s)), we will use another classical result of Titch- marsh essentially based on Jensen’s formula, see [27], 9.4.
Lemma 4.2. Fix 2 > σ0 > 0 and let f be a holomorphic function on the half-plane {Re(s)> 0}. Suppose that |Re(f(2 +it))| ≥m >0 for all t ∈ R and assume that f(s) = f(s) for all s. Suppose in addition that |f(σ+it)| ≤Aσ,t, then if T is not the imaginary part of a zero of f(s), for all σ≥σ0,
|Arg(f(σ+iT))| ≤Cσ0(logAσ0,T+2−logm) + 3π/2.
To be able to apply the above Lemma, we need to prove the following.
Lemma 4.3. There exists m >0 such that for all n large enough, we have for all t,
|Re(ZΓ(n)(2 +it))| ≥m.
Proof. First remark that for Re(s)> δ,
Re(det(I−Ln
s )) = exp − X∞ k=1
1
kRe(Tr(Lnk
s ))
! cos
X∞ k=1
1
kIm(Tr(Lnk
s )
! , and the trace formula combined with the pressure formula (1) shows that for all ǫ >0 and p large,
|Tr(Lp
s)| ≤Cep(P(Re(s))+ǫ). We therefore obtain
|Re(ZΓ(n)(2 +it))| ≥exp −Cen(P(2)+ǫ)
|cos Cen(P(2)+ǫ)
|. Since P(2) < 0 by Bowen’s formula, we have a uniform bound from below as long as n is taken large.
To finish the proof of Proposition 4.1, we use the pointwise estimate of Proposition 2.1 which combined with Titchmarsh Lemma gives
sup
σ0≤σ≤1|Arg(ZΓ(n(T))(σ+iT))|=O Tδ+νP(σ0)
=O(T),
as long as we choose ν≤ P1−δ(σ0). The proof is done, provided that there are no zeros on{Im(s) = T}and{Im(s) =T /2}. IfZΓ(n(T))(s) vanishes on these lines, we simply replace T by some T < Te ≤ T + 1 so that ZΓ(n(T))(s) does not vanish on {Im(s) = Te} and T /2 by some T′ with T /2−1≤T′ < T /2 so thatZΓ(n(T))(s) does not vanish on{Im(s) = T′}. We then write
M(σ, T)≤#{s ∈ RX : Re(s)≥σ and T′ ≤Im(s)≤Te}
≤C Z Te
T′
log|ZΓ(n(T))(σ0+it)|dt+O(T)
=C Z T
T /2
log|ZΓ(n(T))(σ0+it)|dt+O(T), by the pointwise estimate of proposition 2.1.
We now state the main estimate which is the core argument of the paper. Let ϕ0 ∈ C0∞(R) be a smooth compactly supported function such that Supp(ϕ0)⊂[−1,+1],ϕ0 >0 on (−1,+1) andR
ϕ0(x)dx= 1.
We define a probability measure µT onR by the formula Z
R
f dµT := 1 T
Z +∞
−∞
ϕ0
t−2T T
f(t)dt.
Proposition 4.4. There existν >0, 0< ρ < 1such that for allσ > δ2 one can find ε(σ)>0 such that
NX(h) ℓ=1
Z
R
kL(n(T))
σ+it (eℓk)k2(h)dµT(t)≤CρkTδ−ε(σ), where we have taken n(T) := [νlogT], h =T−1.
The proof of this proposition is postponed to §6 and occupies the rest of the paper. Let us show how the combination of Proposition 4.4 and Proposition 4.1 implies Theorem 1.1. By Proposition 4.1, we have for
δ
2 < σ0 < σ, M
σ,5
2T
≤C Z 52T
5 4T
log|ZΓ(n(T))(σ0+it)|dt+O(T), and we write
Z 52T
5 4T
log|ZΓ(n(T))(σ0+it)|dt
≤
[−3/4,1/2]inf ϕ0 −1
T Z
R
log|ZΓ(n(T))(σ0+it)|dµT(t).
Using Lemma 3.3, we obtain Z
R
log|ZΓ(n(T))(σ0 +it)|dµT(t)≤X
k∈N N(h)X
ℓ=1
Z
RkL(n(T))
σ0+it (eℓk)k(h)dµT(t), which by Schwartz inequality is less than
Ch−2δX
k∈N
N(h)X
ℓ=1
Z
R
kL(n(T))
σ0+it (eℓk)k2(h)dµT(t)
1/2
≤CTδ−ǫ(σ0)/2. We have therefore obtained
M(σ, T) =O
T1+max
n δ−1
2ǫ(σ0),0o .
To get an estimate on the counting function N(σ, T), we just write N(σ, T)≤
N0
X
k=0
M
σ, T 2k
+O(1) where N0 is such that 2TN0 ≤1. Since we have for T large,
N0
X
k=0
M
σ, T 2k
≤CT1+max
n δ−1
2ǫ(σ0),0o
,
with C independent on T, the proof is done.
5. A key lower bound Given α, β ∈W j
n, we set for all x∈Ij =Dj ∩R, Φα,β(x) := log|γα′(x)| −log|γβ′(x)|.
The goal of this section is to prove the following fact which will be a key result in the next section on mean square estimates.
Proposition 5.1. There exists η0 >0and 1> θ >0 such that for all n ≥1 and all j = 1, . . . ,2p, we have for all α6=β ∈W j
n,
x∈Iinfj|Φ′α,β(x)| ≥η0θn.
The proof of this proposition will follow from the next Lemma which is of geometric nature.
Lemma 5.2. Let Γ be a Schottky group as above, then there exists a constant CΓ >0 such that for all
γ ≃
a b c d
∈Γ\ {Id}, |c| ≥CΓ.
Proof. Let Γ6=Id be an element of Γ, and assume that γ ≃
a b c d
,
withc= 0. By looking atγ−1 one can assume that|a| ≥1. In addition, since Γ has no non-trivial parabolic element, we have actually |a|>1.
The action of γ on the Riemann sphere is therefore of the type γ(z) =a2z+ab.
Now pick an element of Λ(Γ) which is different from the fixed point of γ. Its orbit under the action of (γn)n∈N goes to infinity as n → +∞ which contradicts the fact that Λ isγ-invariant and compact. Therefore c6= 0. Remark that by definition of Γ, we have
γ−1(∞) =−d/c∈ ∪2pj=1Dj.
Consequently, one can find a constant Q1 such that for all γ 6= Id,
|d/c| ≤Q1. Notice also that we have (i=√
−1) Im(γ(i)) = 1
c2+d2.
Since the limit set Λ(Γ) is compact, the orbit Γ.i has to be bounded in C for the usual euclidean distance. Hence there exists Q2 such that for all γ ∈Γ,
1
c2+d2 ≤Q2. As a consequence we get
1≤c2(Q2+Q21), and the proof is done.
We can now complete the proof of Proposition 5.1. Writing γα(x) = aαx+bα
cαx+dα
, with aαdα−bαcα = 1, we have
|Φ′α,β(x)|= 2 |cβdα−cαdβ|
|cβx+dβ||cαx+dα| = 2|cβdα−cαdβ|(γα′(x))1/2(γβ′(x))1/2. We can now remark that since Γ is a free group,α6=βimpliesγα◦γβ−1 6= Id, and we have the formula
γα◦γβ−1 ≃
aα bα
cα dα
dβ −bβ
−cβ aβ
=
∗ ∗ cαdβ−dαcβ ∗
. We can therefore apply the above Lemma and the proof ends by using bounded distortion and the lower bound for the derivatives.
6. Mean square estimates
The goal of this final section is to prove Proposition 4.4. We recall that in the sequel we take
h=T−1, n(T) = [νlogT], 0< ν <<1.
We first start by writing
NX(h) ℓ=1
Z
RkL(n(T))
σ+it (eℓk)k2(h)dµT(t) = X2p j=1
Z
Ωj(h)
Z
R N(h)X
ℓ=1
|L(n(T))
σ+it (eℓk)|2dµTdm.
For all z ∈ Dj and α, β ∈Wj
n we use the notation Φα,β(z) :=L(γα′(z))−L(γβ′(z)),
where L is the adequate complex logarithm. This notation coincides on the real axis with the one introduced in the previous section. Re- mark that since Φα,β is real on the real axis and because of bounded distortion, we have on any disc Dℓ(h),
sup
z∈Dℓ(h)|eiTΦα,β(z)|=O(1),
uniformly on T. For allz ∈Ωj(h), we have by a change of variable Z
R NX(h)
ℓ=1
|L(n(T))
σ+it (eℓk)(z)|2dµT = X
α,β∈Wnj
(γα′(z))σ(γβ′(z))σG(k)α,β(z)cϕ0(−TΦα,β(z))e2iTΦα,β(z), where ϕc0 is the usual Fourier transform of ϕ0 defined by
c ϕ0(ξ) :=
Z +∞
−∞
ϕ0(x)e−ixξdx, and G(k)α,β is the following sum (see also Lemma 3.4):
G(k)α,β(z) :=
N(h)X
ℓ=1
eℓk◦γα(z)eℓk◦γβ(z).
The goal now is to gain some decay asT goes to +∞by using the key observation of Lemma 5.1. We split the above sum into two contribu- tions, the ”diagonal” one plus the ”off-diagonal” one:
X
α,β∈Wnj
(γα′(z))σ(γβ′(z))σG(k)α,β(z)cϕ0(−TΦα,β(z))e2iTΦα,β(z)=Sdiag+Sof f diag, where we have set
Sdiag := X
α∈Wnj
|γα′(z)|2σG(k)α,α(z)cϕ0(−TΦα,α(z))ei2TΦα,α(z),
Sof f diag := X
α6=β∈Wnj
(γα′(z))σ(γβ′(z))σG(k)α,β(z)cϕ0(−TΦα,β(z))e2iTΦα,β(z). We first deal with the diagonal contribution, which by bounded distor- tion and the pressure estimate (together with Lemma 3.4) gives
|Sdiag| ≤Cen(T)P(2σ)h−2ρk, and therefore
Z
Ωj(h)|Sdiag|dm≤Ch−δen(T)P(2σ)ρk.
We clearly have a gain as long asP(2σ)<0, which by Bowen’s formula [6] amounts to say that σ > δ2. To deal with the off-diagonal sum, we will of course use the estimate for ξ ∈Cand all q≥0,
|cϕ0(ξ)| ≤Cq
e|Im(ξ)|
(1 +|ξ|)q, which implies that for α6=β and allq, we have
|cϕ0(−TΦα,β(z))|=Oq (1 +|TΦα,β(z)|)−q .
The trouble comes from Φα,β(z) which may be vanishing on some part of Ωj(h). We prove the following Lemma.
Lemma 6.1. Fix an ǫ > 0. There exists ν small enough (recall that n(T) = [νlogT]) such that for all α, β ∈Wj
n with α6=β, one can split the set Ej(h) =E′
j ⊔E′′
j so that:
• We have #E′
j =O(h−δ2).
• For all ℓ∈E′′
j and z ∈ Dℓ(h),
|Φα,β(z)| ≥Ch1−δ2+ǫ.
Proof. By Proposition 5.1, for all α 6= β, the function x 7→ Φα,β(x) is strictly monotonic on the interval Ij, and its derivative is uniformly bounded from below by C1(θ)n. Two cases can occur:
• Either Φα,β(x) vanishes at some pointx0 ∈Ij and we setJα,β = [x0 −hη, x0 +hη], where 0 < η < 1 will be adjusted later on.
Then for all x∈Ij\Jα,β we have
|Φα,β(x)| ≥C1hη(θ)n.
• The map x 7→ Φα,β(x) does not vanish on Ij := (aj, bj). Ac- cording to the sign of Φα,β, we set Jα,β = [aj, aj + hη] or Jα,β = [bj−hη, bj]. In both cases we have forx∈Ij\Jα,β,
|Φα,β(x)| ≥C1hη(θ)n.
Now define E′′
j by E′′
j ={ℓ ∈Ej(h) : Dℓ(h)∩Jα,β =∅}. Given ℓ ∈E′′
j , we have by bounded distortion for allz ∈ Dℓ(h),
|Φα,β(z)| ≥C1hη(θ)n−C2h ≥C3hη+ǫ, provided η+ǫ <1 and ν|logθ| ≤ǫ. It remains to count
E′
j :=Ej(h)\E′′
j . Except for possibly two of them,ℓ ∈E′
j impliesR∩Dℓ(h)⊂Jα,β, hence by volume comparison
(#E′
j −2)Ch≤ |Jα,β|=hη. Therefore #E′
j =O(hη−1). The proof ends by choosing η= 1− δ2. Going back to Sof f diag, we have using Lemma 3.4,
Z
Ωj(h)|Sof f diag|dm≤Cqρkh−2X
α6=β
kγα′kσ∞kγβ′kσ∞
Z
Ωj(h)
dm(z) (1 +|TΦα,β(z)|)q. We then write
Z
Ωj(h)
dm(z)
(1 +|TΦα,β(z)|)q = Z
Ω′j(h)
dm(z)
(1 +|TΦα,β(z)|)q+ Z
Ω′′j(h)
dm(z) (1 +|TΦα,β(z)|)q, with the notations
Ω′j(h) := [
ℓ∈E′
j
Dℓ(h) and Ω′′j(h) := [
ℓ∈E′′
j
Dℓ(h).
We recall to the reader that
#E′
j =O(h−δ2) and #E′′
j =O(h−δ).
Applying the above Lemma and taking ǫ >0 small enough so that
δ
2 −ǫ >0, we get with T =h−1 and q large
Z
Ωj(h)
dm(z)
(1 +|TΦα,β(z)|)q =O
h2−δ2
+O
h2−δ+q(δ2−ǫ)
=O
h2−δ2
.
Therefore, Z
Ωj(h)|Sof f diag|dm≤Cρke2nP(σ)h−δ2. Adding all of our estimates, we have reached
NX(h) ℓ=1
Z
R
kL(n(T))
σ+it (eℓk)k2(h)dµT(t)≤Cρk
Tδ+νP(2σ)+Tδ2+2νP(σ)
.
Now taking
ν≤ min
( δ
8|logθ|, δ
16P(δ2),1−δ P(δ2)
)
yields for all σ ≥ δ2,
NX(h) ℓ=1
Z
RkL(n(T))
σ+it (eℓk)k2(h)dµT(t)≤ Cρk Tδ+νP(2σ)+T3δ/4 . The proof of Proposition 4.4 is done. We can now say a few more words on the function τ(σ) of the main Theorem 1.1: using the above choice of ν then for all σ > σ0 ≥ δ2, we can take
τ(σ) = max
δ+ν
2P(2σ0),7δ 8
,
so for example taking σ0 = 12(σ+δ2) gives
(6) τ(σ) = max
δ+ν
2P(σ+δ2),7δ 8
.
We have not attempted to optimize the choice of constants at all. For- mula (6) shows in addition that τ(σ) is strictly decreasing and convex in a right neighborhood of δ2 since the topological pressure has this property. Moreover, the pressure functional σ 7→P(σ) is real analytic and its derivative at σ =δ is given by
P′(δ) =− Z
I
log|T′|dµδ <0,
where I =∪2pi=1Ii, T is the Bowen-Series map andµδis the equilibrium state at δ. In particular, this gives a formula for the derivative
τ′(δ2) =−ν 2
Z
I
log|T′|dµδ
which is of course non-vanishing.
References
[1] Nalini Anantharaman. Spectral deviations for the damped wave equation.
Geom. Funct. Anal., 20(3):593–626, 2010.
[2] Viviane Baladi. Positive transfer operators and decay of correlations, vol- ume16ofAdvanced series in Nonlinear dynamics. World Scientific, Singapore, 2000.
[3] David Borthwick. Spectral theory of infinite-area hyperbolic surfaces, volume 256 of Progress in Mathematics. Birkh¨auser Boston Inc., Boston, MA, 2007.
[4] Jean Bourgain, Alex Gamburd, and Peter Sarnak. Generalization of Selberg’s 3/16 theorem and affine sieve.Arxiv preprint, 2009.
[5] Jean Bourgain and Alex Kontorovich. On representations of integers in thin subgroups of SL2(Z).Geom. Funct. Anal., 20(5):1144–1174, 2010.
[6] Rufus Bowen. Hausdorff dimension of quasicircles. Inst. Hautes ´Etudes Sci.
Publ. Math., (50):11–25, 1979.
