Kloosterman-Fourier inversion for symmetric matrices

133(3), 2005, p. 331–348

KLOOSTERMAN-FOURIER INVERSION FOR SYMMETRIC MATRICES

by Omer Offen

Abstract. — We formulate a Kloosterman transform on the space of generalized Kloosterman integrals on symmetric matrices, and obtain an inversion formula. The formula is a step towards a fundamental lemma of the Jacquet type. At the same time it hints towards a conjectural relative trace formula identity, associated with the metaplectic correspondence.

R´esum´e(Inversion de Kloosterman-Fourier pour les matrices sym´etriques)

Nous d´efinissons une transformation de Kloosterman sur l’espace des int´egrales de Kloosterman g´en´eralis´ees sur les matrices sym´etriques et nous obtenons une formule d’inversion. Cette formule est une ´etape vers un lemme fondamental de type de Jacquet.

En mˆeme temps, elle indique une identit´e conjecturale de la formule des traces relative associ´ee `a la correspondance m´etaplectique.

1. Introduction

LetF be a non-archimedean local field,OF the ring of integers inF and℘ the maximal ideal ofOF. Let | · |denote the normalized absolute value onF so that for a uniformizer$ ofF we have|$|⁻¹= #(OF/℘) is the size of the

Texte re¸cu le 23 avril 2003, r´evis´e le 12 f´evrier 2004, accept´e le 3 mars 2004

Omer Offen, Max-Planck-Institut f¨ur Mathematik, Vivatsgasse 7, D-53111 Bonn (Germany) E-mail :[email protected] • Url :http://www.math.columbia.edu/~omer/

2000 Mathematics Subject Classification. — 11F70, 11L05.

Key words and phrases. — Kloosterman integrals, Kloosterman transform, Jacquet’s relative trace formula.

BULLETIN DE LA SOCI´ET´E MATH ´EMATIQUE DE FRANCE 0037-9484/2005/331/$ 5.00 c

Soci´et´e Math´ematique de France

residual field. Let ψ be a non-trivial additive character of F. We recall the formula

(1)

Z fb(x)ψ(ax²)dx=|2a|⁻¹²γ(a, ψ) Z

f(x)ψ(−a⁻¹x²)dx

which we use to define the Weil constant γ. Here a ∈ F^×, f ∈ C_c^∞(F) is a Schwartz function onF andfbis the Fourier transform off defined by

fb(x) = Z

f(y)ψ(−2xy)dy.

The measure dxis the self dual Haar measure onF with respect toψ. Thus, it satisfies

(2) f(0) =|2|

Z fb(x)dx.

LetN=Nnbe the subgroup of upper triangular unipotent matrices in GLn(F).

Define the non-degenerate characterθ=θn ofN by θ

u

=ψⁿ⁻¹X

i=1

xi,i+1

whereu= (xi,j)∈N. The Haar measure dxonF determines a Haar measure onN and a self dual Haar measure on any finite dimensional F-vector space.

We will use the measures determined by dxunless otherwise specified. Denote byMm×n(F) the set of allm×nmatrices with entries inF. Let

Mn(F) =Mn×n(F) and denote byS=Sn the space of symmetric matrices

S =

X ∈Mn(F); ^tX=X . We consider the action (u, s)7→^tusuofN onS.

Definition 1.1. — An element s∈ S is calledrelevant if θ is trivial on the stabilizerNs ofsinN.

Our objects of interest are the generalized Kloosterman integrals

(3) ω[Φ, ψ;s] =

Z

Ns\N

Φ(^tusu)θ(u²)du for a relevants∈ S,Φ∈C_c^∞(S). Let

Sn=Sn∩GLn(F).

The orbits in Sn are fully described in [6]. To describe a set of representatives for all orbits in Sn we view the elements of the Weyl group as permutation matrices in GLn(F). Thus a complete set of representatives for the orbits inSn is the set of allwa, wherewis the longest element in the Weyl group of a

o

standard Levi subgroupM of GLn(F) andais in the center ofM. All relevant orbits inSnwith zero determinant contain an element of the form ^s0

, where s∈Sn−1. This is proved in [3] for Hermitian matrices. The proof for symmetric matrices is identical and we omit it here. Representatives for the relevant orbits of zero determinant are therefore given as above in terms of representatives of orbits in Sn−1. When w= 1 andais a diagonal matrix, the stabilizer ofain Nn is trivial.

In this sense the diagonal matrices are representatives of the largest orbits.

In a sense explained in [3] and [2], the Kloosterman integrals for smaller orbits, i.e. withw6= 1 are determined by Kloosterman integrals of the largest orbits.

For this reason, our main concern in this work is the space of Kloosterman integrals, restricted to the relevant diagonal matrices. These are of the form a = diag(a1, . . . , an) where a1, . . . , an−1 ∈ F^× and an ∈ F. We will denote byω[Φ, ψ;a1, . . . , an] the Kloosterman integralω[Φ, ψ; diag(a1, . . . , an)].

To state our main theorem it will be convenient to introduce a normalization.

We introduce the normalizing factors

σn(a) =aⁿ⁻¹₁ aⁿ⁻²₂ · · ·an−1, Γn(a, ψ) =γ(a1, ψ)ⁿ⁻¹γ(a2, ψ)ⁿ⁻²· · ·γ(an−1, ψ).

The normalized Kloosterman integral is

(4) ωe^ψ[Φ, ψ;a1, . . . , an] = Γn(−a, ψ)σ_n(a)¹²ω[Φ, ψ;a1, . . . , an].

The purpose of the notationωe^ψis to emphasize the dependence of the normal- ization on the characterψ. Let Ωψ,nbe the space of functionsωon (F^×)ⁿ⁻¹×F of the form

ω(a1, . . . , an) =ωe^ψ[Φ, ψ;a1, . . . , an]

for some Φ ∈ C_c^∞(S). Denote by [·,·] : F^× ×F^× 7→ {±1} the quadratic Hilbert symbol. It is defined by the condition [a, b] = 1 iffais representable by the quadratic forma=x²−by². We define the Kloosterman transformKψ,n

on Ωψ,n by

Kψ,nω(a1, . . . , an) = Z

ω(p1, . . . , pn)ψ

− Xn i=1

pian+1−i+

n−1X

i=1

1 pian−i

(5)

×ⁿ⁻¹Y

i=1 n−iY

j=1

[ai, pj]

dpndpn−1· · ·dp1

where the integral overpi∈F,i= 1, . . . , nis only iterated. Although the inte- grand isa priorionly defined forp1, . . . , pn−1∈F^× andpn∈F we make sense of (5) in the proof of theorem 1.2. Denote by wn ∈GLn(F) the permutation matrix with a unit anti-diagonal. For any matrixX ∈Mn(F) we will denote by Tr(X) the trace ofX. For a function Φ∈C_c^∞(S) let

(6) Φ(s) =ˆ

Z

S

Φ(t)ψ −Tr(st) dt

be the standard Fourier transform of Φ. We will consider the Fourier transform

(7)

ˇ

ˆ

of Φ. Our main theorem is

Theorem 1.2. — The integral (5) defining the Kloosterman transform on Ωψ,n is a convergent iterated integral. Moreover, let Φ∈C_c^∞(S)then,

Kψ,nωe^ψ[Φ, ψ;.]

(a1, . . . , an) (8)

=|2|^12n(n−1)γ(1,ψ)¯ ^12n(n−1)eω^ψ¯[

ˇ

The theorem shows in particular thatKψ,nis a transform from Ωψ,nto Ωψ,n¯ . Combining the theorem with Fourier inversion onS we obtain the inversion of the Kloosterman transform.

Corollary 1.3. — The Kloosterman transform satisfies Kψ,n¯ ◦Kψ,n=|2|ⁿ⁽ⁿ⁻¹⁾Id whereId is the identity map onΩψ,n.

The motivation to the problem lies in a conjectural trace formula identity of the Jacquet type. The identity is concerned with the metaplectic corre- spondence of [1]. It is a lifting of genuine automorphic representations of the metaplectic double coverGLfn of GLn to automorphic representations of GLn. Jacquet suggests the following characterization for the image of this lift: A cus- pidal automorphic representation of GLnwith trivial central character is a lift- ing from GLfn iff it is (H, χ)-distinguished for some subgroupH of orthogonal similitudes of GLn and some id´ele class quadratic characterχ.

For more detail and definitions we refer to [6]. This characterization of the image of metaplectic correspondence will follow from the relative trace formula identity

(9)

Z

KΦ(u)θ(u²)du= Z

Kf(u1, u2)θ(u1u2)du1du2.

Herekis a global field. The integration is overu, u1, u2∈Nn(k)\Nn(A^k), where on the right hand sideNnis viewed as its splitting inGLfn,KfandKΦare kernel functions depending on the quadratic characterχ, of operators corresponding to the smooth functions of compact supportf onGLfn(Ak) and Φ onSn(Ak).

Again for more details we refer to [6]. The fundamental lemma for this situation is a matching of Kloosterman integrals. If Φ0 is the characteristic function of S ∩KwhereK= GLn(OF) is the standard maximal compact of GLn(F), then ω[Φ0, ψ;a] matches in an appropriate way, an integral of the form

(10)

Z

N×N

Ψ0(^tu1au2)θ(u1u2)du1du2

o

where Ψ0is the unit element of the genuine spherical Hecke algebra ofGLfn(F).

In [6], Mao proved the fundamental lemma for the case n = 3 by brute force computation. In [4], Jacquet developed a method to prove Kloosterman inte- gral identities of the above type. The method requires in both sides an inversion formula for a Fourier-Kloosterman transform on the space of Kloosterman inte- grals. For the case of a quadratic extension the inversion formulas are obtained in [3] and the method is carried out in [4] to prove the identity of Kloosterman integrals which serves as a fundamental lemma for a relative trace formula. In this work we provide a step towards the fundamental lemma associated with the trace formula (9). At the same time, the formula (5) and mainly the os- cillating factorQn−1

i=1

Qn−i

j=1[ai, pj] in it, hint to a relation with the metaplectic group. Ifσis the 2-cocycle that defines multiplication inGLfn(F) as defined in [5], then fora= diag(a1, . . . , an) andp= diag(p1, . . . , pn) we have

(11) σ(a, wnpwn) =

n−1Y

i=1 n−iY

j=1

[ai, pj].

The rest of this manuscript is organized as follows: The main tool we use to prove Theorem 1.2 is Weil’s formula. In Chapter 2 we write it in a form convenient for our needs. We then prove the theorem by induction. Chapter 3 provides an inversion formula for some intermediate integrals designed to use an inductive argument. In Chapter 4 the inductive step is carried out to finish the proof of the inversion. Chapter 5 provides a much simpler formula associated with the smallest orbits. We present it here, since once the analogous results for the metaplectic case will be obtained, the method of Jacquet requires this formula in order to prove smooth matching. The proof of the inversion formula closely follows the guidelines of [3], the new ingredient is the occurrence of the Hilbert symbol in the Kloosterman transform. The problem was suggested to me by Jacquet. For the project and for much help and support, I am most thankful to him. Most of this work was written during my visit at IH´ES.

I thank the IH´ES for a very pleasant and productive visit.

2. Weil’s formula Let

Vn,m=n0n X

tX 0m

; X ∈Mn×m(F)o . We viewVn,mas a self-dual space via the pairing

(12) 0n X

tX 0m

,0n tY Y 0m

7−→Trh

−0n X

tX 0m

0n tY Y 0m

i

whereX ∈Mn×m(F) andY ∈Mm×n(F). The Fourier transform of a function Φ∈C_c^∞(Vn,m) is defined by

(13)

ˆ

= Z

Mn×m

Φ0n Y

tY 0m

ψ

−2 Tr(XY) dY,

X ∈Mm×n(F). We recall Weil’s formula for the Fourier transform of a char- acter of second order for the spaceVn,m. Taking our definition of the Fourier transform into acount, from [7] we have that for all A∈Sn, C∈Sm there is a constant γ_mⁿ(A, C, ψ) such that for all Φ∈C_c^∞(Vn,m)

Z

Mn×m(F)

Φ0n X

tX 0m

ψ

Tr(C^tXAX) dX (14)

=|2|^12mn· |detA|^−12m· |detC|⁻¹²ⁿ

×γ_mⁿ(A, C, ψ) Z

Mm×n(F)

ˆ

ψ

−Tr(C⁻¹ZA^−1tZ) dZ.

ForP ∈Mm×n(F) the Fourier transform of the function

(15) 0n X

tX 0m

7−→Φh0n X

tX 0m

iψ

Tr(P X) X ∈Mn×m(F), is the function

(16) 0n tZ

Z 0m

7−→

ˆ

Z ∈Mm×n(F). Applying Weil’s formula (14) to this function, we get after the change of variablesZ7→Z+¹₂P that

Z

Mn×m(F)

Φh0n X

tX 0m

iψ

Tr(P X) + Tr(C^tXAX) dX (17)

=|2|^12mn|detA|^−12m· |detC|⁻¹²ⁿγ_mⁿ(A, C, ψ) Z

Mm×n(F)

ˆ

Φ0n tZ Z 0m

×ψ

−Tr C⁻¹(Z+¹₂P)A^−1t(Z+¹₂P) dZ.

We remark that in the case m = n = 1 the Weil constant is the 1-dimen- sional Weil constant defined in (1)

(18) γ₁¹(A, C, ψ) =γ(AC, ψ).

For generalmandnwe can describe the Weil constant in terms of the 1-dimen- sional case.

Lemma 2.1. — Let A=^tg1diag(a1, . . . , an)g1 andC=g2diag(c1, . . . , cm)^tg2, whereA∈Sn, C ∈Sm, g1∈GLn(F)andg2∈GLm(F), then

(19) γ_mⁿ(A, C, ψ) = Y

1≤i≤n 1≤j≤m

γ(aicj, ψ).

o

Proof. — We start with the case m = 1. SinceC_c^∞(Vn,1) 'C_c^∞(F)^⊗n, it is enough in the definition of γ₁ⁿ to consider functions Φ of the form Φ(v) = Qn

i=1fi(xi) wherev∈Vn,1 is of the formv= ⁰tX^nX0

,X =^t(x1, . . . , xn)∈Fⁿ and fi ∈ C_c^∞(F). Note that in this case

ˆ

i=1fˆi(xi), therefore if A= diag(a1, . . . , an) the factorization ofγⁿ₁(A,1, ψ) is straightforward from (1) and (2). Indeed

Z

Φ0n X

tX 0 ψ

Tr(^tXAX dX =

Yn i=1

Z

fi(xi)ψ(aix²_i)dxi

=|2|¹²ⁿ Yn i=1

|ai|⁻¹²γ(ai, ψ)

Z fˆi(xi)ψ(−a⁻¹_i x²_i)dxi

=|2|¹²ⁿ|detA|⁻¹²Yⁿ

i=1

γ(ai, ψ) Z

ˆ

Φ0n tZ Z 0

ψ

−Tr(ZA^−1tZ) dZ, therefore in this case indeed we get

(20) γ₁ⁿ(A,1, ψ) =

Yn i=1

γ(ai, ψ).

Let g ∈ GLn(F) be such that A = ^tgag with a = diag(a1, . . . , an), for any A∈Sn. Note that the Fourier transform of the function

0n X

tX 0m

7−→Φ 0n g⁻¹X

tX^tg⁻¹ 0 is the function

0n tZ Z 0m

7−→ |detg| ·

ˆ

and therefore using the changes of variables X 7→ g⁻¹X, Z 7→ Zg⁻¹ and applying (20) for awe get

Z

Φ0n X

tX 0 ψ

Tr(^tXAX)

dX=|detg|⁻¹ Z

Φ 0n g⁻¹X

tX^tg⁻¹ 0 ψ

Tr(^tXaX) dX

=|2|¹²ⁿ|deta|⁻¹²Yⁿ

i=1

γ(ai, ψ) Z

ˆ

Φ0n tg^tZ Zg 0

ψ

−Tr(Za^−1tZ) dZ

=|2|¹²ⁿ|detg|⁻¹· |deta|⁻¹²Yⁿ

i=1

γ(ai, ψ) Z

ˆ

Φ0ntZ Z 0

ψ[−Tr(ZA^−1tZ) dZ.

Since detA = detg²deta we get (20) for any A ∈ Sn. For a general m, if C= diag(c1, . . . , cm) is diagonal then

Tr(C^tXAX) = Xm j=1

Tr(^tXcjAX).

Therefore, as in the casem= 1 the fact thatC_c^∞(Vn,m)'C_c^∞(Vn,1)^⊗mimplies that

(21) γ_mⁿ(A, C, ψ) =

Ym j=1

γ₁ⁿ(cjA,1, ψ)

and (19) then follows from the casem= 1. For anyAandCas in the statement of the lemma, since

Tr(C^tXAX) = Tr(XC^tXA)

we may compute as before, denotingc= diag(c1, . . . , cm) and using the changes of variablesX 7→Xg₂⁻¹andZ 7→g₂⁻¹Z that

Z

Φ0n X

tX 0 ψ

Tr(C^tXAX) dX

=|detg2|⁻ⁿ Z

Φ 0n Xg⁻¹₂

tg⁻¹₂ ^tX 0 ψ

Tr(c^tXAX) dX

=|2|^12mn· |detA|^−12m· |detc|⁻¹²ⁿ·γ_mⁿ(A, c, ψ) Z

ˆ

g2Z 0

ψ[−Tr(c⁻¹ZA^−1tZ) dZ

=|2|^12mn· |detg2|⁻ⁿ· |detA|^−12m· |detc|⁻¹²ⁿ·γ_mⁿ(A, c, ψ) Z

ˆ

Z 0

ψ[−Tr(C⁻¹ZA^−1tZ) dZ.

We then have

γ_mⁿ(A, C, ψ) =γ_mⁿ(A, c, ψ) and the lemma follows from the case when C is diagonal.

The symmetry on the right hand side of (19) implies that (22) γ_mⁿ(A, C, ψ) =γ_n^m(C, A, ψ).

From (1) it is easy to see that the one-dimensional Weil constant satisfies

(23) γ(a, ψ) =γ(−a,ψ).¯

Applying Weil’s formula twice we get that

(24) γ(a, ψ)γ(−a⁻¹, ψ) = 1.

We also recall thatγ satisfies

(25) γ(ac, ψ) =γ(1, ψ)⁻¹γ(a, ψ)γ(c, ψ)[a, c]

fora, c∈F^×. Therefore, forAandC as in Lemma 2.1 (26) γ_mⁿ(A, C, ψ) =γ(1, ψ)^−mnYⁿ

i=1

γ(ai, ψ)^mY^m

j=1

γ(cj, ψ)ⁿ Y

i,j

[ai, cj] .

o

In particular

(27) γ_mⁿ(A,1m, ψ) = Yn i=1

γ(ai, ψ)^m=γ_n^m(1m, A, ψ) and

(28) γ_n^m(1n, C, ψ) = Ym j=1

γ(cj, ψ)ⁿ =γ_mⁿ(C,1n, ψ).

The expressionQ

i,j[ai, cj] is thus only dependent onAandCand we can and will denote

(29) [A, C] =Y

i,j

[ai, cj].

We finish this chapter, with an identity we will need later.

Lemma 2.2. — One has

(30) γ_mⁿ(−A,1m, ψ)γ_mⁿ(A⁻¹, C, ψ) =γ(1,ψ)¯ ^mn[A, C]γ_n^m(−C,1n,ψ).¯ Proof. — From (26), (27) and (28) we get that

γ_mⁿ(−A,1m, ψ)γⁿ_m(A⁻¹, C, ψ) (31)

=γ(1, ψ)^−mnγ_mⁿ(−A,1m, ψ)γ_mⁿ(A⁻¹,1m, ψ)γ_n^m(C,1n, ψ)[A, C].

Using (24) and (27) we get that

γ_mⁿ(−A,1m, ψ)γ_mⁿ(A⁻¹,1m, ψ) = 1.

From (23) and (28) we have γ_n^m(C,1n, ψ) = γ_n^m(−C,1n,ψ) and from (23)¯ and (24) that γ(1, ψ)⁻¹ = γ(1,ψ).¯ Therefore, the right hand side of (31) is now equal toγ(1,ψ)¯ ^mn[A, C]γ_n^m(−C,1n,ψ) and the lemma follows.¯

3. Intermediate orbital integrals

For n, m ≥ 1 and a function Φ ∈ C_c^∞(Sm+n) we define the intermediate orbital integral

ω_mⁿh

Φ, ψ;An

Bm

(32) i

= Z

Mn×m(F)

Φh1n tX 1m

An

Bm

1n X 1m

iθm+n

h1n 2X 1m

idX

= Z

Mn×m(F)

Φh An AnX

tXAn Bm+^tXAnX iψ

2 Tr(X) dX

where An ∈ Sn, Bm ∈ Sm and =ⁿ_m = (δ(i,j),(1,n)) ∈ Mm×n(F). We also define a normalized intermediate integral

e ω^n,ψ_m h

Φ, ψ;An

Bm

i (33)

=γ_mⁿ(−An,1m, ψ)· |detAn|^12m·ωⁿ_mh

Φ, ψ;An

Bm

i.

Proposition 3.1. — Let Cm ∈ Sm, Dn ∈ Sn and Φ ∈ C_c^∞(Sm+n). The function mapping An to

[An, Cm]ψ

Tr(wmC_m⁻¹wmⁿ_mA⁻¹_n ^tn_m) (34)

× Z

Sm

e ω_m^n,ψh

Φ, ψ;An

Bm

i ψ

−Tr(BmwmCmwm) dBm

originally defined forAn ∈Sn, extends to a smooth function of compact support on Sn. The iterated integral

Z

Sn

n Z

Sm

e ω^n,ψ_m h

Φ, ψ;An

Bm

iψh

−Tr(BmwmCmwm)i dBm

o· (35)

×[An, Cm]ψ

Tr(wmC_m⁻¹wmⁿ_mA⁻¹_n ^tn_m) ψ

−Tr(AnwnDnwn) dAn

is therefore convergent. It is equal to (36) |2|^12mn·γ(1,ψ)¯ ^mn·ωe_n^m,ψ¯h

ˇ

Φ,ψ;¯ Cm

Dn

i .

Proof. — First we remark that from the right hand side of (32) it is easily observed that for a fixed An∈Sn the function

Bm7−→ω_mⁿh

Φ, ψ;An

Bm

i

is smooth and of compact support onSm. Therefore, for a fixedCm∈Smthe function Θ defined by

(37) Θ(Dn) =|2|^12mnγ(1,ψ)¯ ^mn·ωe_n^m,ψ¯h

ˇ

Φ,ψ;¯ wmCmwm

wnDnwn

i , is in C_c^∞(Sn). Consider the partial Fourier transform with respect to Bm

(38)

Z

Sm

e ω_mⁿh

Φ, ψ;An

Bm

iψ

Tr(−CmBm) dBm.

Expanding along (32) it becomes after a change of variablesBm7→Bm−^tXAnX γ_mⁿ(−An,1m, ψ)· |detAn|^12m·

(39)

× Z

Φh An AnX

tXAn Bm

iψ[Tr(2ⁿ_mX) + Tr(CmtXAnX)]dX ψ

Tr(−CmBm) dBm

o

and after another change of variablesX7→A⁻¹_n X it becomes γ_mⁿ(−An,1m, ψ)|detAn|^−12m

(40) Z

ΦhAn X

tX Bm

iψ

Tr(2ⁿ_mA⁻¹_n X) + Tr(CmtXA⁻¹_n X) dX ψ

−Tr(BmCm) dBm. Applying Weil’s formula (17) withP = 2ⁿ_mA⁻¹_n , we see that this is equal to

|2|^12mnγⁿ_m(A⁻¹_n , Cm, ψ)γ_mⁿ(−An,1m, ψ)|detCm|⁻¹²ⁿ (41)

× Z

ΦAn Y

tY Bm

ψ

−2 Tr(XY) dY ψ

−Tr(C_m⁻¹(X+ⁿ_mA⁻¹_n )An(^tX+A⁻¹_n ^tn_m)) dX ψ

−Tr(BmCm) dBm.

We now use Lemma 2.2. After expanding (41) it becomes:

|2|^12mn·γ(1,ψ)¯ ^mn·γ^m_n(−Cm,1n,ψ)¯ · |detCm|⁻¹²ⁿ (42)

ψ

−Tr(C_m⁻¹ⁿ_mA⁻¹_n ^tn_m)

[An, Cm]

× Z

ΦAn Y

tY Bm

ψ

−2 Tr(XY)−Tr(C_m⁻¹XAntX)

−2 Tr(C_m⁻¹X^tn_m)

dYdX·ψ

−Tr(BmCm) dBm. We showed so far that for a fixedCm∈Sm, the function Ψ onSn defined by

Ψ(An) = [An, Cm]·ψ

Tr(C_m⁻¹ⁿ_mA⁻¹_n ^m_n)

· (43)

× Z

Sm

e ω_m^n,ψh

Φ, ψ;An

Bm

i ψ

−Tr(BmCm) dBm

is equal to

|2|^12mn·γ(1,ψ)¯ ^mn·γ_n^m(−Cm,1n,ψ)¯ · |detCm|⁻¹²ⁿ

(44) Z

ΦAn Y

tY Bm

ψ

−2 Tr(XY)−Tr(C_m⁻¹XAntX)

−2 Tr(C_m⁻¹X^tn_m)−Tr(CmBm)

dYdXdBm. Changing variablesX 7→CmX we get

Ψ(An) =|2|^21mn·γ(1,ψ)¯ ^mn·γ_n^m(−Cm,1n,ψ)¯ · |detCm|¹²ⁿ (45)

× Z

ΦAn Y

tY Bm

·ψ

−2 Tr(CmXY)−Tr(XAntXCm)

−2 Tr(X^tn_m)−Tr(BmCm)

dYdXdBm.

Next we expand Θ(Dn):

Θ(Dn) =|2|^12mn·γ(1,ψ)¯ ^mn·γ^m_n(−Cm,1n,ψ)¯ · |detCm|¹²ⁿ (46)

× Z

Mm×n(F)

ˆ

Φh wm+n

wmCmwm wmCmwmX

tXwmCmwm wnDnwn+^tXwmCmwmX

wm+n

i ψ

−2 Tr(^m_nX) dX

=|2|^12mnγ(1,ψ)¯ ^mn·γ_n^m(−Cm,1n,ψ)¯ · |detCm|¹²ⁿ

× Z

Mm×n(F)

ˆ

ΦhDn+wntXwmCmwmXwn wntXwmCm

CmwmXwn Cm

i

ψ[−2 Tr(^m_nX)]dX.

After a change of variablesX 7→wmXwn,Θ(Dn) becomes

|2|^12mn·γ(1,ψ)¯ ^mn·γ_n^m(−Cm,1n,ψ)¯ · |detCm|¹²ⁿ· (47)

× Z

Mm×n(F)

ˆ

mX Cm

iψ

−2 Tr(X^tn_m) dX where we use the fact that

Tr(^m_nwmXwn) =X1,n= Tr(X^tn_m).

Expanding further we have

Θ(Dn) =|2|^12mn·γ(1,ψ)¯ ^mn·γ_n^m(−Cm,1n,ψ)¯ · |detCm|¹²ⁿ· (48)

× Z

Φhsn Y

tY Bm

i ψh

−Trsn Y

tY Bm

Dn+^tXCmX ^tXCm

CmX Cm

i dYdsndBmψ

−2 Tr(X^tn_m) dX

=|2|^12mn·γ(1,ψ)¯ ^mn·γ_n^m(−Cm,1n,ψ)¯ · |detCm|¹²ⁿ

× Z

Φhsn Y

tY Bm

iψ

−Tr(sntXCmX)−Tr(snDn) dsn

ψ

−Tr(BmCm)−2 Tr(CmXY)

dYdBmψ

−2 Tr(X^tn_m) dX.

Using Fourier inversion we then see that Θ(−Ab n) =

Z

Sn

Θ(Dn)ψ

Tr(AnDn) dAn

(49)

=|2|^12mnγ(1,ψ)¯ ^mnγ_n^m(−Cm,1n,ψ)|¯ detCm|¹²ⁿ

× Z

ΦhAn Y

tY Bm

i ψh

−Tr(AntXCmX)i ψ

−Tr(BmCm)−2 Tr(CmXY)

dYdBm

ψ[−2 Tr(X^tn_m)]dX.

o

Comparing with (45) we get that

(50) Θ(−Ab n) = Ψ(An)

for all An ∈ Sn. This proves the first part of the proposition. Furthermore, regarding Ψ now as its extension to Sn and writing explicitly the equality Ψ(Db n) = Θ(Dn) we have for allCm∈Sm, Dn∈ Sn

Z

S_n

h Z

S_mωe^n,ψ_m h

Φ, ψ;An

Bm

i ψ

−Tr(BmCm) dBm

i (51)

[An, Cm]ψ

Tr(C_m⁻¹ⁿ_mA⁻¹_n ^tn_m) ψ

−Tr(AnDn) dAn

=|2|^12mn·γ(1,ψ)¯ ^mn·ωe^m,_n ^ψ¯h

ˇ

wnDnwn

i.

Observing that [An, Cm] = [An, wmCmwm] we get the proposition by replacing (Cm, Dn) with (wmCmwm, wnDnwn).

We end this chapter with a reduction formula that we will need for the proof of the main theorem. Let Φ∈C_c^∞(Sm+n) and define onSn× Smthe function (52) Ξ(An, Bm) =ωⁿ_mh

Φ, ψ;An

Bm

i .

Associated with the action ofNn×NmonSn× Sm, we consider the generalized Kloosterman integral

(53) ωh

Ξ, ψ; (An, Bm)i

= Z

Ξ(^tu1Anu1,^tu2Bmu2)θn(u1)θm(u2)du1du2. For a relevant elementx∈ Sm+n of the formx= ^xnxm

, wherexn∈Sn and xm∈ Smare relevant, we have

(54) ω[Φ, ψ;x] =ω

Ξ, ψ,(xn, xm) .

4. Proof of the main theorem

We prove the functional equation by induction on n, the casen= 1 being simply the definition of the Fourier transform. For a fixeda1∈F^×, let

Ψ(An−1) = [a1, An−1]ψ

Tr(a⁻¹₁ A⁻¹_n−1^t) (55)

× Z

e ω^n−1,ψ₁ h

Φ, ψ;An−1

pn

i

ψ[−pna1]dpn, Θ(Dn−1) =|2|^12(n−1)·γ(1,ψ)¯ ⁿ⁻¹·ωe^1,_n−1^ψ¯h

ˇ

Dn−1

i . (56)

Applying Proposition 3.1, with (1, n−1) in the role of (m, n), we get that Ψ extends to Sn−1. In fact Ψ,Θ ∈ C_c^∞(Sn−1) and Θ =

ˇ

diag(a2, . . . , an) be relevant inSn−1. By induction applied to Ψ we have Kψ,n−1ωe^ψ[Ψ, ψ;·]

(a⁽¹⁾) (57)

=|2|^{12(n−1)(n−2)}·γ(1,ψ)¯ ^{12(n−1)(n−2)}·ωe^ψ¯

Θ,ψ;¯ a⁽¹⁾ . By (54) and (19) we have

ω[Θ,ψ;¯ a⁽¹⁾] =|2|^12(n−1)·γ(1,ψ)¯ ⁿ⁻¹·γ_n−1¹ (−a1,1n−1,ψ)¯ · |a1|^12(n−1) (58)

×ω

ω_n−1¹ [

ˇ

=|2|^12(n−1)·γ(1,ψ)¯ ⁿ⁻¹·γ(−a1,ψ)¯ ⁿ⁻¹· |a1|^12(n−1)·ω[

ˇ

|2|^{12(n−1)(n−2)}·γ(1,ψ)¯ ^{12(n−1)(n−2)}·ωe^ψ¯[Θ,ψ;¯ a⁽¹⁾] (59)

=|2|^12n(n−1)·γ(1,ψ)¯ ^12n(n−1)·Γn−1(−a⁽¹⁾,ψ)¯ ·σn−1(a⁽¹⁾)¹²

×γ(−a1,ψ)¯ ⁿ⁻¹· |a1|^12(n−1)·ω[

ˇ

=|2|^12n(n−1)·γ(1,ψ)¯ ^12n(n−1)·ωe^ψ¯[

ˇ

Next we treat the left hand side of (57). We start with the computation of the Kloosterman integral of Ψ atx= diag(p1, . . . , pn−1).Note that

[a1,^tuxu]·ψ

Tr(a⁻¹₁ u⁻¹x^−1tu^−1t)

=ⁿ⁻¹Y

j=1

[a1, pj] ψh 1

a1pn−1

i

for allu∈Nn−1is constant on the orbit ofx. So e

ω^ψ[Ψ, ψ;p1, . . . , pn−1] = Γn−1(−p1, . . . ,−pn−1, ψ) (60)

×σn−1(p1, . . . , pn−1)¹²ⁿ⁻¹Y

j=1

[a1, pj] ψh 1

a1pn−1

i

× ZZ

e ω₁^n−1,ψh

Φ, ψ;^tuxu pn

i

ψ[−pna1]dpnθ(u)du.

Since Φ is smooth and of compact support, it is easy to see from the right hand side of (32), that forφ∈C_c^∞(F^×) the function

(61) (An−1, pn)7−→φ(detAn−1)·ωe₁^n−1,ψh

Φ, ψ;An−1

pn

i

is smooth of compact support on GL(n−1, F)×F. Since the orbit ^tuxu, u ∈ Nn−1 has fixed determinant, the double integral in (60) is absolutely

o

convergent and we may switch order of integration. From (54) we get Γn−1(−p⁽ⁿ⁾, ψ)·σn−1(p1, . . . , pn−1)¹²

(62)

× Z

e ω^n−1,ψ₁ h

Φ, ψ;tuxu pn

i θ(u)du

= Γn−1(−p⁽ⁿ⁾, ψ)·σn−1(p1, . . . , pn−1)¹² ·γ₁ⁿ⁻¹(−x,1, ψ)

×|detx|¹² Z

ωⁿ⁻¹₁ h

Φ, ψ;tuxu pn

i θ(u)du

=eω^ψ[Φ, ψ;p1, . . . , pn].

By switching order of integration in the right hand side of (60) we get that e

ω^ψ[Ψ, ψ;p1, . . . , pn−1] (63)

=ⁿ⁻¹Y

j=1

[a1, pj] ψh 1

a1pn−1

i Z e ω^ψh

Φ, ψ;p1, . . . , pn

i

ψ[−pna1]dpn. Finally, we see that

Kψ,n−1ωe^ψ[Ψ, ψ;·] (a⁽¹⁾) (64)

=Z ⁿ⁻¹Y

j=1

[a1, pj] ψh 1

a1pn−1

i Z eω^ψ

Φ, ψ;p1, . . . , pn

ψ[−pna1]dpn

×ψh

−

n−1X

i=1

pian+1−i+

n−2X

i=1

1 pian−i

iⁿ⁻¹Y

i=2 n−iY

j=1

[ai, pj]

dpn−1· · ·dp1

= Kψ,nωe^ψ[Φ, ψ;·]

(a1, . . . , an).

The theorem now follows from (57), (59) and (64).

Corollary 1.3 is now immediate sinceγ(1, ψ)·γ(1,ψ) = 1.¯ 5. A formula for the smallest orbits Letκ(n) = ¹₂n(n+ 1)−1.

Proposition 5.1. — ForΦ ∈ C_c^∞(Sn), the function φ(a) =ω[Φ, ψ, wna] is a smooth function of compact support on F^×. Furthermore, it satisfies the functional equation

(65) |a|^κ(n)·ω[Φ, ψ, wna] = Z

ωh

ˇ

Φ,ψ;¯ −wn−1a⁻¹ b

i db.

Proof. — We can writeφas follows (66) ω[Φ, ψ, wna] =

Z

Φ(m⁰)ψXⁿ

i=2

xi,n+2−i

⊗

i+j≥n+2dxi,j

where

m⁰_i,j=







0 if i+j≤n, a if i+j=n+ 1, axi,j if i+j≥n+ 2, andxi,j=xj,i ifi+j≥n+ 2. Let

κ1(n) = 1

4n² if n−even,

1

4(n²−1) if n−odd, κ2(n) =κ1(n−1) +n−1.

Thenκ(n) =κ1(n) +κ2(n).After a change of variables (66) can be writen as (67) ω[Φ, ψ, wna] =|a|^−κ1(n)

Z

Φ(m)ψ a⁻¹

Xn i=2

xi,n+2−i

⊗

i+j≥n+2dxi,j

where

mi,j=





0 ifi+j ≤n, a ifi+j =n+ 1, xi,j ifi+j ≥n+ 2,

and xi,j = xj,i if i+j ≥ n+ 2. The smoothness of φ follows from the fact that Φ is smooth. Also Φ(m) = 0 for large enough|a|and for allxi,j as above, since Φ is of compact support. Thereforeφ(a) vanishes when|a|is sufficiently large. Let X be the n×n symmetric matrix with 0 in the (i, j)-th entry wheneveri+j ≤n+ 1 and xi,j in the (i, j)-th entry whenever i+j ≥n+ 2.

LetZ be a similar variable matrix with entries zi,j, thus

(68) m=







a

· x2,n

· ... a x2,n · · · xn,n





, X=







0

· x2,n

· ... 0x2,n· · · xn,n





, Z=







0

· z2,n

· ... 0z2,n· · · zn,n





.

Since Φ is smooth and of compact support, there are integers` < ksuch that Φ(m) = 0 unlessxi,j ∈℘<sup>`</sup>and Φ(m+Z) = Φ(m) wheneverzi,j ∈℘^k for alli, j.

Therefore, φ(a) = X

xi,j∈℘^`/℘^k

Φ(m) Z

zi,j∈℘^k

ψ a⁻¹

Xn i=2

(xi,n+2−i+zi,n+2−i)

⊗

i+j≥n+2dzi,j. This integral factors, for example, through the integral R

℘^kψ(a⁻¹z2,n)dz2,n

which vanishes whenever |a|is small enough. So we get also thatφ(a) = 0 for

|a| sufficiently small. We can write Z

ωh

ˇ

idb=

Z

ˇ

− Xn i=1

yi,n+1−i

⊗

2n>i+j≥n+1dyi,jdb

o

where

p⁰_i,j =











0 if i+j≤n−1,

−a⁻¹ if i+j=n,

−a⁻¹yi,j if 2n > i+j ≥n+ 1, b if i+j= 2n,

andyi,j=yj,iifi+j≥n+ 1. After a change of variables this becomes Z

ωh

ˇ

Φ,ψ;¯ −wn−1a⁻¹ b

i

db=|a|^κ2(n)

Z

ˇ

Xn i=1

yi,n+1−i

i+j≥n+1⊗ dyi,j

where

p⁰⁰_i,j=







0 if i+j≤n−1,

−a⁻¹ if i+j=n, yi,j if i+j≥n+ 1, andyi,j=yj,iifi+j≥n+ 1 which is the same as writing

Z ωh

ˇ

Φ,ψ;¯ −wn−1a⁻¹ b

i db (69)

=|a|^κ2(n)

Z

ˆ

Xn i=1

yi,n+1−i

i+j≤n+1⊗ dyi,j

where

pi,j=







yi,j if i+j≤n+ 1,

−a⁻¹ if i+j=n+ 2, 0 if i+j≥n+ 3,

andyi,j =yj,iifi+j≤n+ 1. Next letY be then×nsymmetric matrix with Yi,j=

yi,j if i+j≤n+ 1, 0 if i+j≥n+ 2.

LetA⁰ be defined byp=Y+A⁰ and letXandZ be matrix variables as defined in (68). Thus

p=







y11 · · · y1n

... −a⁻¹

· y1n −a⁻¹





, Y =







y11· · · y1n

... 0

· y1n 0





, A⁰=







0 · · · 0 ... −a⁻¹

· 0−a⁻¹





.

Using Fourier inversion formula for the functionΦ(Y

ˆ

|a|^−κ2(n) Z

ωh

ˇ

Φ,ψ;¯ −wn−1a⁻¹ b

idb

=

Z Φ(Yb +Z)ψ

−Tr(XZ) dZ ψ

Tr(XA⁰) dXψ

a Xn i=1

yi,n+1−i

⊗

i+j≤n+1dyi,j. LetA=wna, one easily observes now that Tr(ZA) = Tr(Y X) = 0 and there- fore Tr((Y +Z)(A−X)) = Tr(Y A−ZX) =aPn

i=1yi,n+1−i−Tr(XZ), so we get by the Fourier inversion formula applied to

ˆ

|a|^−κ2(n) Z

ωh

ˇ

Φ,ψ;¯ −wn−1a⁻¹ b

idb (70)

=

Z Φ(Yb +Z)ψ

Tr((Y +Z)(A−X))]dYdZ ψ[Tr(XA⁰) dX

= Z

Φ(A−X)ψ

Tr(XA⁰) dX=

Z

Φ(A+X)ψ

Tr(−XA⁰) dX.

Butm=A+X and Tr(−XA⁰) =a⁻¹Pn

i=2xi,n+2−i, so comparing with (67), the right hand side of (70) is equal to |a|^κ1(n)ω[Φ, ψ;wna].

BIBLIOGRAPHY

[1] Flicker (Y.Z.)& Kazhdan (D.A.)– Metaplectic correspondence, Publ.

Math. Inst. Hautes ´Etudes Sci., t.64(1986), pp. 53–110.

[2] Jacquet (H.)– Facteurs de transfert pour les int´egrales de Kloosterman, C. R. Acad. Sci. Paris S´er. I Math., t.336(2003), no. 2, pp. 121–124.

[3] , Smooth transfer of Kloosterman integrals, Duke Math. J., t.120 (2003), no. 1, pp. 121–152.

[4] ,Kloosterman identities over a quadratic extension, Ann. Math. (2), t.160(2004), no. 2, pp. 755–779.

[5] Kazhdan (D.A.)&Patterson (S.J.)–Metaplectic forms, Publ. Math.

Inst. Hautes ´Etudes Sci., t.59(1982), pp. 35–142.

[6] Mao (Z.)–A fundamental lemma for metaplectic correspondence, J. reine angew. Math., t.496(1998), pp. 107–129.

[7] Weil (A.)–Sur certains groupes d’op´erateurs unitaires, Acta Math., t.111 (1964), pp. 143–211.

o