An inequality for local unitary Theta correspondence

ANNALES

DE LA FACULTÉ DES SCIENCES

Mathématiques

Z. GONG, L. GRENIÉ

An inequality for local unitary Theta correspondence

Tome XX, n^o1 (2011), p. 167-202.

<http://afst.cedram.org/item?id=AFST_2011_6_20_1_167_0>

© Université Paul Sabatier, Toulouse, 2011, tous droits réservés.

L’accès aux articles de la revue « Annales de la faculté des sci- ences de Toulouse Mathématiques » (http://afst.cedram.org/), implique l’accord avec les conditions générales d’utilisation (http://afst.cedram.

org/legal/). Toute reproduction en tout ou partie cet article sous quelque forme que ce soit pour tout usage autre que l’utilisation à fin strictement personnelle du copiste est constitutive d’une infraction pénale. Toute copie ou impression de ce fichier doit contenir la présente mention de copyright.

cedram

Article mis en ligne dans le cadre du

Centre de diffusion des revues académiques de mathématiques

pp. 167–202

An inequality for local unitary Theta correspondence

Z. Gong⁽¹⁾ and L. Greni´e⁽²⁾

ABSTRACT.— Given a representationπof a local unitary groupGand another local unitary groupH, either the Theta correspondence provides a representationθH(π) of H orwe setθH(π) = 0. IfGis fixed andH varies in a Witt tower, a natural question is: for whichHisθH(π)= 0 ? Forgiven dimensionmthere are exactly two isometry classes of unitary spaces that we denoteHm^±. Forε∈ {0,1}let us denotem^±ε(π) the minimal m of the same parity of εsuch that θ_H±

m(π) = 0, then we prove that m⁺ε(π) +m⁻ε(π)^{2n+ 2 wheren}is the dimension ofπ.

R^´ESUM´E.— ´Etant donn´ee une repr´esentationπd’un groupe unitaire local Get un autre groupe unitaire localH, on sait que soit la correspondance Theta fournit une repr´esentation θH(π) deH soit on pose θH(π) = 0.

Si on fixeGet on laisseH varier dans une tour de Witt, une question naturelle est : pour quelsHa-t-onθH(π)= 0 ? Pourchaque dimensionm il y a exactement deux classes d’´equivalence d’espaces unitaires que nous d´enotonsHm^±. Pourε∈ {0; 1}, dnotonsm^±ε(π) le plus petitmde la parit´e deεtel queθ_H±

m(π)= 0, alors nous montrons quem⁺ε(π) +m⁻ε(π)

2n+ 2 o`unest la dimension deπ.

(∗) Re¸cu le 08/12/2009, accept´e le 08/09/2010

(1) Lyc´ee annexe `a l’Universit´e Fudan, N.383 Rue Guo Quan, Shanghai, Chine ersanzi@gmail.com

(2) Universit`a degli Studi di Bergamo, viale Marconi 5, 24044 Dalmine (BG), Italy loic.grenie@gmail.com

1. Introduction

The Theta correspondence is a powerful tool for the study of automor- phic and local representations. It has been studied and used in the global and in the local case by various authors, see for instance [Har07], [HKS96], [How], [Kud86], [KR05], [MVW87], [Ral84], [Wal90]. We will restrict our- selves to the local case: we suppose that the base field is ap-adic field with p= 2. The Theta correspondence builds a duality between the representa- tions of two reductive groups forming a dual pair inside a given symplectic (or metaplectic) group. The theory will be explained in greater detail in sec- tion 2. We will be interested in the so-called unitary case where both groups are unitary. To an irreducible representation π of the first group G corre- sponds at most one representation of the second groupH that we denote θ(π) = θ(G, H, π) where θ(π) = 0 if there is no representation of H corre- sponding to π (in the unitary case,θ depends on the choice of a auxiliary character χ, we will thus write θχ instead ofθ in that case). One can fix a representationπof an unitary groupG= U(W) and vary the second group H = U(V), whereW and V are Hermitian spaces and Gand H are their respective unitary groups. One way to vary the space V is to start from a given irreducible space V₀ and to add hyperbolic planes V_1,1. One obtains a so-called Witt tower of spacesV_r=V₀⊕(V_1,1)^rand groupsH_r=H(V_r).

We have (up to isometry) four such towers depending on the parity ofrand on the sign of the Hasse invariant (see below for its definition). We denote them, with a slight notation shift,V_2r+m^± ₀wherem0= 0 or 1, the dimension ofV_2r+m^± ₀ is 2r+m0and±is the sign of the Hasse invariant. It is now well known that if θχ(G, H(V_2r+m^± ₀), π) = 0 then θχ(G, H(V_2r+2+m^± ₀), π) = 0.

We can thus consider, for a givenm0, the two integersm^±_χ(π) which are the minimalm= 2r+m0 such thatθχ(G, H(V_m^±), π)= 0.

We prove here a part of a conjecture of Harris, Kudla and Sweet (see Conjecture 2.7), namely

Theorem 3.10. — Letπ be an irreducible admissible representation of G(W)wheredimW =n. Then

m⁺_χ(π) +m⁻_χ(π)2n+ 2 .

The conjecture (the Conservation Relation, see Conjecture 2.7) asserts that the inequality is in fact an equality.

In some important cases, Theorem 3.10, combined with the results of [HKS96] on local zeta integrals, suffices to prove stronger results. In parti-

cular, it is known, thanks to [HKS96], that

m= inf(m⁺_χ(π), m⁻_χ(π))n.

Whenm = n Harris and Kudla use this inequality and Theorem 3.10 to prove the Dichotomy Conjecture of [HKS96] ([Har07][Theorem 2.1.7]), which determines whetherm=m⁺_χ(π) orm=m⁻_χ(π) in terms of local root numbers.

The (still-conjectural) Conservation Relation, the Dichotomy Conjecture (now proved), and Kudla’s Persistence Principle (Proposition 2.6) go a long way toward providing a complete explicit determination of the local theta correspondence. Resolving the remaining ambiguities will require a better understanding of the poles of local zeta integrals. A key step in the present paper, as in [KR05], is to prove simplicity of these poles for unramified representations. This implies the Conservation Relation whenπis the trivial representation, and a doubling argument that goes back to Kudla and Rallis, together with a cocycle calculation, then implies Theorem 3.10.

The inequality proved in Theorem 3.10 is applied in a global situation in [Har07] to study special values ofL-functions.

While we were writing this manuscript, Harris brought to our attention that a proof in his article [Har07] was incomplete. Since the arguments are related to the ones explained here, we have added that proof as an appendix to this paper.

The authors would like to thank Michael Harris for suggesting this re- search and for helping them throughout the project. The second author would like to thank also the team “Formes Automorphes” from theInstitut de Math´ematiques de Jussieu for their kind invitation while finishing this paper. We would also like to thank the referee who carefully reviewed this paper and made several useful observations which improved substantially this paper.

2. Notations

This section recalls the local Theta correspondence as in [Kud96] and cites some of the results of [HKS96].

We fix once and for all a non archimedean local fieldF of residual char- acteristic different from 2.

The mapping ∆ will always be a diagonal embedding, usually fromGto G×Gexcept in one point where it will be precised.

2.1. Heisenberg group

LetW be a vector space with a symplectic form., .on which the group GL(W) will act on the right – accordingly, iffandgare two endomorphisms ofW, we will denotef◦gthe endomorphism such that (f◦g)(w) =g(f(w)).

We will denote, as usual,

Sp(W) ={g∈GL(W)|∀(x, y)∈W²,xg, yg=x, y} its isometry group.

Definition 2.1. — The Heisenberg group of W if the group H(W) = W F with product

(w1, t1)(w2, t2) = (w1+w2, t1+t2+1

2w1, w2).

The centre ofH(W) is {(0, t)|t∈F} and Sp(W) acts on H(W) via its action onW:

(w, t)^g= (wg, t). We recall

Theorem 2.2 (Stone–von Neumann). — Letψbe a non trivial uni- tary character ofF. There exists, up to isomorphism, one smooth irreducible representation(ρ_ψ, S)ofH(W)such that

ρ_ψ (0, t)

=ψ(t)·id_S .

If we fix such a representation (ρψ, S), then for any g ∈ Sp(g), the representation h −→ ρ^g_ψ(h) = ρψ(h^g) is a representation of H(W) with the same central character, which means that it must be isomorphic toρψ. Hence there is an isomorphismA(g)∈GL(S), unique up to a scalar, such that

∀h∈H, A(g)⁻¹ρψ(h)A(g) =ρ^g_ψ(h). (2.1) The group

Mp(W) ={(g, A(g))| equation (1) holds}

is independent of the choice of ψ and is a central extension of Sp(W) by C^×:

0−→C^×−→Mp(W)−→Sp(W)−→1 .

The group Mp(W) has a natural representation, called the Weil represen- tation,ω_ψ onS given by

ω_ψ : Mp(W) −→ End(S) (g, A(g)) −→ A(g)

2.2. The Schr¨odinger model of the Weil representation

The natural mapping (g, A(g))→A(g) defines a representation of Mp(W) which has several models. We are interested in the so-called Schr¨odinger model.

LetY be a Lagrangian ofW, i.e. a maximal isotropic subspace ofW and W =X⊕Y a complete polarisation ofW. We consider Y as a degenerate symplectic space and seeH(Y) =Y F as a maximal abelian subgroup of H(W). We consider the extensionψY of the characterψ from F to H(Y) defined byψ_Y(y, t) =ψ(t). Let

S_Y = Ind^H(W_H(Y)⁾ψ_Y .

We recall thatS_Y is the space of the functionsf :H(W)−→Csuch that

∀h∈H,∀h1∈H(Y), f(h1h) =ψY(h1)f(h)

and such that there exists a compact open subgroupLofW satisfying

∀h∈H,∀l∈L, f h(l,0)

=f(h).

We fix an isomorphism ofS_Y with the spaceS(X) of Schwartz functions on X by

SY −→ S(X)

f −→ ϕ: X → C

x → ϕ(x) =f(x,0).

The groupH(W) acts onSY by right translation while it acts onϕ∈ S(X)

by

ρ(x+y, t)ϕ)(x0) =ψ

t+x0, y+1 2x, y

ϕ(x0+x)

wherex+y∈W is withx∈X andy∈Y. Then (see [MVW87]) (ρ,S(X)) is a model for the Weil representation.

We specify the operatorωψ as follows. We identify an element w∈W with the row vector (x, y) ∈ X ⊕Y. An element g ∈ Sp(W) will be of

the form g = _{a b}

c d

with a ∈ End(X), b ∈ Hom(X, Y), c ∈ Hom(Y, X) and d∈End(Y). Let P_Y ={g∈Sp(W)|c = 0} be the maximal parabolic subgroup of Sp(W) that stabilises Y and N_Y = {g ∈ P_Y|d = id_Y} its unipotent radical. We have a Levi subgroupM_Y ={g ∈P_Y|b= 0} of P_Y andP_Y =M_YN_Y.

We define the following natural mappings:

m: GL(X) −→ MY

a −→ m(a) = a 0

0 a^∨

n: Her(X, Y) −→ N_Y

b −→ n(b) =

id_X b 0 id_Y

wherea^∨is the inverse of the dual ofaand Her(X, Y) is the subset of those b∈Hom(X, Y) which are Hermitian (in both cases we identify the dual of X⊕Y withY ⊕X using., .).

Proposition 2.3 ([Kud96, Proposition 2.3, p.8). — Let g = a b

c d

∈Sp(g). The operatorr(g)of S(X)defined by r(g)(ϕ)(x) =

Kerc^\Y

ψ 1

2xa, xb − xb, yc+1

2yc, yd

ϕ(xa+yc)dµ_g(y) is proportional toA(g)and moreover is unitary for a unique Haar measure dµg(y)on Kerc^\Y.

2.3. Dual reductive pairs

Definition 2.4. — A dual reductive pair (G, G)inSp(W)is a pair of subgroups ofSp(W)such that bothGandG are reductive and

Cent_Sp(W)(G) =G and Cent_Sp(W)(G) =G .

If (G, G) is a dual reductive pair in Sp(W), we denote G and G the pullbacks of the subgroups in Mp(W). As seen in [MVW87], there exists a natural morphism

j:G ×G −→Mp(W)

such that the restriction ofj to C^××C^× is the product.

We consider the pullback (j^∗(ω_ψ), S) ofω_ψ toG ×G. We note that the central character for bothG andG is the identity:

ωψ(j(z1, z2)) =z1z2·idS .

Let π be an irreducible admissible representation of G such that the central character ofπis the identity. If

N(π) =

λ∈Hom^G(S,π) Kerλ

thenS(π) =S/N(π) is the largest quotient ofS on whichG acts byπ. The action ofG onS commutes with the action ofG so thatG acts onS(π) and thus S(π) is a representation ofG ×G. There exists (see [MVW87]) a smooth representation Θ_ψ(π) ofG, unique up to isomorphism, such that

S(π)π⊗Θψ(π). The principal result of the theory is the following

Theorem 2.5 (Howe duality principle). — Let F be a non archi- medean local field with residual characteristic different from 2 and let π be an irreducible admissible representation ofG.

i) IfΘ_ψ(π)= 0, then it is an admissible representation of G of finite length.

ii) If Θψ(π)= 0, there exists a unique G-submodule Θ⁰_ψ(π) such that the quotient

θ_ψ(π) = Θ_ψ(π)/Θ⁰_ψ(π) is irreducible. IfΘψ(π) = 0, we letθψ(π) = 0.

iii) If two irreducible admissible representationsπ1andπ2 ofG are such thatθψ(π1)θψ(π2)= 0 thenπ1π2.

2.4. The unitary case

LetE/F be a quadratic extension and-_E/F the corresponding quadratic character ofF^×.

We fix a quadratic spaceW of dimensionnwith skew-Hermitian form ., .:W×W −→E

(linear in the second argument). We will denoteG(W) its isometry group.

LetV be a quadratic space of dimension mwith Hermitian form (.|.) :V ×V −→E

(linear in the second argument). We will denote

G(V) ={g∈GL(V)|∀v, w∈V,(gv|gw) = (v|w)}

the isometry group of V. The space V will vary in the remaining of the paper.

LetW= R_E/F(V ⊗EW) with symplectic form ., .: W⊗W −→ F

(v₁⊗w₁, v₂⊗w₂) −→ v₁⊗w₁, v₂⊗w₂

= 1

2Tr_E/F((v₁, v₂)w₁, w₂). The pair (G(V), G(W)) is a dual reductive pair in Sp(W). We have a natural inclusion

ı:G(V)×G(W) −→ Sp(W)

(g, h) −→ ı(g, h) =g⊗h.

For any pair of charactersχ= (χ_m, χ_n) ofE^× such that χ_n|F^× =-_E/Fⁿ , χ_m|F^× =-_E/F^m ,

one can define, see [Kud94, Proposition 4.8, p.396], a homomorphism

˜ıχ:G(V)×G(W)−→Mp(W)

liftingı (the homomorphism ˜ıχ does depend on χ). Since the context will usually make clear which of χm and χn is considered, we will often use χ instead ofχmor χn. Moreover we defineıV,χ (resp.ıW,χ) the restriction of ıχ toG(V)×1 (resp. 1×G(W)).

We will denoteω_ψthe Weil representation of Mp(W) andω_χ its pullback through ˜ı_χ. As before, if π is an irreducible admissible representation of G(V), we get a representation Θ_χ(π, V) ofG(W) such that

S(π)π⊗Θ_χ(π, V)

and if Θχ(π, V)= 0, we say thatπ appears in the local Theta correspon- dence for the pair (G(V), G(W)). This condition depends on χm but not onχn. As above we defineθχ(π, V) to be the unique irreducible quotient of Θχ(π, V) (or 0 if Θχ(π, V) = 0).

Witt towers.For a fixed dimensionm, there are two equivalence classes of Hermitian spaces of dimensionmoverE. These two classes are distinguished by their Hasse invariant

-(V) =-_E/F

(−1)^m(m2−1)detV .

We thus get two families of spaces V_m^± where the sign is the sign of the Hasse invariant. As Hermitian spaces we have V_m+2^± V_m^± ⊕V1,1, where V1,1 is an hyperbolic plane and the direct sum is orthogonal. We thus get four so-called Witt towers

V_2r⁺=V₀⁺⊕(V_1,1)^r, V_2r+2⁻ =V₂⁻⊕(V_1,1)^r, V_2r+1⁺ =V₁⁺⊕(V1,1)^r, V_2r+1⁻ =V₁⁻⊕(V1,1)^r

whereV₀⁺ is the null vector space,V₂⁻ is an anisotropic 2-dimensional Her- mitian space andV₁^± are one dimensional anisotropic Hermitian spaces. In each case the integer r is the Witt index of the corresponding Hermitian space¹.

We have

Proposition 2.6 [HKS96],[Kud96]. — Consider a Witt tower{V_m}with -=±.

i) (Persistence) Ifθ_χ(π, V_m)= 0 thenθ_χ(π, V_m+2 )= 0.

ii) (Stable range) We haveθ_χ(π, V_m)= 0 if the Weil index r₀ of V_m is such thatr₀n.

We fixm0∈ {0,1} and a characterχ ofE^× such thatχ_|F× =-_E/F^m0 and we consider the two towers V_m^± with mof the parity of m₀ (ifm₀ = 0 we disregardV₀⁻ which does not exist). Letm^±_χ(π) be the smallestmsuch that

θχ(π, V_m^±)= 0. Based on several examples, we have

Conjecture 2.7 (Conservation relation,[HKS96, Speculations 7.5 and 7.6], [KR05, Conjecture 3.6]). — Ifπ is an irreducible admissible rep- resentation ofG(W), then

m⁺_χ(π) +m⁻_χ(π) = 2n+ 2 .

(1) We recall that the Witt index of a quadratic space is the dimension of a maximal totally isotropic subspace

2.5. Degenerate principal series

Let W₊ and W₋ be two copies of W with respectively the same form as W and its opposite. We keep the pair of characters χ = (χ_m, χ_n). We fix for the space W₊⊕W₋ the complete polarisation X⊕Y where X = {(w,−w)|w∈W} andY ={(w, w)|w∈W}= ∆(W) (recall that ∆ is the diagonal embedding ofW inW+⊕W₋). We let then

W+ = R_E/F(V ⊗EW₊) W− = R_E/F(V ⊗EW₋) X = R_E/F(V ⊗EX) Y = R_E/F(V ⊗EY).

and we consider the representation ωV,W+⊕W−,χ ofG(V)×G(W+⊕W₋) induced by the Weil representation ofW+⊕W− onS=S(X) S(Vⁿ). Let R_n(V, χ) be the maximal quotient ofSon whichG(V) acts by the character χ_m. The spaceR_n(V, χ) can be seen as a representation ofG(W)×G(W) via the natural embedding

i:G(W)×G(W) =G(W+)×G(W₋)2→G(W+⊕W₋).

From now on, we will denoteG=Gn =G(W) and ˜G= ˜Gn=G(W+⊕W₋) so thati:G×G 2→G.˜

We then have

Proposition 2.8 ([HKS96], Proposition 3.1 and discussion before). — If πbe an irreducible admissible representation ofG(W), then

Θ_χ(π, V)= 0 ⇐⇒ Hom_G×G(R_n(V, χ), π⊗(χ_m·π^∨))= 0 . Let PY be the parabolic subgroup of ˜G stabilising Y. We will denote MY its maximal Levi subgroup and NY its unipotent radical. As for the symplectic case, M_Y andN_Y are parametrised respectively by GL(X) and Her(X, Y).

Fors∈Candχa character ofE^×, let In(s, χ) = Ind^G_P^˜_Yχ|.|^s

be the degenerate principal series (the induction is unitary and the elements ofI_n(s, χ) are locally constant functions Φ(g, s)).

We can identifyRn(V, χ) as a subspace of someIn(s, χ) by sending an element ϕ ∈ S(X) to the function g −→ ωχ(g)ϕ(0) – (we recall that we denoteωχ =ωψ◦˜ıV,χ). The spacesRn(V_m^±, χ) allows us to decompose the variousIn(s, χ) as explained by the following proposition.

Proposition 2.9 ([KS97, Theorem 1.2, p.257]). — Let V_m^± be an m-dimensional unitary space and Hasse invariant ±. Let s₀ = ^m−n₂ and χ a character ofE^× such that χ|F^×=-_E/F^m .

i) Ifmn, i.e. ifs₀ 0, then the modules R_n(V_m^±, χ)are irreducible andR_n(V_m⁺, χ)⊕R_n(V_m⁻, χ)is the maximal completely reducible sub- module ofI_n(s₀, χ).

ii) Ifm=n, i.e. if s₀= 0, thenI_n(0, χ) =R_n(V_n⁺, χ)⊕R_n(V_n⁻, χ).

iii) If n < m < 2n, i.e. if 0 < s0 < ⁿ₂, then In(s0, χ) =Rn(V_m⁺, χ) + R_n(V_m⁻, χ)andR_n(V_m⁺, χ)∩R_n(V_m⁻, χ)is the unique irreducible sub- module ofI_n(s₀, χ).

iv) Ifm= 2n, i.e. if s0= ⁿ₂, thenIn(s0, χ) =Rn(V_2n⁺, χ),Rn(V_2n⁻, χ)is of codimension1and is the unique irreducible submodule ofI_n(s₀, χ).

v) Ifm >2n, i.e. ifs0> ⁿ₂, thenIn(s0, χ) =Rn(V_m^±, χ) is irreducible.

In all other casesI_n(s, χ)is irreducible.

To refine the aforementioned decompositions we begin with the Bruhat decomposition of ˜G:

G˜ =

n

j=0

P_Yω_jP_Y, withω_j=





I_n−j 0 0 0

0 0 0 I_j

0 0 I_n−j 0

0 −Ij 0 0





and let us introduce, as in [Kud96, p.19] and [Rao93] the mapping x: G˜ −→ E^×/N_E/FE^×

p1ω_j⁻¹p2 −→ det(p1p2|Y)mod N_E/FE^× Wheneverχ|F^× =1we can introduce the characterχG˜ of ˜G

χG˜(g) =χ(x(g)). We extend the definition ofR_n as follows:

Rn(V₀⁺, χ) =Rn(0, χ) =C·χG˜

and Rn(V₀⁺, χ) is a submodule of dimension 1 o f In(−ⁿ₂, χ) (we are, at least formally, in the case i) of Proposition 2.9). As a last step, we define the intertwining operators

Mn(s, χ) :In(s, χ)−→In(−s, χ)

by the integral Mn(s, χ)(Φ) =

NY

Φ(wnug, s)du=

Her(X,Y)

Φ(wnn(b)g, s)db , which is convergent for Res > ⁿ₂ and by meromorphic continuation for s∈C. The Haar measure dbis chosen self-dual with respect to the Fourier transform

φ(y) =ˆ

φ(b)ψ(Tr(by))db .

We normaliseMn(s, χ) using

a(s, χ) =

n−1

j=0

L_F

2s+j−(n−1), χ-_E/F^j

and thenM_n^∗(s, χ) = _a(s,χ)¹ Mn(s, χ) is holomorphic and non zero (see [KS97, Proposition 3.2]).

Proposition 2.10 [KS97]. — Let V_m^± be the m-dimensional unitary space of dimensionmand Hasse invariant±. Lets₀= ^m−n₂ andχa char- acter of E^× such that χ|F^× =-_E/F^m .

i) If m = 0, i.e. if s₀ =−ⁿ₂, then Ker(M_n^∗(−ⁿ₂, χ)) =R_n(V₀⁺, χ) and Im(M_n^∗(−ⁿ₂, χ)) =Rn(V_2n⁻, χ).

ii) If1m < n, i.e. if−ⁿ₂ < s0<0, thenKer(M_n^∗(s0, χ)) =Rn(V_m⁺, χ)

⊕Rn(V_m⁻, χ)andIm(M_n^∗(s0, χ)) =Rn(V_2n⁺_−m, χ)∩Rn(V_2n⁻_−m, χ).

iii) Ifnm <2n, i.e. if0s0< ⁿ₂, thenKer(M_n^∗(s0, χ)) =Rn(V_m⁺, χ)∩

R_n(V_m⁻, χ), M_n^∗(s₀, χ)(R_n(V_m^±, χ)) = R_n(V_2n^±_−m, χ) thus we have Im(M_n^∗(s0, χ)) =Rn(V_2n−m⁺ , χ)⊕Rn(V_2n−m⁻ , χ).

iv) If m = 2n, i.e. if s₀ = ⁿ₂, then Ker(M_n^∗(ⁿ₂, χ)) = R_n(V_2n⁻, χ) and Im(M_n^∗(ⁿ₂, χ)) =M_n^∗(ⁿ₂, χ)(Rn(V_2n⁺), χ) =Rn(V₀⁺, χ).

2.6. Local Zeta integral

The last element we will use is the local Zeta integral of a representation.

We fixπan irreducible admissible representation ofG(W).

Definition 2.11. — A matrix coefficient of π is a linear combination of functions of the form

φ(g) =π(g)ξ, ξ^∨

whereξ andξ^∨ are vectors of the space of πandπ^∨ respectively.

Moreover ifξ_◦ andξ_◦^∨are preassigned spherical vectors ofπandπ^∨, we let

φ^◦(g) =π(g)ξ_◦, ξ_◦^∨.

We parametrise the space of matrix coefficients with the space ofπ⊗π^∨ through the obvious projection. If s ∈ C with Res large enough, ξ ∈ π, ξ^∨∈π^∨, Φ∈In(s, χ), let

Z(s, χ, π, ξ⊗ξ^∨,Φ) =

G

π(g)ξ, ξ^∨Φ(i(g, I_n), s)dg

and extend it linearly to the space of matrix coefficients of π. We fix a maximal compact subgroupK of ˜G.

Definition 2.12. — Astandard section Φis a mapping from C to the set of functions fromG˜ toCsuch that∀s∈C,Φ(g, s) = Φ(s)(g)∈In(s, χ) and, moreover, Φ(s)|K is independent of s.

It is rather obvious that any element Φ(g, s)∈I_n(s, χ) can be inserted in a (unique) standard section. The Zeta integral above defines, for Res sufficiently large, an intertwining operator

Z(s, χ, π)∈HomG×G

In(s, χ), π⊗(χ·π^∨) .

If Φ is a standard section, this operator can be meromorphically extended for alls∈Cto an operator

Z^∗(s, χ, π)∈Hom_G×G

I_n(s, χ), π⊗(χ·π^∨) .

3. Our results

3.1. Decomposition of the degenerate principal series

Let Ω(W+⊕W₋) be the Grassmannian of the Lagrangians ofW+⊕W₋. We can identify

PY\G(W+⊕W₋)Ω(W+⊕W₋)

using the map P_Y ·g −→Y g. There is a right action ofi(G(W)×G(W)) on Ω(W₊⊕W₋) which orbits are parametrised by the elements of the de- composition

G(W₊⊕W₋) =

r0

r=0

P_Yδ_ri(G(W)×G(W))

wherer0 is the Witt index ofW. The aforementioned orbits are of the form Ωr=P_Y\PYδri(G(W)×G(W)).

The orbit Ωris made of the LagrangiansZsuch that dimZ∩W+= dimZ∩ W₋=r. The only open orbit is that ofY, which is Ω0, while the only closed one is that of Ωr0 and the closure of the orbit Ωris

Ω_r=

jr

Ω_j . We consider the filtration

In(s, χ) =I_n^(r0)(s, χ)⊃ · · · ⊃I_n⁽¹⁾(s, χ)⊃I_n⁽⁰⁾(s, χ), where

I_n^(r)(s, χ) ={Φ∈In(s, χ)|Φ|_Ωr+1= 0}. Let

Q^(r)_n (s, χ) =I_n^(r)(s, χ)/I_n^(r−1)(s, χ)

be the successive quotients of the filtration. AllIn^(r)(s, χ) andQ^(r)n (s, χ) are G×G-stable.

LetTW be the Witt tower containingW. For anyW∈TW of dimension n =n−2rn, letGn =G(W). We identifyW with a subspace of W isomorphic toW. There is a Witt decomposition

W =U⊕W⊕U

whereU andU are dual isotropic subspaces of dimensionr. LetP_rbe the parabolic subgroup ofGstabilisingU. The Levi subgroup ofPris isomorphic to GL(U)×Gn so that, if we denote Mr its Levi component and Nr its unipotent radical, we have isomorphisms

Mr GL(U)×Gn (3.2)

P_r (GL(U)×G_n)N_r.

Note in particular forr= 0 thatU =U ={0},W=W andP0=Gn=G.

Let

St_r=i⁻¹(δ_r⁻¹P_Yδ_r∩i(G×G)) be the stabiliser of PYδr ini⁻¹(P_Y)\G×G.

Lemma 3.1. — For a convenient choice ofδr(specified in Equation (3.3) below), we have

Str= (GL(U)×GL(U)×∆(Gn))(Nr×Nr)⊂Pr×Pr . Moreover

Q^(r)_n (s, χ)Ind^G_Pr^×_×^G_Pr

χ|.|^s+r2 ⊗χ|.|^s+r2 ⊗

S(Gn)·(1⊗χ) where the action of G_n ×G_n on the space S(G_n)·(1⊗χ) is given by (g₁, g₂)ϕ(g) =χ(detg₂)ϕ(g₂⁻¹gg₁).

Proof. — We letG=Gn. Recall the Witt decomposition

W =U⊕W⊕U and consider the Lagrangian

Z=U× {0} ⊕∆(W)⊕ {0} ×U

in W₊⊕W₋. Since the action of ˜G on Ω(W₊⊕W₋) is transitive, there existsδ_r∈G˜ such thatZ=Y δ_r. Since any linear map fromY toZ can be extended to an element of ˜G, we can furthermore require that

∀v∈U, δ_r|∆(U)(v, v) = (0, vd)∈ {0} ×U

δr|∆(W) = id_∆(W) (3.3)

∀u∈U, δ_r|∆(U)(u, u) = (u,0)∈U× {0}

where d:U −→U is any isomorphism. Note in particular thatδ₀ = id_G. Following [Kud96, Proof of Proposition 2.1, p.68], we find that there is a bijection between the orbit Ωr ofZ and the set

{(Z+, Z₋, λ)}

whereZ_± is an isotropic subspace ofW_± of dimensionr and λ:Z₊^⊥/Z₊−→Z₋^⊥/Z₋

is an isometry². The action of (g₊, g₋)∈G×Gon this set is given by (g+, g₋)(Z+, Z₋, λ) = (Z+g+, Z₋g₋, g₊⁻¹◦λ◦g₋).

The stabiliser of (Z₊, Z₋, λ) is

{(g+, g₋)∈G×G|g± stabilisesZ_± andg₊⁻¹◦λ◦g−=λ}.

In our situation and with our choice ofδr, we haveZ+=Z₋=U,Z₊^⊥/Z₊= W and λ= idW. Hence, denoting pr_W the projection on W parallel to U⊕U,

Str =

(g+, g₋)∈Pr×Prg+|W+U◦prW =g₋|W+U◦prW

= (GL(U)×GL(U)×∆(G))(N_r×N_r). For further reference, an element ofPrhas the form



a b c 0 e b^∗ 0 0 a^∨





whereb^∗depends onb,aandeand wherecsatisfies an equation depending ona,bande. We thus have

g_±=



a_± b_± c_± 0 e_± b^∗_± 0 0 a^∨_±



 (3.4)

and the conditiong₊|W+U◦pr_W =g₋|W+U◦pr_W is simplye₊=e₋. The description of the stabiliser allows us to describe the induced repre- sentations. If ˜g∈Str, thenp(˜g) =δri(˜g)δ⁻¹_r =n·m(ar(˜g))∈PY. Letξs,rbe the character of Str defined by ξs,r(˜g) = χ(ar(˜g))|detar(˜g)|^s+r2. Consider the morphism ofG×G-modules

Q^(r)n (s, χ) −→ Ind^G×G_Str (ξ_s,r) f −→ φ_f(g₁, g₂) =

N_rf(δ_rn(u)i(g₁, g₂))du

where f ∈In^(r)(s, χ) is a representative off. This morphism is an isomor- phism (see [HKS96, Equation (4.9), p.963]). Let ˜g = (g+, g₋) be an ele- ment of Strdecomposed as in (3.4). Then det(ar(˜g)) = deta+deta₋dete+

(where we recall thate+=e₋). Sincee+∈G,|dete+|= 1 hence Q^(r)_n (s, χ) Ind^G×G_Str (χ|.|^s+r2 ⊗χ|.|^s+r2 ⊗χ)

Ind^G×G_Pr×Pr

Ind^P_St^r_r^×Pr(χ|.|^s+r2 ⊗χ|.|^s+r2 ⊗χ)

.

(2) in [Kud96] it is an anti-isometry but, sinceW₋has the opposite form ofW+, here λis an isometry.

The induction from St_r toP_r×P_r is an induction from ∆(G) to G×G. Moreover, iff ∈Ind^G_∆(G^×G)χthenf(h1, h2) =χ(h2)f(h₂⁻¹h1,1). Hence

Ind^G_∆(G^×G₎χ S(G)·(1⊗χ) where the action ofG×G onS(G)·(1⊗χ) is given by

ρ(g1, g2)ϕ(g) =χ(detg2)ϕ(g₂⁻¹gg1). Hence

Ind^P_St^r_r^×Pr(χ|.|^s+r2 ⊗χ|.|^s+r2 ⊗χ) χ|.|^s+r2 ⊗χ|.|^s+r2 ⊗Ind^G_∆(G^×G)χ χ|.|^s+r2 ⊗χ|.|^s+r2 ⊗

S(G)·(1⊗χ) . The result follows.

3.2. Simplicity of poles

We prove in our case the result of [KR05, section 5]. We follow the same method. We denote χ0 the trivial character ofF^×.

Proposition 3.2. — Let z_s ∈ H(G // K)⊗C[q^s, q^−s] be the element defined by

z_s=

r0

i=1

(1−q^−s−12t_i)(1−q^−s−12t_i⁻¹).

where we recall thatH(G // K)C[t1, . . . , tn, t₁⁻¹, . . . , t_n⁻¹]^WG. For an un- ramified representation π of G, let π(zs)be the scalar by which z_s acts on the unramified vector in π. Then for all matrix coefficients φ of π and all standard sectionsΦ(s)∈In(s), the function

π(zs)·Z(s, χ0, π, φ,Φ) is an entire function of s.

Proof of Proposition 3.2. — We divide the proof into four steps.

3.2.1.Step 1

By linearity of Z, we can limit ourselves to the case where φ is of the form

φ(g) =π(g)π(g1)ξ_◦, π^∨(g2)ξ_◦^∨

whereξ_◦ andξ_◦^∨are spherical vectors inπandπ^∨andg₁,g₂∈G. Then we have

Z(s, χ₀, π, φ,Φ) =

G

π(g)π(g₁)ξ_◦, π^∨(g₂)ξ^∨_◦Φ_s(i(g, I_n))dg (3.5)

=

G

π(g)ξ_◦, ξ^∨_◦Φ_s(i(g₂gg₁⁻¹, I_n))dg

= |detg₂|^s+r0−12

G

φ^◦(g)Φ_s(i(g, I_n)i(g₁⁻¹, g₂⁻¹))dg since|detg2|= 1 andφ^◦ is bi-K invariant, for allk1,k2∈K,

=

G

φ^◦(g)Φs(i(k₂⁻¹gk1, In)i(g₁⁻¹, g₂⁻¹))dg

=

G

φ^◦(g)Φs(i(g, In)i(k1, k2)i(g₁⁻¹, g₂⁻¹))dg and thus

=

G

φ^◦(g)Ψ_s(i(g, I_n))dg where, for anyh∈H =G_2n,

Ψ_s(h) :=

K×KΦ_s(hi(k₁, k₂)i(g₁⁻¹, g₂⁻¹))dk₁dk₂. (3.6) Note that Ψs is K×K-invariant section of In(s) which is not necessarily standard.

3.2.2.Step 2

We consider the algebra

A=C[X, X⁻¹]⊗ H(G // K)C[X, X⁻¹]⊗C[t1, . . . , tn, t₁⁻¹, . . . , t_n⁻¹]^WG , where H(G // K) is the K-spherical Hecke algebra of G and the element z∈ Adefined as:

z=

r0

i=1

(1−Xq⁻¹²t_i)(1−Xq⁻¹²t_i⁻¹).

We letG×Gact onI_n(s) throughi. We extend this action toH(G // K)× H(G // K) and we let any φ ∈ H(G // K) act as (φ,1) ∈ H(G // K)× H(G // K). We define the action of A on the space In(s)^K×1 of K×1- fixed vectors of In(s) by the aforementioned action of H(G // K) and by X·ϕ=q^−sϕfor anyϕ∈In(S). Note that action of 1×Gcommutes with the action ofA.