WHITTAKER SUPPORTS FOR REPRESENTATIONS OF REDUCTIVE GROUPS

(1)

ANNALES DE

L’INSTITUT FOURIER

LesAnnales de l’institut Fouriersont membres du

Raul Gomez, Dmitry Gourevitch & Siddhartha Sahi Whittaker supports for representations of reductive groups Tome 71, n^o1 (2021), p. 239-286.

<http://aif.centre-mersenne.org/item/AIF_2021__71_1_239_0>

Cet article est mis à disposition selon les termes de la licence Creative Commons attribution – pas de modification 3.0 France.

http://creativecommons.org/licenses/by-nd/3.0/fr/

(2)

WHITTAKER SUPPORTS FOR REPRESENTATIONS OF REDUCTIVE GROUPS

by Raul GOMEZ,

Dmitry GOUREVITCH & Siddhartha SAHI (*)

Abstract. —LetFbe eitherRor a finite extension ofQp, and letGbe a finite central extension of the group ofF-points of a reductive group defined overF. Also letπbe a smooth representation ofG(Fréchet of moderate growth ifF =R). For each nilpotent orbitOwe consider a certain Whittaker quotientπOofπ. We define the Whittaker support WS(π) to be the set of maximalOamong those for which πO6= 0.

In this paper we prove that allO ∈WS(π) are quasi-admissible nilpotent orbits, generalizing results of Mœglin and Jiang–Liu–Savin. IfF isp-adic andπis quasi- cuspidal then we show that allO ∈WS(π) areF-distinguished, i.e. do not intersect the Lie algebra of any proper Levi subgroup ofGdefined overF.

We also give an adaptation of our argument to automorphic representations, generalizing results of Ginzburg–Rallis–Soudry, Shen, and Cai, and confirming some conjectures of Ginzburg.

Our methods are a synergy of the methods of the above-mentioned authors, and of our own earlier work.

Résumé. —Soit F le corps R, ou une extension finie de Qp, et soit G une extension centrale finie du groupe des F-points d’un groupe réductif fini surF. Soit aussiπ une representation lisse deG(Fréchet à croissance modérée dans le casF=R). Pour chaque orbite nilpotenteO, on considère un certain quotient de WhittakerπOdeπ. Nous définissons le support de Whittaker WS(π) comme étant l’ensemble desOmaximales parmi celles pour lesquellesπO6= 0.

Dans cet article, nous prouvons que toutes les O ∈ WS(π) sont des orbites nilpotentes quasi-admissibles, généralisant les résultats de Mœglin et de Jiang–Liu–

Savin. SiF est p-adique etπest quasi-cuspidale, alors nous montrons que toutes lesO ∈WS(π) sont F-distinguées, c’est-à-dire qu’elles ne rencontrent l’algèbre de Lie d’aucun sous-groupe de Levi de G défini surF.

Keywords:Fourier coefficient, wave-front set, oscillator representation, Heisenberg group, metaplectic group, admissible orbit, distinguished orbit, cuspidal representation, automorphic form.

2020Mathematics Subject Classification:20G05, 20G20, 20G25, 20G30, 20G35, 22E27, 22E46, 22E50, 22E55, 17B08.

(*) D.G. was partially supported by ERC StG grant 637912, and ISF grant 249/17. S.S.

was partially supported by Simons Foundation grant 509766.

(3)

Nous donnons aussi une adaptation de nos méthodes aux représentations auto- morphes, généralisant ainsi des résultats de Ginzburg–Rallis–Soudry, Shen et Cai, et confirmant certaines conjectures de Ginzburg.

Nos méthodes combinent celles des auteurs susmentionnés et de nos propres travaux antérieurs.

1. Introduction

The study of Whittaker and degenerate Whittaker models for representations of reductive groups over local fields evolved in connection with the theory of automorphic forms (via their Fourier coefficients), and has found important applications in both areas. See for example [15, 16, 19, 29, 32, 35, 49, 53, 59, 61].

LetF be eitherRor a finite extension ofQp, and letGbe a finite central extension of the group ofF-points of a connected reductive algebraic group defined overF. Let Rep^∞(G) denote the category of smooth representations of G (see Section 2.3 below). Let g denote the Lie algebra of G and g^∗ denote its dual space. To every coadjoint nilpotent orbitO ⊂g^∗ and every π ∈ Rep^∞(G) we associate a certain generalized Whittaker quotient π_O (see Section 2.5 below). Let WO(π) denote the set of all nilpotent orbitsO withπ_O 6= 0 and WS(π) the set of maximal orbits in WO(π) with respect to the closure ordering.

We recall the notion of admissible nilpotent orbit. It has to do with splitting of a certain metapletic double cover of the centralizerGϕ for any ϕin the orbit. We also define a weaker notion of a quasi-admissible orbit.

See Section 2.6 below for both notions.

For split p-adic groups, admissibility is also related to the notion of a special nilpotent orbit in the sense of Lusztig (see Section 6.2 below). In particular, forp-adic classical groups the two notions are equivalent ([44]).

Theorem 1.1 (Section 6). — Let π ∈Rep^∞(G)and let O ∈ WS(π).

ThenOis a quasi-admissible orbit.

Note that in the Archimedean case, Ois not always admissible, e.g. for minimal representations ofU(2,1), see Section 6.2 below. The notions of admissible and quasi-admissible also differ for the split real forms ofE₇and E8, though we do not know whether the non-admissible quasi-admissible orbits appear in Whittaker supports of representations. For the symplectic and orthogonal groups the two notions are equivalent.

(4)

Proposition 1.2 (Section 6.2). — LetG be eitherSp_2n(F), or O(V) orSO(V)(for a quadratic spaceV overF), and letO ⊂g^∗ be a nilpotent orbit. Then the following are equivalent.

(a) Ois admissible (b) Ois quasi-admissible

(c) Ois special

We deduce this proposition from [44, 50]. It is possible that the notions of admissible and quasi-admissible are equivalent for allGin the case when F is non-Archimedean. In this case, and under the additional assumption thatGis classical, it is shown in [31, 41] that allO ∈WS(π) are admissible, for anyπ∈Rep^∞(G). For exceptionalG, a slightly weaker result is shown in [31].

For p-adic F it is conjectured that if G is algebraic then all orbits in WS(π) are special. For classicalGthis follows from [31, 41]. ForG=G₂(F) this follows from [31, 37]. For F = R the analogous statement does not hold. Namely, [58] constructs a small unitary irreducible representationπ of G2(R). We show in Section 6.2 that WS(π) is also small, i.e. consists of the minimal orbit forG2. This orbit is non-special but admissible. For classical algebraic groups over all local fields, all special orbits are quasi- admissible. It is possible that this holds for all groups.

It is quite probable that if G is algebraic and π is admissible and has integral infinitesimal character then allO ∈WS(π) are special, cf. [4, The- orem D] and [5, Theorem 1.1].

We also prove that for quasi-cuspidalπ∈Rep^∞(G), the orbits in WS(π) areF-distinguished. Here,F is non-Archimedean, and πis quasi-cuspidal if all its Jacquet reductions vanish (i.e.r_Pπ= 0 for any parabolic subgroup P⊂G) and a nilpotent orbitO ⊂g^∗isF-distinguished if the corresponding orbit ingdoes not intersect the Lie algebra of a Levi subgroup of any proper parabolic subgroupP⊂Gdefined overF.

Theorem 1.3 (Section 5.2). — Let F be non-Archimedean and π ∈ Rep^∞(G)be a quasi-cuspidal representation. Then allO ∈WS(π)areF- distinguished.

For classical G, it was shown in [41] that all the orbits in the Whit- taker support of all tempered admissible (finitely generated) π are F- distinguished. For a similar result in the caseF =Rsee [26].

One of our basic tools is Lemma 4.8 below, that follows from the Stone–

von-Neumann theorem. An analogous lemma first appeared in the non- Archimedean case in [17, Lemma 2.2], and is sometimes referred to as the

(5)

“root exchange” Lemma. We also use [6, Lemma 5.10] (see Lemma 4.5 below) and its Archimedean analog Proposition 3.1 that we prove in Section 3 below using some properties of modules over algebras of Schwartz functions established in [9]. Two more central tools we use are the deformation tech- nique of [20] and the notion of quasi-Whittaker quotients introduced in this paper, see Section 4.1 below. For our strategy of proof see 1.4 below.

1.1. Additional results

Let γ = (f, h, e) be an sl2-triple. Let Gγ be the centralizer of γ in G, and Ge_γ be its metaplectic cover (see Section 2.6 below). Let ϕ ∈ g^∗ be given by the Killing form pairing withf. ThenGeγ acts on the generalized Whittaker quotientπϕ=πγ (see Section 2.6 below).

We denote by M_γ the subgroup of G_γ generated by the unipotent elements. LetMg_γ denote the preimage ofM_γ under the projectionGf_γ →G_γ. Theorem 1.4 (Section 5.1). — Let π ∈ Rep^∞(G) and assume that G·ϕ∈WS(π). Then

(i) IfF is non-Archimedean thengMγ acts onπϕ by±Id.

(ii) IfF is Archimedean then the action ofgMγ on the dual spaceπ_ϕ^∗ is locally finite.

LetS∈gbe such that the adjoint action ad(S) diagonalizes overQand ad(S)^∗(ϕ) =−2ϕ. We will call such pairs (S, ϕ) Whittaker pairs. Follow- ing [42] we attach to (S, ϕ) a certain degenerate Whittaker quotientπS,ϕ. If S =h then this is the generalized (a.k.a. neutral) Whittaker quotient (see Section 2.5 for the definitions).

Theorem 1.5 (Section 4). — Let π ∈ Rep^∞(G) and let (S, ϕ) be a Whittaker pair such that G·ϕ ∈ WS(π). Then πS,ϕ 6= 0. Moreover, if F is non-Archimedean then the epimorphism πϕ πS,ϕ constructed in [20, Theorem A]is an isomorphism.

The special case of p-adicF and admissible πfollows from [20, 42, 57].

Another natural question to ask is: given WS(π), what smaller orbits lie in WO(π)? In the case of GLn the answer is: all the orbits lying in the closure of orbits in WS(π). For general reductive groups we deduce from Theorem 1.5 a partial result, see Theorem 7.4 below.

(6)

1.2. Global case

We also provide global analogs of Theorems 1.1–1.5, see Section 8 below.

These analogs generalize several results from [8, 18, 31, 54]. In particular, this puts restrictions on Whittaker supports of cuspidal representations, confirming conjectures from [15, Section 4]. We also provide an analog of Theorem 7.4 and deduce the following corollary.

Corollary 1.6. — LetKbe a number field and letπbe an automorphic representation ofGLn(K). LetO ∈WS(π)and letO⁰⊂ O, whereO denotes the Zariski closure ofOin gl_n(K). ThenO⁰ ∈WO(π).

For a certain analogous statement forSLn(K) see Corollary 8.10 below.

For other groups, we only have a partial result (see Theorem 8.9 below).

1.3. Open questions

Let us summarize some open questions that arise naturally from the results discussed above.

(1) Over a non-archimedeanF, do there exist representations of linear reductive groups with non-special Whittaker support?

(2) Over a non-archimedeanF, are all special orbits admissible?

(3) Analogs of the two questions above over a global fieldK.

(4) Over R, do non-admissible quasi-admissible orbits of split groups appear in Whittaker supports of representations?

(5) Over all fields, are all special orbits quasi-admissible?

(6) Given WS(π), how does WO(π) look like?

Another deep conjecture, posed in [42], says that for any irreducible π, all the orbits in WS(π) lie in the same orbit over the algebraic closure. This is conjectured for all global and local fields, but known only in some exam- ples, in particular for GLn, see [42, Chapter II] for the non-Archimedean case and [52, Theorem D] and [20, Theorem B] for the Archimedean case.

For further open questions we refer the reader to [15, Section 5], and [24, Section 1].

1.4. Structure of the paper

In Section 2 we give the necessary preliminaries on sl2-triples, smooth representations, oscillator representations, Schwartz induction, generalized and degenerate Whittaker models and covering groups.

(7)

In Section 3 we prove several statements on non-generic π∈Rep^∞(B).

Here,B is a Borel subgroup of the metaplectic groupSL^₂(R), and we say thatπis non-generic if it has no non-zero functionals equivariant under the nilradicalN ofB by a non-trivial unitary character. Overp-adic fields, [6, Lemma 5.10] implies that the action ofN onπis trivial. OverRwe prove in Proposition 3.1 that for any non-genericπ, the action of the Lie algebra nofN onπ^∗ is locally nilpotent.

In Section 4 we prove Theorem 1.5. The proof uses the epimorphism WϕWS,ϕconstructed in [20]. Here,WϕandWS,ϕare degenerate Whit- taker models, that define the quotients πϕ and πS,ϕ as the coivariants π_ϕ = (W_ϕ⊗π)_G and π_S,ϕ = (W_S,ϕ⊗π)_G (see Section 2.5 below). Let us recall the construction of the epimorphismWϕWS,ϕ. One can show thatS can be presented ash+Z, wherehis a neutral element forϕandZ commutes withhandϕ. Consider a deformationSt=h+tZ, and denote byutthe sum of eigenspaces of ad(St) with eigenvalues at least 1. We call a rational number 0 < t < 1 regular if u_t = u_t+ε for any small enough rationalε, and critical otherwise. Note that there are finitely many critical numbers, and denote them by t₁ <· · · < t_n. Denote also t₀ := 0 and tn+1 := 1. For eacht we define two subalgebraslt,rt⊂ut. Bothlt and rt

are maximal isotropic subspaces with respect to the form ωϕ, rt contains all the eigenspaces ofZ in ut with positive eigenvalues andlt contains all the eigenspaces with negative eigenvalues. Note that the restrictions ofϕ tol_t andr_t define characters of these subalgebras. Let L_t := Exp(l_t) and Rt := Exp(rt) denote the corresponding subgroups and χϕ denote their characters defined byϕ. The Stone–von-Neumann theorem implies (1.1) WSt,ϕ 'ind^G_L

t(χ_ϕ)'ind^G_R

t(χ_ϕ).

This is an analog of the root exchange Lemma [17, Lemma 2.2]. We show that for any 06i6n, rt_i ⊂lt_i+1. This gives a natural epimorphism

WS_ti,ϕ'ind^G_R

ti(χϕ)ind^G_L

ti+1(χϕ)' WS_ti+1,ϕ. Altogether, we get

Wh,ϕ=WSt0,ϕWSt1,ϕ · · ·WS_tn+1,ϕ=WS,ϕ.

This sequence of epimorphisms naturally defines a sequence of epimorphisms

(1.2) π_h,ϕ =π_S_t

0,ϕπ_S_t

1,ϕ · · ·π_S_tn+1_,ϕ=π_S,ϕ.

We see that for each i, πS_ti+1,ϕ is the quotient of πS_ti,ϕ by the group Ai:=Lt_i+1/Rt, that we show to be commutative. By Proposition 3.1 and

(8)

[6, Lemma 5.10] discussed above, in order to prove the theorem it is enough to show thatπS_ti,ϕ is a non-generic representation ofAi. For that purpose we show that every unitary character ofA_i is given by someϕ⁰∈g^∗ with ad^∗(St_i+1)ϕ⁰=−ϕ⁰ such thatϕ⁰ does not lie in the tangent space toOat ϕ. We then define a quasi-Whittaker quotient πS_ti+1,ϕ,ϕ⁰, and show that its dual is the space of (A_i, χ_ϕ⁰)-equivariant functionals onπ_S_ti_,ϕ. Then we generalize (1.2) to quasi-Whittaker quotients, construct some additional epimorphisms and deduce the vanishing of πS_ti+1,ϕ,ϕ⁰ from the vanishing of π_O⁰ for all O⁰ 6= O with O ⊂ O⁰. We find quasi-Whittaker quotients to be an important new notion. For an additional evidence for that see Remark 4.7.

To prove Theorem 1.4 we show in Section 5.1 that the action of any subgroup of Gfγ isomorphic to a quotient ofSL^2(F) is locally finite. By a corollary from Proposition 3.1 and [6, Lemma 5.10] it is enough to show that it is non-generic. To show that we choose an sl₂-triple (e⁰, h⁰, f⁰) in the Lie algebra of such a subgroup and let ϕ⁰ ∈ g denote the nilpotent element given by the Killing form pairing withf⁰. Consider the deformation St:=h+th⁰. Fort >1/2,e⁰acts trivially on the Whittaker quotientπS_t,ϕ. We show that the action ofe⁰commutes with the maps in (1.2), and deduce thatπS_t,ϕ has no (e⁰, ϕ⁰)-equivariant functionals for anyt>0.

In Section 5.2 we deduce Theorem 1.3 from Theorem 1.5, by way of contradiction. Namely, for any notF-distinguishedOwe find a Whittaker pair (S, ϕ) withϕinOsuch thatπS,ϕis a quotient of a Jacquet module of π, and thus vanishes. By Theorem 1.5 we getO∈/ WS(π).

In Section 6 we discuss quasi-admissible orbits. In Section 6.1we deduce Theorem 1.1 from Theorem 1.4 in the following way. We first note that Wϕ(π) is a genuine representation of Mgγ, which by Theorem 1.4 has a finite-dimensional irreducible subrepresentationρ. Then we construct from ρa finite-dimensional genuine representation ofGf_γand extend it trivially to Gf_ϕ. In Section 6.2 we state several geometric results from [44, 45, 47, 50, 51]

and discuss the connection between the notions of special, admissible and quasi-admissible. We also deduce from Theorem 1.4 and from [39] that the Whittaker support of minimal representations is also minimal.

In Section 7 we formulate and prove Theorem 7.4 that provides informa- tion on WO(π) given WS(π). The proof is based on the method of Section 4, including the quasi-Whittaker quotients, and on Theorem 1.5. We deduce that for π ∈ Rep^∞(GL_n(F)), the set WO(π) is closed under the closure order. We also prove a partial analog for SLn(F).

(9)

In Section 8 we formulate global analogs of Theorems 1.1-1.5 and explain how to adapt the proofs from Section 4–7 to the global case. For example, [6, Lemma 5.10] is replaced by the Fourier decomposition. Our exposition follows [27, Chapter 5].

1.5. Acknowledgements

We thank Joseph Bernstein, Guillaume Bossard, David Ginzburg, Maxim Gurevich, Henrik Gustafsson, Anthony Joseph, Dihua Jiang, Joseph Hund- ley, David Kazhdan, Axel Kleinschmidt, Erez Lapid, Baiying Liu, Daniel Persson, Gordan Savin, Eitan Sayag, and David Soudry for fruitful discussions, the anonymous referee for useful remarks, and Martin Andler for helping us with the French translation of the abstract.

Part of the work on this paper was done during the visit of the three authors to the Max Planck Institute fur Mathematik in Bonn. We thank the administration of the MPI for the invitation and for wonderful working conditions. Another part of this paper was written while two of us participated in the program “Automorphic forms, mock modular forms and string theory” at the Simons Center for Geometry and Physics, Stony Brook University. We are grateful to the administration of the SCGP and the organizers of the program for wonderful working conditions and for very interesting and helpful talks and discussions on the topic.

2. Preliminaries 2.1. Notation

Let F be either Ror a finite extension of Qp and let gbe a reductive Lie algebra overF. We say that an element S ∈gis rational semi-simple if its adjoint action on g is diagonalizable with eigenvalues in Q. For a rational semi-simple elementSand a rational numberrwe denote byg^S_r the r-eigenspace of the adjoint action ofS and byg^S_>_rthe sum L

r⁰>rg^S_r0. We will also use the notation (g^∗)^S_r and (g^∗)^S_>_r for the corresponding grading and filtration of the dual Lie algebrag^∗. For X ∈gor X ∈g^∗ we denote byg_X the centralizer of X in g, and byG_X the centralizer of X in G.

If (f, h, e) is ansl2-triple ing, we will say thateis a nil-positive element forh,f is a nil-negative element forh, andhis a neutral element fore. For a representationV of (f, h, e) we denote byV^e the space spanned by the

(10)

highest-weight vectors and byV^f the space spanned by the lowest-weight vectors.

From now on we fix a non-trivial unitary additive character

(2.1) χ:F→S¹

such that ifF =Rwe haveχ(x) = exp(2πix) and ifF is non-Archimedean the kernel ofχ is the ring of integers.

2.2. sl2-triples

We will need the following lemma which summarizes several well known facts aboutsl2-triples.

Lemma 2.1 ([7, Section 11] or [34]).

(i) Any nilpotent element is the nil-positive element of somesl2-triple ing.

(ii) Ifhhas a nil-positive element theneis a nil-positive element forh if and only ife∈ g^h₂ andad(e)defines a surjection g^h₀ g^h₂. The set of nil-positive elements forhis open ing^h₂.

(iii) Ifeis nilpotent thenhis a neutral element foreif and only ife∈g^h₂ andh∈Im(ad(e)). All such hare conjugate underGe.

(iv) If(f, h, e)and(f⁰, h, e)aresl2-triples thenf =f⁰.

(v) If(f, h, e)is ansl2-triple andZ commutes with two of its elements then it commutes also with the third one.

It is easy to see that the Lemma 2.1 continues to hold true if we replace the nil-positive elements by nil-negative ones (andg^h₂ byg^h₋₂).

Definition 2.2. — We will say that h ∈ g is a neutral element for ϕ∈g^∗ ifhhas a nil-positive element ing,ϕ∈(g^∗)^h₋₂, and the linear map g^h₀ →(g^∗)^h₋₂ given byx7→ad^∗(x)(ϕ)is an epimorphism. We also say that 0∈gis a neutral element for0∈g^∗.

Note that if we identify gwith g^∗ in a G-equivariant way and assume ϕ6= 0, this property becomes equivalent to ϕbeing a nilnegative element forh, or−hbeing a neutral element forϕ.

2.3. Schwartz induction

For non-ArchimedeanF we will work withl-groups, i.e. Hausdorff topo- logical groups that have a basis for the topology at the identity consisting

(11)

of open compact subgroups. This generality includesF-points of algebraic groups defined overF, and their finite covers (see [6, Section 1]).

For F =R we will work with affine Nash groups, i.e. groups that are given inRⁿby semi-algebraic equations, and so is the graph of the multiplication map. This generality includesR-points of algebraic groups defined over R, and their finite covers (see [9, Section 1.1], [2, Section 2,3], [13], [55, Section 3]).

Notation 2.3. — If G is an l-group, we denote by Rep^∞(G) the category of smooth representations ofGin complex vector spaces. ForV, W ∈ Rep^∞(G),V ⊗W will denote the usual tensor product overCandV^∗will denote the linear dual.

If G is an affine Nash group, we denote by Rep^∞(G) the category of smooth nuclear Fréchet representations of G of moderate growth. This is essentially the same definition as in [9, Section 1.4] with the additional assumption that the representation spaces are nuclear (see e.g. [56, Sec- tion 50]). ForV, W ∈Rep^∞(G),V ⊗W will denote the completed projective tensor product andV^∗will denote the continuous linear dual, endowed with the strong dual topology.

Definition 2.4. — If Gis an l-group,H ⊂G a closed subgroup and π ∈ Rep^∞(H), we denote by ind^G_H(π) the smooth compactly-supported induction as in[6, Section 2.22].

If G is an affine Nash group,H ⊂Ga closed Nash subgroup and π ∈ Rep^∞(H),we denote byind^G_H(π)the Schwartz induction as in[9, Section 2].

More precisely, in [9] du Cloux considers the space S(G, π) of Schwartz functions from G to the underlying space of π, and defines a map from S(G, π)to the space C^∞(G, π) of all smoothπ-valued functions on Gby f 7→f, where

f(x) = Z

h∈H

π(h)f(xh)dh,

anddhdenotes a fixed left-invariant measure onH. The Schwartz induction ind^G_H(π) is defined to be the image of this map. Note that ind^G_H(π) ∈ Rep^∞(G).

From now till the end of the Subsection 2.3 letGbe either anl-group or an affine Nash group, andH⁰ ⊂H ⊂Gbe closed (Nash) subgroups.

Lemma 2.5 ([6, Proposition 2.25 (b)] and [9, Lemma 2.1.6]). — For any π∈Rep^∞(H⁰)we have

ind^G_H0(π)'ind^G_Hind^H_H0(π).

(12)

Corollary 2.6. — For anyπ∈Rep^∞(H)we have a natural epimor- phismind^G_H0(π|H⁰)ind^G_H(π).

Definition 2.7. — Forπ∈Rep^∞(G), denote byπ_G the space of coinvariants, i.e. quotient ofπ by the intersection of kernels of allG-invariant functionals. Explicitly,

πG =π/{π(g)v−v|v∈π, g∈G}, where the closure is needed only forF =R.

Note that if F =RandGis connected thenπ_G =π/g_Cπwhich in turn is equal to the quotient of H0(g, π) by the closure of zero.

Lemma 2.8. — Letρ∈Rep^∞(H),π∈Rep^∞(G). Consider the diagonal actions ofH on(π⊗ρ)and ofGonπ⊗ind^G_H(ρ). Then

(π⊗ρ∆_H∆⁻¹_G )_H = (π⊗ind^G_H(ρ))_G.

Proof. — It is easy to see (cf. [20, Appendix A]) that the integration mapf 7→f¯gives the following natural isomorphism of representations of G, whereGacts on functions by left shifts, and coinvariants are taken under the diagonal action on the representation and by right multiplication on the argument:f^h(x) :=ρ(h)f(xh):

S(G, ρ⊗∆_H)_H ∼= ind^G_H(ρ).

Substituting in this formulaGitself as the subgroupH, we obtainS(G, π)_G

∼=π⊗∆⁻¹_G .

We also have a natural isomorphism of representations ofG×G×G S(G, π)∼=S(G)⊗π

In the non-Archimedean case this is evident, and in the Archimedean case this is [9, Proposition 1.2.6]. We will also use the linear automorphism of S(G, π) given by

T(f)(x) :=π(x)f(x)

Note thatT intertwines the action ofG×GonS(G, π) given byf^(g¹^,g²⁾(x) :=π(g2)f(g⁻¹₁ xg2) with the action given byf^(g¹^,g²⁾(x) :=π(g1)f(g⁻¹₁ xg2).

Altogether we have

(π⊗ind^G_H(ρ))G∼= (π⊗ S(G)⊗ρ⊗∆H)_G×H

∼= (S(G, π)⊗ρ⊗∆H)G ×H ∼= (π⊗ρ⊗∆H⊗∆⁻¹_G )H

(13)

2.4. Oscillator representations of the Heisenberg group

Definition 2.9. — LetW_n denote the 2n-dimensionalF-vector space (Fⁿ)^∗⊕Fⁿ and letωbe the standard symplectic form onWn. The Heisen- berg group Hn is the algebraic group with underlying algebraic variety W_n×F and with the group law given by

(w1, z1)(w2, z2) = (w1+w2, z1+z2+ (1/2)ω(w1, w2)).

Note thatH₀=F.

Definition 2.10. — Letχ be the additive character ofF, as in(2.1).

Extendχtrivially to a character of the commutative subgroup0⊕Fⁿ⊕F ⊂ Hn. The oscillator representation $χ is the unitary induction of χ from 0⊕Fⁿ⊕F toHn. Define the smooth oscillator representationσχ to be the space of smooth vectors in$_χ.

Lemma 2.11 (see e.g. [20, Corollary 2.4.5]). — σχ= ind^H_0⊕Fⁿ n⊕F(χ) Theorem 2.12(Stone–von-Neumann). — The oscillator representation

$χ is the only irreducible unitary representation ofHn with central char- acterχ.

Corollary 2.13. — LetL⊂W be a Lagrangian subspace. Extendχ trivially to the abelian subgroupL⊕F ⊂Hn. Then ind^H_L⊕Fⁿ χ∼=σχ.

2.5. Degenerate Whittaker models

Let G be a finite central extension of the group G^alg of F-points of a reductive algebraic group defined overF. LetG^addenote the corresponding adjoint algebraic group.

Lemma 2.14 ([43, Appendix I]). — LetU ⊂G^alg be a unipotent subgroup, andUb be the preimage ofU in G. Then there exists a unique open subgroupU⁰ ⊂Ub that projects isomorphically ontoU.

We will therefore identify the unipotent subgroups of G^alg with their liftings inG.

Definition 2.15.

(i) A Whittaker pairis an ordered pair (S, ϕ) such that S ∈g is rational semi-simple, andϕ∈ (g^∗)^S₋₂. Given such a Whittaker pair, we define the space of degenerate Whittaker models WS,ϕ in the

(14)

following way: letu:=g^S_>₁. Define an anti-symmetric formω_ϕ on g by ωϕ(X, Y) := ϕ([X, Y]). Let n be the radical of ωϕ|u. Note that u,nare nilpotent subalgebras of g, and [u,u] ⊂g^S_>₂ ⊂n. Let U := Exp(u)andN := Exp(n)be the corresponding nilpotent subgroups ofG. Letn⁰:=n∩Ker(ϕ), N⁰:= Exp(n⁰). Ifϕ= 0we define (2.2) WS,0:= ind^G_U(C).

Assume now that ϕ is non-zero. Then U/N⁰ has a natural structure of a Heisenberg group, and its center isN/N⁰. Letχϕ denote the unitary character of N/N⁰ given by χϕ(exp(X)) := χ(ϕ(X)).

Let σϕ denote the oscillator representation of U/N⁰ with central characterχϕ, andσ_ϕ⁰ denote its trivial lifting toU. Define

(2.3) WS,ϕ:= ind^G_U(σ_ϕ⁰).

(ii) For a nilpotent element ϕ ∈g^∗, define the generalized Whittaker model Wϕ corresponding to ϕ to be Wh,ϕ, where h is a neutral element for ϕ if ϕ 6= 0 and h = 0 if ϕ = 0. We will also call Wh,ϕ neutral Whittaker model. By Lemma 2.1 Wϕ depends only on the coadjoint orbit ofϕ, and does not depend on the choice of h. Thus we will also use the notationW_O for a nilpotent coadjoint orbitO ⊂g^∗. See[20, Section 5]for a formulation of this definition without choosingh.

(iii) To π∈ Rep^∞(G) associate the degenerate and generalized Whit- taker quotients by

(2.4) πS,ϕ:= (WS,ϕ⊗π)G andπϕ:= (Wϕ⊗π)G.

Lemma 2.16. — Let l ⊂ g^S_>₁ be a maximal isotropic subalgebra and L:= Exp(l). Letπ∈Rep^∞(G). Then

πS,ϕ∼= (π⊗χϕ)L.

Proof. — By Corollary 2.13 and Lemma 2.5 we haveWS,ϕ∼= ind^G_L(χ_ϕ).

Using Lemma 2.8 we obtain

π_S,ϕ∼= (ind^G_L(χ_ϕ)⊗π)_G∼= (π⊗χ_ϕ)_L. Slightly different degenerate Whittaker models are considered in [20] and denotedW_S,ϕ(π). By Lemma 2.16 and [20, Lemma 2.5.2] they relate toπ_S,ϕ byWS,ϕ(π) =π^∗_S,ϕ. We changed the notion in this paper since for p-adicF, π_S,ϕare the models considered in [42, 57] and forF=R,π_S,ϕare (nuclear) Fréchet spaces.

(15)

Remark 2.17. — If F =R, one can defineWS,ϕ for any semi-simple S with real eigenvalues in the same way, and all our proofs will be valid for this case without changes.

Lemma 2.18. — Assume that Gis an adjoint group, and letS ∈gbe semi-simple. Then there exists an algebraic group morphismν :F^× →G (defined overF) withdν(1) =Sif and only if all the eigenvalues ofSin the adjoint action ong( ¯F)are integers, whereF¯ denotes the algebraic closure ofF.

Proof. — EmbedGintoG⁰:= GL(g) using the adjoint action. Then any ν:F^× →Gdefinesν⁰:F^× →G⁰. Note that a semi-simpleS⁰ ∈g⁰ equals dν⁰(1) for some ν⁰ : F^× → G⁰ if and only if all the eigenvalues of S⁰ on g( ¯F) are integers. The “only if” part follows. For the “if” part, note that if S = dν⁰(1) then Im(dν⁰) ⊂ g, hence Im(ν⁰) ∈ G and thus ν⁰ defines

ν:F^×→Gwith the required property.

Corollary 2.19. — Let S ∈ g be rational semi-simple. Then there exists an algebraic group morphismν :F^× →G^ad and a central element Z∈gsuch thatS−Z∈Im(dν).

Proof. — ReplacingSby an integer multiple we can assume that all the eigenvalues of S in the adjoint action on g( ¯F) are integers. Thus there exists an algebraic group morphism from F^× to the adjoint group of G which includes the projection ofS in its image.

2.6. Covering groups

Letγ= (e, h, f) be ansl2-triple ing. LetGγ denote the joint centralizer of the three elements ofγ. It is well known thatGγ is a Levi subgroup of Gf. Let ϕ∈ g^∗ be given by the Killing form pairing withf. Recall that it induces a nondegenerate symplectic form ωϕ on g^h₁ and note that Gγ

acts ong^h₁ preserving the symplectic form. That is, there is a natural map G_γ →Sp(g^h₁) = Sp(ω_ϕ). LetSp(ω^_ϕ)→Sp(ω_ϕ) be the metaplectic double covering, and set

Gfγ=Gγ×_Sp(ω_ϕ₎Sp(ω^ϕ).

Observe that the natural mapGf_γ →G_γ defines a double cover ofG_γ. We denote by Mγ the subgroup of Gγ generated by the unipotent elements.

LetgMγ denote the preimage of Mγ under the projectionGfγ →Gγ. Note

(16)

that different choices of γ with the same f lead to conjugate groups G_γ andMγ.

One can also define a covering Gfϕ of the group Gϕ = Gf, using the symplectic form defined by ϕ on g/g^ϕ. It is easy to see that this cover splits over the unipotent radical ofG_ϕ, and that the preimage ofG_γ inGf_ϕ is isomorphic toGfγ, see e.g. [44, Section 4].

Definition 2.20. — LetHbe a linear algebraic group defined overF, and fix an embeddingH,→GLn. Denote byH0the open normal subgroup ofH(F)generated by the image of the exponential map.

Note that H₀ does not depend on the embedding of H into GL_n. Note also that ifHis semi-simple and connected thenH0=H(F) and ifF =R thenH₀is the connected component ofH(F). For a finite central extension H⁰ of H, we define H₀⁰ to be the preimage of H0 under the projection H⁰ H.

Definition 2.21 ([44, Section 4]). — We say that a nilpotent orbit O ⊂g^∗ is admissible if for some (equivalently, for any) choice of ϕ ∈ O, the coveringGfϕ→Gϕ splits over(Gϕ)0.

As observed in [44], this definition of admissibility is compatible with Du- flo’s original definition for the Archimedean case, given in [12, Section II.2].

Definition 2.22. — We say that a nilpotent orbit O ⊂ g^∗ is quasi- admissibleif for some (equivalently, for any)ϕ∈ O, the coveringGfϕ→Gϕ

admits a finite dimensionalgenuine representation, that is, a finite dimensional representation on which the non-trivial elementεin the preimage of 1∈G_ϕ acts by−Id.

Definition 2.23. — Let us define the action of Gfγ on πϕ. Since the adjoint action ofG_γ preservesg^h₁ and the symplectic form on it, it preserves U/N⁰. Since σϕ is the unique smooth irreducible representation of U/N⁰ with central characterχϕ, we have a projective action ofGγ onσϕ. By[60]

this action lifts to a genuine representation of Gfγ. This gives rise to an action of Gfγ on Wϕ by (egf)(x) =eg(f(xg)). This action commutes with the action ofGand thus defines an action ofGf_γ onπ_ϕ= (Wϕ⊗π)_G.

More generally, for a Whittaker pair (S, ϕ)letG(S, ϕ)denote the subgroup ofGcorresponding to the Lie algebra(g^S_>₀∩gϕ)⊕g^S_>1. ThenG(S, ϕ)^ normalizes the groupsU andN⁰ corresponding to the pair(S, ϕ)and acts onWS,ϕand onπS,ϕas above.

(17)

It is well known that the metaplectic cover splits over the Siegel parabolic corresponding to any Lagrangian. Indeed, the Weil representation on the Siegel parabolic is given by the action on the Heisenberg group by automor- phisms (in the realization of the representation functions on the Heisenberg group as in Lemma 2.11). These considerations imply the following lemma.

Lemma 2.24. — Let(S, ϕ)be a Whittaker pair, letl⊂g^S_>₁be a maximal isotropic subalgebra andL:= Exp(l). LetG⁰ denote the normalizer of LinG(S, ϕ). Then

(i) The metaplectic cover ofG(S, ϕ)splits overG⁰.

(ii) The action ofG⁰onWS,ϕdefined by splitting the metaplectic cover corresponds under the isomorphismW_S,ϕ∼= ind^G_L(χ_ϕ)to the action (gf)(x) :=f(xg).

(iii) For anyπ∈Rep^∞(G), the action ofG⁰onπS,ϕdefined by splitting the metaplectic cover corresponds under the isomorphism

π_S,ϕ∼= (π⊗χ_ϕ)_L

from Lemma 2.16 to the action on(π⊗χϕ)L given by action on representatives inπ.

3. Some Archimedean technical lemmas

LetP2(R) denote the group of affine transformations of the line. LetN denote the unipotent radical ofP₂(R) and ndenote the Lie algebra ofN. LetB⁰ denote the connected component of the identity inP2(R)

Proposition 3.1. — Let V ∈Rep^∞(B⁰). Suppose that V is not generic, i.e.(V^∗)^N,ψ = 0for any non-trivial unitary characterψofN. Thenn acts locally nilpotently onV^∗.

In the p-adic case, an analogous lemma is proven by Bernstein and Zelevinsky using l-sheaves. We will prove this proposition in Section 3.1 using [9, Section 2]. Let us now derive some corollaries.

Corollary 3.2. — LetV ∈Rep^∞(B⁰)be non-zero. Then(V^∗)^N,ψ6= 0 for some (possibly trivial) unitary characterψ ofN.

LetM pdenote the metaplectic groupSL^₂(R), ands:= Lie(M p) =sl₂(R) denote its Lie algebra.

(18)

Corollary 3.3. — LetV ∈Rep^∞(M p) be non-zero and non-generic.

Then anyλ∈V^∗ generates a finite-dimensional subrepresentation, andV has a (non-zero) finite-dimensional quotient. In particular, every irreducible genuineV ∈Rep^∞(M p)is generic.

Proof. — Choose a standard basis e, h, f for s. Let B denote a Borel subgroup of SL2(R) andBe⊂M pdenote its preimage. Note that the connected component ofBe is isomorphic toB⁰ as a Nash group. Thus Propo- sition 3.1 implies that botheandf act locally nilpotently onV^∗. By e.g.

[3, Lemma C.0.3] this implies that anyλ∈V^∗generates a finite-dimensional subrepresentationW. ThisW non-degenerately pairs with a quotient ofV,

and thus this quotient is finite-dimensional.

Remark 3.4. — Under the assumption of the corollary we cannot in general claim that the action onV is locally finite. For example,V could be the direct product of all irreducible finite-dimensional representations of SL2(R), with the topology given by projections.

In order to apply Corollary 3.3 we will need the following lemmas.

Lemma 3.5. — Let m be a Lie algebra generated by ad-nilpotent elements. Thenm has a basis consisting of ad-nilpotent elements.

Proof. — Let r ⊂m denote the subspace spanned by ad-nilpotent elements. Thenris a module for the Lie group corresponding tom, and thus a subalgebra (in fact, an ideal) ofm. Sincergeneratesm, we obtainr=m.

Note that ifmis semi-simple then ad-nilpotent is the same as nilpotent.

Lemma 3.6. — Letmbe a Lie algebra andV be anm-module. Suppose thatm has a basisX₁, . . . X_n such that eachX_i acts locally finitely on V. Thenmacts locally finitely onV.

Proof. — LetU denote the universal enveloping algebra ofm. For each i, letp_i denote the subspace ofU spanned by all the powers ofX_i. By the Poincaré–Birkoff–Witt Theorem,U =pnp_n−1. . . p1. By the assumption of the lemma, for any finite-dimensional subspaceW ⊂V, and anyi,p_iW is finite-dimensional. By induction we obtain that for anyv∈V, andk6n, pkp_k−1. . . p₁v is finite-dimensional. Taking k = n we obtain that Uv is

finite-dimensional.

3.1. Proof of Proposition 3.1

Lemma 3.7. — Let V be a Fréchet module over the Fréchet algebra of power seriesC[[t]]. Thentacts locally nilpotently on the dual spaceV^∗.

(19)

Proof. — Any λ∈ V^∗ and v ∈V define a distribution onRsupported at 0 byξλ,v(f) :=λ( ¯f v), where ¯f denotes the Taylor series off at 0. Then, for anyλ∈ V^∗ there exists k such that t^kξ_λ,v = 0, and thus λ(t^kv) = 0.

ThusV =∪Ker(t^kλ). Since V is metrizable, the Baire category theorem

implies that Ker(t^kλ) =V for somek.

Corollary 3.8. — Consider the Fréchet algebra of Schwartz functions S(R), under multiplication, and letV be a FréchetS(R)-module annihilated by the idealS(R\ {0}). Letf ∈ S(R)withf(0) = 0. Thenf acts locally nilpotently onV^∗.

Using Fourier transform we obtain the following corollary.

Corollary 3.9. — LetV ∈Rep^∞(R). Let (3.1) A:={f ∈ S(R)s.t. ∀k.

Z ∞

−∞

t^kf(t)dt= 0}.

Suppose thatAV = 0. Then _dt^d acts locally nilpotently onV^∗.

Definition 3.10 ([9, Section 2]). — Let G be an affine Nash group andX be an affine NashG-space. A(G,S(X))-moduleis a representation E ∈ Rep^∞(G) with a continuous action π of the Fréchet algebra S(X) satisfyinggπ(f)g⁻¹ =π(f^g)v, where f^g(x) :=f(g⁻¹x). We say thatE is non-degenerateif the action mapS(X)⊗E→Ehas dense image, and for everyv6= 0∈E, there existsf ∈ S(X)withπ(f)v6= 0.

We denote the category of (G,S(X))-modules by Rep_G,X and the sub- category of non-degenerate modules byRep^nd_G,X.

Theorem 3.11 ([9, Lemma 2.5.7 and Theorem 2.5.8]). — LetGbe an affine Nash group and X be a transitive Nash G-space. Define S(Xe ) :=

C⊕ S(X). Fix x₀ ∈ X and let m_x₀ ⊂ S(Xe ) denote the maximal ideal consisting of functions vanishing atx0. LetE ∈Rep^nd_G,X. Thenmx₀E is a closed proper subspace ofE.

Proof of Proposition 3.1. — Let the algebra A from (3.1) act on V using the identificationN 'R. By Corollary 3.9 it is enough to show that AV = 0. Suppose by way of contradiction thatAV 6= 0 and letE denote the closure of AV. Note that Fourier transform defines an isomorphism A ' S(R\ {0}). We further identify R\ {0} with the Nash manifold X of non-trivial unitary characters of N. The action of P2(R) on X is by multiplication by the reductive part, and the action ofB⁰ has two orbits:

X+ andX−.

(20)

We let S(X) act onE throughA, and note that this is compatible with the action of the groupB⁰. Theorem 3.11 implies that either (E^∗)^N,ψ6= 0 for anyψ∈X₊or (E^∗)^N,ψ6= 0 for anyψ∈X₋. Fix aψwith (E^∗)^N,ψ6= 0.

From the short exact sequence

0→(V /E)^∗→V^∗→E^∗→0 we obtain the exact sequence

(V^∗)^(N,ψ)→(E^∗)^(N,ψ)→H¹(n,(V /E)^∗⊗ψ).

Note that n acts locally nilpotently on (V /E)^∗ by Corollary 3.9. Thus the action of any generator of n on (V /E)^∗⊗ψ is invertible and hence H¹(n,(V /E)^∗⊗ψ) = 0. Since (E^∗)^N,ψ 6= 0 we obtain that (V^∗)^(N,ψ)6= 0, which contradicts the conditions of the proposition. ThusAV = 0 and thus

nacts locally nilpotently onV^∗.

4. Proof of Theorem 1.5

Fix a Whittaker pair (S, ϕ). Let G·ϕ denote the coadjoint orbit of ϕ andG·ϕdenote its closure.

Lemma 4.1 ([20, Lemma 3.0.2]). — There existsZ∈g^S₀ such that(S− Z, ϕ)is a neutral Whittaker pair.

FixZ as in the lemma and leth:=S−Z. For any rational numbert>0 define

(4.1) St:=St,Z :=S+ (t−1)Z, ut:=g^S_>^t₁, vt:=g^S_>1^t and wt:=g^S₁^t. Definition 4.2. — We callt regularifu_t=u_t+εfor any small enough ε∈ Q. Observe that this is equivalent to wt ⊂ gZ. Ift is not regular we call itcritical. For convenience, we will say that 0 is critical.

Note that for anyZ andϕthere are only finitely many critical numbers.

Recall the anti-symmetric formω onggiven byω(X, Y) =ϕ([X, Y]).

Lemma 4.3 ([20, Lemma 3.2.6]). (i) The formωisad(Z)-invariant.

(ii) The radical ofω isRadω=g_ϕ=g^f ⊂g^h₆₀. (iii) Rad(ω|w_t) = Rad(ω)∩wt.

(iv) Rad(ω|_u_t) =v_t⊕Rad(ω|_w_t).

(v) wt∩gϕ⊂uT for anyt < T.

(21)

Choose a Lagrangianm⊂g^Z₀ ∩g^S₁ and let

(4.2) l_t:=m+ (u_t∩g^Z_<0) + Rad(ω|ut) andr_t:=m+ (u_t∩g^Z_>0) + Rad(ω|ut).

Note that these are maximally isotropic subspaces.

Lemma 4.4. — Let 0 6 t < T and suppose that there are no critical numbers in(t, T). Then

(4.3) l_T =r_t⊕(w_T∩g_ϕ).

Moreover,r_tis an ideal inl_T with commutative quotient andv_T is an ideal inrtwith commutative quotient.

Proof. — Decomposing to the eigenspaces of ad(Z) we obtain r_t=m⊕(w_t∩g^Z_>0)⊕(v_t∩g^Z_>₀)⊕(v_t∩g^Z_<0)

(4.4)

lT =m⊕(wT∩g^Z_<0)⊕(vT ∩g^Z_<0)⊕(vT ∩g^Z_>₀)⊕(wT ∩gϕ) (4.5)

Since there are no critical numbers in (t, T) we have v_t∩g^Z_<0= (w_T∩g^Z_<0)⊕(v_T∩g^Z_<0) (4.6)

vT∩g^Z_>₀= (wt∩g^Z_>0)⊕(vt∩g^Z_>₀) (4.7)

This implieslT =rt⊕(wT∩gϕ) andvT ⊂rt. The rest is straightforward.

4.1. Basic comparison lemmas

Lemma 4.5 ([6, Lemma 5.10]). — Assume thatF 6=Rand let Abe a finite-dimensional vector space over F, viewed as an l-group. Let ρ be a smooth representation of A. Suppose that Hom(ρ, χ) = 0 for every non- trivial smooth characterχofA. Thenρis a trivial representation.

Our main tools are Lemma 4.5, Proposition 3.1 and the following notion 4.6 and Lemma 4.8.

Definition 4.6. — We say that(S, ϕ, ϕ⁰)is aWhittaker tripleif(S, ϕ) is a Whittaker pair andϕ⁰ ∈(g^∗)^S_>−2.

For a Whittaker triple(S, ϕ, ϕ⁰)we define a smooth representation ofG that we call aquasi-Whittaker modeland denoteWS,ϕ,ϕ⁰, in the following way. Let

u:=g^S_>₁,v:=g^S_>1,z:=v⊕((g^S₁)∩g_ϕ)andkbe the kernel ofϕ+ϕ⁰ onz.

Then Exp(u)/Exp(k) is a Heisenberg group with center Exp(z)/Exp(k), and ϕ+ϕ⁰ is a character of this center. Let σϕ,ϕ⁰ denote the oscillator