L’INSTITUTFOURIER DE ANNALES

A L

E S

E D L ’IN IT ST T U

F O U R

ANNALES

DE

L’INSTITUT FOURIER

Véronique FISCHER & Fulvio RICCI

Gelfand transforms ofSO(3)-invariant Schwartz functions on the free groupN3,2

Tome 59, n^o6 (2009), p. 2143-2168.

<http://aif.cedram.org/item?id=AIF_2009__59_6_2143_0>

© Association des Annales de l’institut Fourier, 2009, tous droits réservés.

L’accès aux articles de la revue « Annales de l’institut Fourier » (http://aif.cedram.org/), implique l’accord avec les conditions générales d’utilisation (http://aif.cedram.org/legal/). Toute re- production en tout ou partie cet article sous quelque forme que ce soit pour tout usage autre que l’utilisation à fin strictement per- sonnelle du copiste est constitutive d’une infraction pénale. Toute copie ou impression de ce fichier doit contenir la présente mention de copyright.

cedram

Article mis en ligne dans le cadre du

GELFAND TRANSFORMS OF SO(3)-INVARIANT SCHWARTZ FUNCTIONS ON THE FREE GROUP N

by Véronique FISCHER & Fulvio RICCI

Abstract. — The spectrum of a Gelfand pair(KnN, K), whereNis a nilpo- tent group, can be embedded in a Euclidean space. We prove that in general, the Schwartz functions on the spectrum are the Gelfand transforms of Schwartz K- invariant functions onN. We also show the converse in the case of the Gelfand pair (SO(3)nN3,2, SO(3)), where N3,2 is the free two-step nilpotent Lie group with three generators. This extends recent results for the Heisenberg group.

Résumé. — Il est toujours possible d’injecter dans un espace euclidien le spectre d’une paire de Gelfand du type(KnN, K), oùN est un groupe de Lie nilpotent.

Nous démontrons que de manière générale, les fonctions de la classe de Schwartz sur le spectre sont les transformées des fonctions de la classe de Schwartz surN qui sont invariantes parK. Nous prouvons également l’inclusion inverse dans le cas oùN=N3,2est le groupe de Lie nilpotent libre à trois générateurs etK=SO(3).

Ceci étend des résultats récents sur le groupe de Heisenberg.

1. Introduction

Let N be a connected, simply-connected, two-step nilpotent Lie group.

LetKbe a compact group acting by automorphism onN. We assume that (KnN, K) is a Gelfand pair. The Gelfand spectrum can be homeomor- phically embedded in a Euclidean space as follows.

Let D(N)^K be the algebra of left-invariant and K-invariant differential operators on N and {D1, . . . , Dq} a finite set of essentially self-adjoint generators ofD(N)^K. We callDthe ordered family(D₁, . . . , Dq). To each boundedK-spherical functionφonN we assign theq-tuples of eigenvalues µ(φ) = (µ1(φ), . . . , µq(φ)), i.e. such that Djφ = µj(φ)φ. The set Σ_D of suchq-tuples is in 1-1 correspondence with the Gelfand spectrum and the topology induced on it fromR^q coincides with the Gelfand topology [5].

Keywords:Gelfand pair, Schwartz space, nilpotent Lie group.

Math. classification:43A80, 22E25.

We define the Gelfand transform G:L¹(N)^K →Co(Σ_D)by:

GF(µ(φ)) = Z

N

Fφ.¯ We are interested in the following conjecture:

G establishes an isomorphism betweenS(N)^K andS(ΣD) (as Fréchet spaces)

The validity of this statement is independent of the choice ofD (see Sec- tion 3); therefore once proved for one particular choice ofD, it is true for any choice ofD.

Proposition 3.3 of this paper shows the continuous inclusion S(Σ_D),→ G(S(N)^K). This property is already known for the case of the Heisenberg group [2, Theorem 5.5]. The proof relies on a generalisation [2, Theorem 5.2]

of Hulanicki’s Schwartz kernel Theorem [14].

The converse inclusion has been recently shown for any Heisenberg Gelfand pair [2] and we prove it here for(SO(3)nN3,2, SO(3))whereN3,2is the free two-step nilpotent Lie group with three generators; we realiseN_3,2 asR³x×R³y,{0}×R³ybeing the centre. It is known that(SO(3)nN3,2, SO(3)) is a Gelfand pair [3, Theorem 5.12]. We will give explicit formulae for a fam- ily of three essentially self-adjoint operatorsDthat generateD(N3,2)^SO(3), the bounded spherical functions and their corresponding eigenvalues forD.

Historically, the first description of the image of the Schwartz space on the2n+1-dimensional Heisenberg groupHnunder the group Fourier trans- form has been described by D. Geller [8]. In the same spirit, for a Heisenberg Gelfand pair(KnHn, K), a characterisation of the Gelfand transform of the radial Schwartz functions was given in [4] for closed subgroups K of the unitary groupU(n). For more details, we refer the reader to the introduc- tions of [1, 2].

Our goal here is to prove that for any Schwartz SO(3)-invariant func- tion F ∈ S(N3,2)^SO(3), there exists a Schwartz extension of its Gelfand transform:

i.e. ∃f ∈ S(R³) f_|ΣD =GF.

In the proof, we will use the known result for the three-dimensional Heisenberg groupH₁. For this, let us consider N⁰, the quotient group of N3,2 by the central subgroup R²_(y1,y2), andK⁰, the stabiliser of R²_(y1,y2) in SO(3). We will see thatN⁰is isomorphic toH1×Rx₃, andK⁰is isomorphic toU₁oZ2(see Section 2). The Gelfand transform for the pair(K⁰nN⁰, K⁰) will be denoted byG⁰.

Whenever it makes sense, we denote byRF the function onN⁰given by integration of a functionFofN3,2on the central subgroupR²_(y1,y2). The op- eratorRmapsSO(3)-invariant functions onN3,2toK⁰-invariant functions onN⁰, and Schwartz functions on N_3,2 to Schwartz functions onN⁰. Ris 1-1, but does not sendS(N3,2)^SO(3) onto S(N⁰)^K

0

(see Proposition 4.3).

The definition ofRcan be extended to left-invariant differential operators in such a way thatR(DF) = (RD)(RF)for any left-invariant differential operatorsD and any smooth compactly supported functionsF onN. We will see that the image of the operators inD byR completed with −∂_x²

3

gives a familyD⁰ of essentially self-adjoint generators for D(N⁰)^K0. Again we will give explicit formulae forD⁰, the bounded spherical functions, and their corresponding eigenvalues forD⁰.

The spectrum of (K⁰nN⁰, K⁰) can be projected onto the spectrum of (SO(3)nN3,2, SO(3))in the following sense: composing an homomorphism ofL¹(N⁰)^K

0

withR provides a mapping Π : ΣD⁰ →ΣD between the two spectra, that is completely explicit here. In factΠmaps continuously Σ_D⁰ ontoΣD, but is 1-1 only on the regular part of the spectrum (see Section 2).

For any SchwartzSO(3)-invariant functionF ∈ S(N3,2)^SO(3), we have:

G⁰(RF) =GF◦Π.

The existence of a Schwartz extension toR⁴ forG⁰(RF), can be deduced easily from the Heisenberg case [1, 2]; it does not imply directly the exis- tence of a Schwartz extension forGF but is constantly used all along the proof.

This article is organised as follows. In Section 2, we introduce the no- tations and the basic facts concerning the Gelfand spectra of (SO(3)n N3,2, SO(3))and(K⁰nN⁰, K⁰). In Section 3, we give some general settings and the precise statements of our results. In Section 4 we describeRand the restriction mappings. In Section 5 we give the proof of Theorem 5 us- ing an extension of a mean value formula due to Geller in the case of the Heisenberg group [8]. In the appendix, we give, for completeness, detailed proof of some results appearing in this paper and concerning differential operators and functional calculus on them.

2. The Gelfand spectra of (SO(3)nN3,2, SO(3)) and (K⁰nN⁰, K⁰)

We realiseN3,2as R³x×R³y endowed with the law:

(x, y).(x⁰, y⁰) = (x+x⁰, y+y⁰+1 2x∧x⁰),

where ∧ indicates the usual wedge product in R³. N3,2 denotes its Lie algebra. Forj = 1,2,3, let Xj be the left-invariant vector field onN that equals∂_xj at 0, andY_j the left-invariant vector field onN that equals∂_yj. (Xj)_j=1,2,3and(Yj)_j=1,2,3form the canonical basis of N3,2, and satisfy:

[X1, X2] =Y3 , [X3, X1] =Y2 , [X2, X3] =Y1.

The groupSO(3) acts onR³ and thus onN_3,2 by acting simultanously on each copy ofR³. One checks easily that this action is by automorphisms onN_3,2.(SO(3)nN_3,2, SO(3))is a Gelfand pair [3, Theorem 5.12].

Let us define the sub-Laplacian L, the central Laplacian∆ and a third operatorD by:

L=−

3

X

j=1

X_j² , ∆ =−

3

X

j=1

Y_j² , D=−

3

X

j=1

XjYj.

In Section 3, we will show that these operators form a familyD= (L,∆, D) of essentially self-adjoint operators that generateD(N3,2)^SO(3).

The bounded spherical functions and their corresponding eigenvalues for D are known explicitly. Let us define some notation first: for any vector x= (x1, x2, x3) ∈R³, we will writex˜ or x˜for(x1, x2) and, occasionally, [x]₃forx₃.L_l(u) = (1/l!)e^u/2(d/du)^lu^le^−udenotes thel-Laguerre function of order 0 onR. Then the bounded spherical functions onN3,2are:

φ_λ,l,r(x, y) = Z

k∈SO(3)

e^−iλ[k.y]3L_lλ

2|[k.x]˜|²

e^−ir[k.x]3dk , λ >0, l∈N, r∈R.

and

φ0,R(x, y) = Z

k∈SO(3)

e^−iR[k.x]3dk= sin(R|x|)

R|x| dk , R>0;

their eigenvalues forDare given by:

µφ_λ,l,r = λ(2l+ 1) +r², λ², λr , µ_φ0,R = R²,0,0

.

The Gelfand spectrumΣ_D of(SO(3)nN3,2, SO(3))is then realised as the union of the collection of the µ_φλ,l,r, λ > 0, l ∈ N, r ∈ R (the regular part of the spectrum), with the collection of theµφ_R, R∈R(the singular part of the spectrum). Calling(η₁, η₂, η₃)the coordinates corresponding to L,∆, D respectively, Σ_D is then the union, for l > 0, of the surfaces Γl

defined by the equationη²₃=η₂ η₁−(2l+ 1)√ η₂

. They all meet together on the positiveη2-axis, the singular part of ΣD.

N⁰ is the quotient group of N_3,2 by the central subgroup R²(y₁,y₂); we realiseN⁰ asR³x×Rtendowed with the law:

(x, t).(x⁰, t⁰) = (x+x⁰, t+t⁰+1

2[x∧x⁰]₃).

N⁰ denotes its Lie algebra; this is a quotient of N_3,2 by RY₁⊕RY₂ and q = qN_3,2 : N3,2 → N⁰ denotes the quotient mapping. q(Xj) = X_j⁰ is the left-invariant vector field X_j⁰ on N⁰ that equals ∂x_j at 0, j = 1,2,3;

q(Y1) = q(Y2) = 0, and q(Y3) = T is the left-invariant vector field on N that equals∂t. In particularX₃⁰ =∂x₃ lies in the centre ofN⁰.(X_j⁰)

j=1,2,3

andTform the canonical basis ofN⁰. It is easy to see thatN⁰is isomorphic toH1×R. LetK⁰ be the stabiliser ofR²_(y1,y2)⊂R³y in SO(3). The group K⁰ is S O(2)×O(1)∼=O(2)∼=U₁o Z2.

(K⁰nN⁰, K⁰)is a Gelfand pair and its bounded spherical functions are explicitly known:

φ⁰_λ,l,r(x, t) = cos(λt+rxx₃)Ll(λ

2|[k.x]˜|²)dk , λ >0, l∈N, r∈R, and

φ⁰_ζ,r(x, y) =J_o(ζ|˜x|) cos(rx3) , ζ, r∈R, Jobeing the Bessel function of order 0.

We define the following operators:

L⁰=−

3

X

j=1

X_j⁰² , ∆⁰ =−T² , D⁰=−X₃⁰T.

The operatorsL⁰,∆⁰,D⁰and−X₃⁰²areK⁰-invariant, essentially self-adjoint and generateD(N⁰)^K0 (see Proposition 3.1 and Subsection 5.1). We set the familyD⁰= (L⁰,∆⁰, D⁰,−X₃⁰²).

The eigenvalues of the bounded spherical functions forD⁰ are given by:

µ_φ⁰

λ,l,r = λ(2l+ 1) +r², λ², λr, r² , µ_φ⁰

ζ,r = ζ²+r²,0,0, r² .

As in the case of (SO(3)nN3,2, SO(3)), the Gelfand spectrum Σ⁰_D0 of (K⁰nN⁰, K⁰)is then realised as the union of a regular and a singular part:

the regular part is the collection of theµ_φ⁰

λ,l,r,λ >0, l∈N, r∈R, and the singular part is the collection of theµ_φ⁰

ζ,r,ζ, r∈R.

With coordinates (η₁, η₂, η₃, η₄) corresponding to L⁰,∆⁰, D⁰,−X₃⁰² re- spectively, Σ⁰_D0 is the union of the set {(η1,0,0, η4) : 0 6 η4 6 η1} (the

singular set) and the two-dimensional surfaces Γ⁰_l, l > 0, defined by the system of equations

(η²₃=η2 η1−(2l+ 1)√ η2

η²₃=η2η4 .

Notice that the projection onto the hyperplane η₄ = 0 parallel to the η4-axis mapsΣ⁰_D0 onto Σ_D, and is bijective between the two regular sets.

This fact, alluded to already in the introduction, will be relevant in view of the mappingRdefined in Subsection 4.1.

Let us give an equivalent and intrinsic point of view of this fact. As explained in the introduction, the spectrum of (K⁰nN⁰, K⁰)can be pro- jected in the following sense onto the spectrum of(SO(3)nN3,2, SO(3)):

the composition of an homomorphism of L¹(N⁰)^K

0

with R(see also Sub- section 4.1) provides a mapping between the two Gelfand spectra. Realising the Gelfand spectra as explained in the introduction (see also Section 3), this mappingΠ : Σ_D⁰ →Σ_D is then given by:

Π λ(2l+ 1) +r², λ², λr, r²

= λ(2l+ 1) +r², λ², λr , Π ζ²+r²,0,0, r²

= ζ²+r²,0,0 .

Π maps continuouslyΣ_D⁰ onto Σ_D. Moreover Π maps homeomorphically the regular part ofΣD⁰ onto the regular part ofΣD; Πmaps the irregular part ofΣ_D⁰onto the irregular part ofΣ_D, but this correspondence is not 1-1.

3. Results

In this section, we describe the general settings of our work and explain the conjectureG(S(N)^K) =S(Σ_D). We will give the precise statement of our main result in Theorem 3.5.

Let N be a connected, simply-connected Lie group, N its Lie algebra, exp : N → N the exponential mapping and (Ei)^p_i=1 a basis of N. The canonical basis(Ei)^p_i=1ofN being chosen, this induces a Lebesgue measure dX on N and, via the exponential map, a Haar measure dn on N; the spacesL^p(N)are defined with respect to this Haar measure. WhenN is a graded Lie group, following [7, ch1.D], we fix a homogeneous gauge|.|onN and we keep the same notation for the basis(Ej)ofN and the associated left-invariant vector fields onN; we set the following family of semi-norms parametrised by a ∈ N on the Schwartz space S(N) which induces the usual Fréchet space structure onS(N):

kFk_a,N = sup

n∈N,d(I)6a

(1 +|n|)^a|E^IF(n)|.

P(N) denotes the algebra of polynomials on N with real coefficients where N is then identified with the Euclidean vector space R^p. D(N) denotes the algebra of real left-invariant differential operators on N, as operators defined on C_c^∞(N), the space of smooth, compactly-supported functions onN.

ToP ∈ P(N), we associateDP ∈D(N)by:

D_PF(n) =



P(i⁻¹∂_u)F(nexp(

p

X

j=1

u_jE_j))





|u=0

.

We obtain the symmetrisation mapping P 7→ D_P, that is a linear iso- morphism between the algebrasD(N)and P(N)[12, Ch.II Theorems 4.3 and 4.9]

In the appendix we show:

Proposition 3.1. — If (KnN, K) is a Gelfand pair, each operator ofD(N)^K is essentially self-adjoint, that is, it admits a unique self-adjoint extension to an unbounded operator ofL²(N).

Furthermore the operators ofD(N)^Kcommute strongly, in the sense that the spectral resolutions of their self-adjoint extensions commute.

We will use the same notation for an operator of D(N)^K and its self- adjoint extension.

By Hilbert’s Basis Theorem, if a groupKacts orthogonally on some Eu- clidean spaceR^p, the algebraP(R^p)^K ofK-invariant polynomials onR^p is finitely generated [12, Ch.II Corollary 4.10]. Ifρ₁, . . . , ρ_q is a set of gener- ators, we call{ρ1, . . . , ρq}a Hilbert basis for (R^p, K)andρ= (ρ1, . . . , ρq) the corresponding Hilbert mapping. Furthermore, ifρ = (ρ₁, . . . , ρ_q) and ρ⁰ = (ρ⁰₁, . . . , ρ⁰_q0) are two Hilbert mappings for (R^p, K), then there exists Q= (Q₁, . . . , Q_q),Q_j ∈ P(R^q

0), such thatρ=Q◦ρ⁰ (and viceversa). We will make extensive use of G. Schwarz’s Theorem [16]: everyK-invariant smooth function onR^pcan be expressed as a smooth function of a Hilbert basisρ of (R^p, K). In other words, the Hilbert map, ρ, induces an appli- cation given byρ^∗(h) =h◦ρ. Moreoverρ^∗ is a linear continuous mapping fromC^∞(R^q)ontoC^∞(R^p)^K, and also fromS(R^q)ontoS(R^p)^K [2, The- orem 6.1].

Assume that (K nN, K) is a Gelfand pair. Any family of generators ofD(N)^K is obtained as the symmetrisation of a Hilbert basis, and con- versely, if {ρ₁, . . . , ρ_q} denotes a Hilbert basis for (N, K), then {Dρ₁, . . . , Dρ_q}is a set of generators ofD(N)^K.

Let us fix (ρ₁, . . . , ρ_q) an ordered Hilbert basis for(N, K), to which we associate the ordered family of operatorsDρ= (Dρ₁, . . . , Dρ_q). We denote by Σ_Dρ, the set of the q-tuples of eigenvalues µ(φ) = (µ₁(φ), . . . , µ_q(φ)) of Dρ for the bounded K-spherical functions φ on N. As mentioned in the introduction and proved in [5], Σ_Dρ is the realisation of the Gelfand spectrum associated toDρ, in the sense that the setΣDρ of suchq-tuples is in 1-1 correspondence with the Gelfand spectrum and the topology induced on it fromR^qcoincides with the Gelfand topology. In the appendix, we will show thatΣ_Dρ is also the joint spectrum ofDρ:

Proposition 3.2. — Let (ρ1, . . . , ρq) be an ordered Hilbert basis for (N, K). The joint spectrum of the family of strongly commuting, self- adjoint operatorsDρ= (Dρ₁, . . . , Dρ_q)isΣD_ρ.

For a closed subset EofR^q,S(E)denotes the space of restrictions toE of Schwartz functions, endowed with the quotient topology ofS(R^q)/{f : f_|E = 0}; we will often define a class in this quotient as being given as the restriction of a Schwartz function onR^q. The spectrumΣ_D is a closed subset of R^q. We are interested in the conjecture S(N)^K ∼ S(Σ^G D). The existence of a polynomial mapping between two Hilbert mappings implies that the validity of this conjecture is independent of the choice ofD (see [2, Section 3]). The continuous inclusion S(Σ_D) ,→ G(S(N)^K) relies on the following statement, which is a generalisation of Hulanicki’s Schwartz Kernel Theorem proved in the appendix:

Proposition 3.3. — Let (ρ1, . . . , ρq) be an ordered Hilbert basis for (N, K), andD_ρ= (D_ρ1, . . . , D_ρq)the associated family of operators.

Letm be inS(R^q). The operatorm(Dρ)is a convolution operator with aK-invariant Schwartz kernelM =M_m,Dρ ∈ S(N)^K:

∀F ∈L²(N) m(Dρ)F =F∗M.

The Gelfand transform ofM is:

GM =m_|ΣD

ρ.

Furthermore the mappingm∈ S(R^q)7→M_m,Dρ ∈ S(N)^K is continuous.

For the Gelfand spectra of Heisenberg groups or the free two-step nilpo- tent Lie groups, the inclusion of the spectrum in the image of the Hilbert mapping:

ΣDρ ⊂imρ,

is true,independently of the choice of the Hilbert mapping ρ (but we do not know if it is true in general). Here we will use this property only in

the case ofN⁰=H₁×R, where it is known, the spectrum and the Hilbert mapping being explicit.

Lemma 3.4. — The polynomials|x|²,|y|²andx·ygenerate the algebra of polynomials onR³x×R³ythat are invariant under the simultaneous action ofSO(3)on each copy of R³.

Proof. — IfP(x, y)is anSO(3)-invariant polynomial onR³x×R³y, then for each independent vectorsx, y∈R³, we have P(x, y) =P(−x,−y)because the linear transformation that equals−Id on the vector space spanned by xandy, and1on the orthogonal complement line, is inSO(3); this shows that P is invariant under −Id_R3, and thus also under the simultaneous action ofO(3)on each copy of R³. This implies:

P(R³x×R³y)^SO(3)=P(R³x×R³y)^O(3).

By [10, Theorem 4.2.2.(1)],P(R³x×R³y)^O(3)is spanned by|x|²,|y|²andx·y.

Thusρ(x, y) = (|x|²,|y|², x·y)gives a Hilbert mapping for(N3,2, SO(3)).

We compute easily that the associated family of operators by symmetrisa- tion isD = (L,∆, D)defined in Section 2, where we give also an explicit description of the associated realisation of the Gelfand spectrum.

Theorem 3.5. — The Gelfand transform of any Schwartz SO(3)- invariant function onN3,2 admits a Schwartz extension toR³:

∀F ∈ S(N3,2)^SO(3) GF ∈ S(Σ_D).

Moreover the mapping F ∈ S(N3,2)^SO(3) 7→ GF ∈ S(Σ_D) is an isomor- phism of Fréchet spaces.

The group N3,2 admits a slightly bigger group of automorphisms than SO(3), namelyO(3)acting by:

k(x, y) = kx,(detk)ky

, k∈O(3),

It is easily verified that {|x|²,|y|²,(x·y)²} gives a Hilbert basis for (N3,2, O(3)) and the associated family of operators by symmetrisation is D˜ = (L,∆, D²). Following the same lines as in [2, Section 8], we have the following.

Corollary 3.6. — The Gelfand transform is an isomorphism between S(N_3,2)^O(3) andS(ΣD˜)as Fréchet spaces.

From now on, N will stand forN3,2and KforSO(3).

4. R and restriction mappings

4.1. The mapping R

In the introduction, we denoted by RF the function onN⁰ given by in- tegration of a functionFofN on the central subgroupR²(y₁,y₂)whenever it makes sense, for example onL¹(N). It is sometimes convenient to consider R as acting between functions defined on the Lie algebras, rather than on the groups. We will do so without any further mention. The operator RmapsK-invariant functions onN toK⁰-invariant functions onN⁰, inte- grable functions onN to integrable functions onN⁰continuously, Schwartz functions onN to Schwartz functions onN⁰ continuously. It respects con- volution on the groups and abelian convolution on the Lie algebras.

We extend the definition of Rto the algebraD(N)of left-invariant dif- ferential operators onN in the following way: ifD∈D(N), then we define D⁰ =RD∈D(N⁰)by

(D⁰G)◦q=D(G◦q), G∈C_c^∞(N⁰),

whereq=q_N :N →N⁰ is the quotient mapping. Note that ifD =D_P ∈ D(N), P ∈ P(N), then easy changes of variables, see e.g. (A.1) below, leads to:

∀F∈C_c^∞(N) , ∀G∈C_c^∞(N⁰) hR(DF), Gi=hRF, D_{P|N 0}Gi.

This shows thatRDP =D_Q whereQ=P_|N0 is the restriction ofP toN⁰. The mapping Ron functions is dual to the restriction mapping fromN toN⁰in the following sense. Let us denoteF_yandF_tthe Fourier transform with respect to the variablesy∈R³andt∈Rrespectively given by:

FyF(x,y)ˆ = Z

R³

F(x, y)e^−iy.yˆdy, F_tG(x,ˆt) =

Z

R

G(x, t)e^−it.ˆtdt;

whenever it makes sense for a functionG onN⁰ and a functionF on N, identified with functions onN⁰ andN respectively, we have:

(4.1) G=RF⇐⇒ FtG=FyF_|N⁰

In the following subsection, we describe the restriction mapping.

4.2. Restriction and radialisation mappings

For a functionf onN, we denote by Restf =f_|N⁰ its restriction toN⁰. We set:

No=R³x×(R³y\{0}) and N_o⁰ =R³x×(R^t\{0}).

In the next lemma, we define the radialisation mapping Radial :

Lemma 4.1. — For a function h ∈ C^∞(N_o⁰)^K0 and (x, y) ∈ No, the following:

Radial(h)(x, y) =h(k⁻¹x, t) , where y=k(0,0, t)for somek∈K.

defines aK-invariant functionRadial(h), that is smooth onNo.

Radial is an isomorphism between the topological vector spacesC^∞(No)^K andC^∞(N_o⁰)^K

0

, whose inverse isRest. Proof. — For a functionh∈C^∞(N_o⁰)^K

0

and(x, y)∈ No, it is easy to see that Radial(h)(x, y)is well defined andK-invariant.

Let us show that Radial(h)∈C^∞(No)^K. We choose a basis(Aj)_j=1,2,3 for the Lie algebra ofK. At a point(x0, y0)∈ N0 (withy0 =k0(t0,0,0), t₀=|y₀| 6= 0) we choose a local coordinate system(x, y) = x, k(t,0,0)

= σ(x, k, t), where x∈R³, t ∈R⁺ and k varies in a small two-dimensional surface inKcontainingk0and transversal tok0K⁰. This change of variables does not affect the derivatives inx, whereas

∂y_j =cj,0(k, t)∂t+ X

j⁰=1,2,3

cj,j⁰(k, t)Aj⁰.

By homogeneity,c_j,0∈C^∞(K_k×R^∗t)is homogeneous of degree 0 int, and thecj,j⁰ ∈C^∞(Kk×R^∗t)homogeneous of degree(−1)int. More generally, we can write the derivative

∂^I_y=∂_yⁱ¹

1∂_yⁱ²

2∂_yⁱ³

3 , I= (i₁, i₂, i₃)∈N³, as:

(4.2) ∂_y^I =X

cI,J(k, t)∂_t^j0A^j₁¹A^j₂²A^j₃³,

where the sum is over J = (j0, j1, j2, j3) ∈ N⁴, with |J| = |I|, and the cI,J ∈ C^∞(Kk ×R^∗t) are homogeneous of degree (j₀− |I|) in t. As the function(x;k, t)7→h(k⁻¹x, t)is smooth onR³×K×R, (4.2) implies that Radialhis smooth onNo. Furthermoreh∈C^∞(N_o⁰)7→Radialh∈C^∞(No)

is continuous.

Lemma 4.1 implies that the mapping Rest is 1-1 on C^∞(N)^K and on S(N)^K. Let us determine Rest(C^∞(N)^K). We will need the following no- tation:

• Forf ∈C^∞(N)^K, we denote byP_M^(f)(x, t)the homogeneous poly- nomial of degreeM in the Taylor expansion off(x,·)aty= 0:

P_M^(f)(x, y) = X

|j|=M

1

j!∂_y^jf(x,0)y^j.

• Forg∈C^∞(N⁰)^K

0

, we denote byQ^(g)_M(x, t)the homogeneous poly- nomial of degreeM in the Taylor expansion ofg(x,·)at t= 0:

Q^(g)_M(x, t) = 1

M!∂_t^Mg(x,0)t^M. We see:

Q⁽Restf)

M =RestP_M^(f) and thus RadialQ⁽Restf)

M =P_M^(f). Thus a function g ∈ C^∞(N⁰)^K

0

that is the restriction of some function f ∈C^∞(N)^K, necessarily has the following property:

Property (R). For any M ∈ N, Radial(Q^(g)_M) extends to a smooth function onN which is a homogeneous polynomial iny of degreeM, with smooth coefficients inx.

It turns out that this condition is also sufficient:

Proposition 4.2. — Letg∈C^∞(N⁰)^K0.

The function g is in the image ofRest if and only if it satisfies Prop- erty (R).

In this case, Radial(g) extends to aK-invariant smooth functionf on N, whose restriction toN⁰ isg and we haveQ^(g)_M =Rest[P_M^(f)]. Moreover if in additiong∈ S(N⁰)^K

0

, thenRadial(g)∈ S(N)^K

In the proof, we adapt the ideas of the Euclidean setting [13, Theo- rem 2.4].

Proof. — Let g ∈ C^∞(N⁰)^K0 satisfying Property (R). For each M, we denotePM the extension of Radial(Q^(g)_M)to a smooth function onN that is a homogeneous polynomial iny of degree M, with smooth coefficients inx.

LetMo∈N. The Taylor Formula gives:

(4.3) g(x, t)−

Mo

X

j=0

Q^(g)_j (x, t) = t^Mo+1 Mo!

Z 1 0

(1−w)^Mo

∂_t^Mo+1g

(x, wt)dw.

LetI_o∈N³with|Io|=M_o+ 1. We have onNo:

∂_y^IoRadial(g) =∂^I_y^o



Radial(g)−

M_o

X

j=0

Pj



=∂_y^Io



Radial



g−

M_o

X

j=0

Q^(g)_j







.

Now for any(x, y)∈ No,y6= 0can be writteny=k(0,0, t),t∈R^∗,k∈K, and by (4.2) and (4.3), ∂_y^IoRadial(g)

(x, y) can be written as the sum overJ∈N⁴,|J|=Mo+ 1, of:

c_Io,J(k, t) Mo!

Z 1 0

(1−w)^Mo∂_t^j0A^j₁¹A^j₂²A^j₃³h t^Mo+1

∂_t^Mo+1g

(k⁻¹x, wt)i dw.

This last term remains bounded if 0 < |y| = |t| 6 1 because cI_o,J is homogeneous of degree jo−(Mo+ 1). This implies that ∂_y^IoRadial(g) is bounded on a compact neighborhood of (x,0) for any x, and any I_o and Mo. It is easy to see that for anyI∈N³,∂_x^I∂_y^IoRadial(g)satisfies the same conditions. Local boundedness of all derivatives is sufficient to imply that

Radial(g)has a smooth extension toN.

We deduce easily from (4.1) the following characterisation ofR(S(N)^K):

Proposition 4.3. — LetG∈ S(N⁰)^K0. The functionGis inR(S(N)^K) if and only if either of the following equivalent conditions is satisfied:

(i) FtG(identified with a function onN⁰) satisfies Property (R);

(ii) denoting byFx,tthe Fourier transform with respect to the variables x∈R³ andt∈Rgiven by:

Fx,tG(ˆx,ˆt) = Z

R³×R

G(x, t)e^−ix.ˆxe^−it.ˆtdxdt ,

Fx,tG(identified with a function onN⁰) satisfies Property (R);

From G. Schwarz’s Theorem, it follows (compare with [13, Theorem 2.4]):

Corollary 4.4. — LetG∈ S(N⁰)^K0. The functionGis inR(S(N)^K) if and only if either of the following equivalent conditions is satisfied:

(i) for each j ∈ N there exist Schwartz functions aj,i ∈ S(R), i = 0, . . . , j satisfying:

∀x∈R³ Z

R

G(x, t)t^jdt=

j

X

i=0

a_i,j(|x|²)xⁱ₃ ;

(ii) for each j ∈ N there exist Schwartz functions bj,i ∈ S(R), i = 0, . . . , j satisfying:

∀ζ∈R³ Z

R³

Z

R

G(x, t)t^je^−ix.ζdtdx=

j

X

i=0

b_i,j(|ζ|²)ζ₃ⁱ.

5. Proof of Theorem 3.5

Here we give the proof of Theorem 3.5. It is based on the properties of mappings explained in Section 4 and on results already shown on the Heisenberg group [1, 2]. These two key ingredients are used in the proofs of a

“Geller-type” Lemma (Subsection 5.2) and of Theorem 3.5 (Subsection 5.3).

5.1. The Gelfand pair (K⁰nN⁰, K⁰) We easily check that

ρ⁰(x, t) = (|x|², t², x₃t, x²₃), defines a Hilbert mapping of(N⁰, K⁰), which satisfies:

ρ⁰(x, t) = ρ_|N0(x,(0,0, t)), x²₃

and D⁰=Dρ⁰. From the Heisenberg case [1, 2], we deduce:

Lemma 5.1. — For any G ∈ S(N⁰)^K0, there exists ˜g ∈ S(R⁴) with G⁰G= ˜g_|Σ

D0.

Furthermore the mappingG∈ S(N⁰)^K

0

7→ G⁰G∈ S(ΣD⁰)is continuous.

Precisely, continuity of the last mapping means that

∀a∈N ∃C=C(a)>0 ∃a⁰∈N ∀G∈ S(N⁰)^K0

∃˜g∈ S(R⁴) ˜g_|Σ

D0 =G⁰G k˜gk_a,R4 6CkGk_a0,N⁰. (5.1)

Notice that the extension˜gdepends on the Schwartz semi-normk.k_a,

R⁴.

5.2. The Geller-type Lemma

In this subsection, we will state and prove a “Geller-type Lemma”, ex- tending [8, 2]. For this purpose we will need the following remark.

LetF ∈ S(N)^K. The mapping R7→ GF(R²,0,0) =

Z

F(x, y)e^−iRx1dx dy,

is a Schwartz even function on R; by Whitney’s Theorem, there exists a Schwartz functionfo∈ S(R)such that

∀R∈R fo(R²) =GF(R²,0,0);

by Hulanicki’s Schwartz Kernel Theorem or Proposition 3.3, f_o(L) is a convolution operator with aK-invariant Schwartz kernel which we denote byGF(L,0,0) (for brevity reasons, in this section we will often denote a convolution operator and its kernel by the same symbol).

Proposition 5.2 (Geller-type Lemma). — LetF ∈ S(N)^K. There ex- istF₁∈ S(N)^K andF₂∈ S(N)^K satisfying:

F =G(F)(L,0,0) + ∆F1+DF2.

Proof. — LetF be in S(N)^K and G=RF ∈ S(N⁰)^K0. By Lemma 5.1 there exists˜g∈ S(R⁴)withg˜_|ΣD0 =G⁰G. By Proposition 3.3, the operator given by:

Z 1 w=0

∂₂˜g(L, w∆,0,0)dw,

is a convolution operator with a K-invariant Schwartz kernel which we denote byF₁∈ S(N)^K. By spectral calculus, we have:

∆F₁= ˜g(L,∆,0,0)−g˜(L,0,0,0).

We will have finished the proof of Proposition 5.2 once we have shown:

(5.2) ∃F2∈ S(N)^K F− GF(L,0,0)−∆F1=DF2

We denote byH ∈ S(N)^K andI∈ S(N⁰)^K0 the functions given by:

H = F− GF(L,0,0)−∆F₁=F−g˜(L,∆,0,0), I = RH =G−˜g(L⁰,∆⁰,0,0).

The Gelfand transform ofI is given by:

(5.3) G⁰I(µφ⁰) =G⁰G(µφ⁰)−g˜(L⁰φ⁰(0),∆⁰φ⁰(0),0,0). On the singular part of the spectrum, (5.3) yields to:

G⁰I µ_φ⁰

ζ,r

= ˜g ζ²+r²,0,0, r²

−g ζ˜ ²+r²,0,0,0

= 0, because g ζ˜ ²+r²,0,0, r²

= ˜g ζ²+r²,0,0,0

as G⁰G = GF ◦Π; this implies:

(5.4) ∀x∈R³

Z

R

I(x, t)dt= 0.

On the regular part of the spectrum, (5.3) yields to:

G⁰I µ_φ⁰

λ,l,r

= ˜g λ(2l+ 1) +r², λ², λr, r²

−g λ(2l˜ + 1) +r², λ²,0,0 , and in particular forr= 0:

G⁰I µ_φ⁰

λ,l,0

= ˜g λ(2l+ 1), λ²,0,0

−˜g λ(2l+ 1), λ²,0,0

= 0;

this implies for allλ >0:

∀l∈N Z

N⁰

I(x, t)e^−iλtL_l λ

2|˜x|²

dxdt= 0, and{Ll}_l∈N being an orthogonal basis ofL²(R⁺),

∀x˜∈R² Z

R²

I(˜x, x₃;t)e^−iλtdx₃dt= 0.

Eventually, we get:

(5.5) ∀˜x∈R² , ∀t∈R Z

R

I(˜x, x3;t)dx3= 0.

Let us set:

G2(x, t) = Z x₃

−∞

Z t

−∞

I(˜x, w;s)ds dw.

Because of (5.4) and (5.5), we see that G₂ ∈ S(N⁰)^K

0

. Let us show that Fx,tG2 (identified with a function onN⁰) satisfies Property (R). We have fortˆ6= 0andxˆ₃6= 0:

Fx,tG₂(ˆx,ˆt) = (ˆx₃ˆt)⁻¹Fx,tI(ˆx; ˆt) and

Q^(F_M−1^x,tG2)(ˆx; ˆt) = (ˆx₃ˆt)⁻¹Q^(F_M^x,tI)(ˆx; ˆt).

By Proposition 4.3(ii), asI=RH,Fx,tI(identified with a function onN⁰) satisfies Property (R), that is Radial

Q^(F_M^x,tI)

extends to a smooth K- invariant function onN which is a homogeneous polynomial iny of degree M, with Schwartz coefficients inx. By G. Schwarz’s Theorem, there exists a functionQ˜M ∈C^∞(R³)of the form:

Q˜_M(r₁, r₂, r₃) = X

2j₁+j₂=M

c_j(r₁)r^j₂¹r^j₃² , c_j∈ S(R), satisfying:

Radial

Q^(F_M^x,tI)

= ˜QM ◦ρ.

That is:

Q^(F_M^x,tI)(ˆx,ˆt) = ˜QM(|ˆx|²,tˆ²,xˆ3ˆt) = X

2j1+j2=M

cj(|ˆx|²)ˆt^2j1(ˆx3ˆt)^j2. Because of (5.5), we have:

∀x˜ˆ∈R² , ∀ˆt∈R Fx,tI(˜x,ˆ 0; ˆt) = 0,

thus the term cj(|ˆx|²) with j = (j1,0) is zero: we can factor out one (ˆx3ˆt). This implies that for M > 0, Radial(Q^(F_M−1^x,tG2)(ˆx; ˆt)) extends to a