Geometry of <span class="mathjax-formula">$2$</span>-step nilpotent groups with a left invariant metric

A NNALES SCIENTIFIQUES DE L ’É.N.S.

P ^ATRICK E ^BERLEIN

Geometry of 2-step nilpotent groups with a left invariant metric

Annales scientifiques de l’É.N.S. 4^e série, tome 27, n^o5 (1994), p. 611-660

<http://www.numdam.org/item?id=ASENS_1994_4_27_5_611_0>

© Gauthier-Villars (Éditions scientifiques et médicales Elsevier), 1994, tous droits réservés.

L’accès aux archives de la revue « Annales scientifiques de l’É.N.S. » (http://www.

elsevier.com/locate/ansens) implique l’accord avec les conditions générales d’utilisation (http://www.numdam.org/conditions). Toute utilisation commerciale ou impression systé- matique est constitutive d’une infraction pénale. Toute copie ou impression de ce fi- chier doit contenir la présente mention de copyright.

Article numérisé dans le cadre du programme Numérisation de documents anciens mathématiques

http://www.numdam.org/

4° serie, t. 27, 1994 p. 611 a 660.

GEOMETRY OF 2-STEP NILPOTENT GROUPS WITH A LEFT INVARIANT METRIC

BY PATRICK EBERLEIN

ABSTRACT. - We consider properties of closed geodesies in a compact nilmanifold T\N, where N is a simply connected 2-step nilpotent Lie group with a left invariant metric and r is a discrete cocompact subgroup of N.

Among other results we show 1) There is an obstruction (resonance) to the density in T\ (T\N) of the set of vectors P that are periodic with respect to the geodesic flow. In particular P is not always dense in T\ (T\AQ, but P is dense in T\ (T\AQ for any F if N is of Heisenberg type. 2) Every free homotopy class of closed curves in T\N contains a closed geodesic of largest period. Define the maximal length spectrum of FW to be the collection with multiplicities of these largest periods. If T\N, r* \N* are compact 2-step nilmanifolds with the same marked maximal length spectrum, then we show that T\N, T*\N* are equivalent up to isometry and r-almost inner automorphism in the sense of Gordon and Wilson.

Introduction

Nilpotent Lie groups play an important role in many areas of mathematics, and 2-step nilpotent groups have a special significance. They are the nonabelian Lie groups that come as close as possible to being abelian, but they admit interesting phenomena that do not arise in abelian groups. In this paper we study the differential geometry of simply connected, 2-step, nilpotent Lie groups N with a left invariant Riemannian metric ( , ).

We are especially interested in those geometric properties of {TV, ( , )} that do not depend on the choice of ( , ). One would expect to find some properties that are similar to those in flat Euclidean space, which in this context one may regard as a simply connected, abelian Lie group of translations with a canonical left invariant metric. Such properties do exist, but other geometric properties of {N, ( , )} are foreign to Euclidean geometry.

For example, J. Wolf in [Wol] proved that any nonabelian nilpotent Lie group with a left invariant metric must admit both positive and negative sectional curvatures, and J. Milnor in [M] extended this result to Ricci curvatures. More generally, Milnor showed in [M] that the geometry of any Lie group G with a left invariant metric reflects strongly the algebraic structure of the Lie algebra Q. Many of the results of this paper illustrate that principle.

The geometry of simply connected nilpotent Lie groups TV with a left invariant metric is also relevant to the study of simply connected homogeneous spaces M whose sectional

This research was supported in part by NSF Grant DMS-8901341.

ANNALES SCIENTIFIQUES DE L'ECOLE NORMALE SUPERIEURE. - 0012-9593/94/057$ 4.00/© Gauthier-Villars

curvatures are bounded above by a negative constant. In this context the groups N arise as groups of isometries of M that fix a point x in the boundary sphere M (oo) and act simply transitively on each horosphere at x. Each horosphere H at x with the induced metric from M is isometric to N with an appropriate left invariant metric ( , ) which depends on H. If M is symmetric, then N has 2-steps and {N, ( , )} is a manifold of Heisenberg type [see below and also in (1.6)]. For a discussion of homogeneous manifolds of negative sectional curvature see [Hei].

The literature does not seem to contain much discussion of the geometry of nilpotent Lie groups with a left invariant metric. For this reason we include in the first two sections some basic geometric facts that are probably known to individual researchers but have not been written down. In the last three sections of the paper we consider the behavior of geodesies in a simply connected, 2-step nilpotent Lie group N with a left invariant metric.

In particular, we study aspects of the behavior of closed geodesies in a compact quotient manifold F\N, where F is a discrete cocompact subgroup of N that acts on N by left transformations. We present the main results below but avoid a more detailed discussion of the organization of the paper.

To study the geometry of 2-step nilpotent groups with a left invariant metric we adopt the approach of A. Kaplan used in [Kl], and we now describe its main features. Let J\f be a 2-step nilpotent Lie algebra with an inner product ( , ). Let N be the unique, simply connected, 2-step nilpotent Lie group whose Lie algebra is At, and equip N with the left invariant metric determined by the inner product ( , ) on J\T = Te N. Let Z denote the center of A/", and let V denote the orthogonal complement of Z in J\T.

Each element Z of Z defines a skew symmetric linear map j (Z) : V —^ V given by j (Z) X == (adX)* (Z) for all X <E V, where (ad X)* is the adjoint of ad X relative to the inner product ( , ). Equivalently and more usefully j (Z) is defined by the equation ( j ( Z ) X , Y ) = ([X, V], Z ) for all X, Y G V.

Conversely, for each pair of positive integers m, n and each linear map j : R71 —^ so (m) we obtain a metric, 2-step nilpotent Lie algebra A/" = R7' 9 R771 (orthogonal direct sum), where Z = R^ is the center of Af and the Lie bracket on V = R771 is defined by the equation above. See (1.5).

All of the basic geometry of {N, { , )} can be described by the maps {j (Z) : Z <E Z} as we show in section 2. This was first made clear by A. Kaplan in [Kl], who used the maps j (Z) to study the geometry of groups of Heisenberg type, those groups {N, ( , )} for which j (Z)2 = -1 Z |2 Id for every Z e Z. Many of his proofs are valid without change in the general 2-step nilpotent case, and others require only small modifications. The spaces {N, ( , }} of Heisenberg type should be regarded as the model spaces in the class of simply connected, 2-step nilpotent Lie groups with a left invariant metric. In this class they play a role that is similar to the role played by the Riemannian symmetric spaces in the class of all Riemannian manifolds. Groups of Heisenberg type have especially large isometry groups, and the geodesic symmetries at each point preserve the Riemannian volume form [K2]. Moreover, every unit speed geodesic in a group N of Heisenberg type lies in at least one 3-dimensional totally geodesic submanifold of N. This property also characterizes groups of Heisenberg type in the class of simply connected, 2-step nilpotent Lie groups with a left invariant metric [El]. If M is a symmetric space of strictly negative sectional

4e SERIE - TOME 27 - 1994 - N° 5

curvature, then 2-step nilpotent groups N of Heisenberg type arise as groups of isometrics of M that act simply transitively on horospheres of M. In particular, if G = KAN is an Iwasawa decomposition of G = IQ (M), then the group N with a natural left invariant metric is a group of Heisenberg type [Ko]. See also the examples following (1.3).

We now describe the main results of the paper. Let N denote a simply connected, 2-step, nilpotent group with a left invariant metric, and let F denote a lattice in TV; that is, r is a discrete subgroup of N such that the quotient manifold F\N is compact, where r acts on At by left translations. Let {</} denote the geodesic flow in the unit tangent bundles SN or 5(r\AQ.

1. FIRST INTEGRALS FOR {^}. - In Corollary (3.3) we show that {gt} admits a smooth

^-valued first integral /; that is, there exists a smooth function / : SN —> Z such that / (^* v) = f (v) for all v G SN and for all t e R. More precisely, let ^ G Tn N be any vector and write ^ = dLn (XQ + ZQ\ where Ln is the left translation by n and Xo, ZQ are uniquely determined vectors m T e N = A T = V ^ Z such that Xo e V and Zo G Z.

Then / (^) = Zo defines a first integral for {(^} on TN and hence also on SN by restriction. The first integral / is clearly invariant under {dLn : n e N}, and hence it descends to a Z-valued first integral for {^t} on S (T\N) for any discrete subgroup F of N. In particular the geodesic flow in S (r\7V) does not have a dense orbit since the first integral / is nonconstant.

The first integral / is reminiscent of the canonical Revalued first integral of the geodesic flow in flat Euclidean space R71. In fact, the subspace Z of At = Tg N defines an integrable left invariant distribution Z in N whose maximal integral manifolds are flat, totally geodesic imbedded submanifolds, the orbits of the center Z = exp (Z) of N , where Z acts by left translations.

2. DENSITY OF PERIODIC VECTORS IN S (T\N). - Let r be a lattice in TV. A unit vector v G S (r\7V) is periodic relative to the geodesic flow {^} on S (T\N) if g^ v = v for some uj > 0; that is, v is tangent to a closed geodesic of T\N. For any flow {(|)t} on a space X it is a basic problem to determine if the periodic vectors for the flow are dense in X. We show that this is not always the case for {g1} on S (T\N).

The density of periodic vectors for {^} on S (T\N) turns out to be related to a property of the skew symmetric linear maps {j (Z) : Z e Z} which we call resonance. Given Z € Z a map j (Z) : V —^ V is said to be in resonance if the ratio of any two nonzero eigenvalues ofj (Z) is a rational number. Note that this ratio is always a real number since the eigenvalues of j (Z) are purely imaginary. If N is of Heisenberg type, then every map j {Z\ Z G i?, is in resonance since the condition j (Z)2 = —\Z\2 Id implies that j (Z) has eigenvalues ±i \ Z |. In (5.6) and (5.7) we prove the following two results. The first of these has recently been generalized in [Ma].

1. Let TV be a simply connected, 2-step nilpotent Lie group of Heisenberg type, and let r be any lattice in N. Then the periodic vectors for the geodesic flow {^} in S (T\N) are dense in S(T\N).

2. Let TV be a simply connected, 2-step nilpotent Lie group with a left invariant metric and a 1-dimensional center. Then the following properties are equivalent.

a) The linear map j {Z) : V —^ V is in resonance for every Z e Z.

ANNALES SCIENTIFIQUES DE L'ECOLE NORMALE SUPERIEURE

b) For some lattice F in J\f the periodic vectors for {g*} in S (F\N) are dense in

s(r\N).

c) For every lattice F in TV the periodic vectors for {^} in S {T\N) are dense in

5(ryv).

We do not know if the hypothesis that N have a 1-dimensional center can be removed.

Note that checking the hypothesis a) in 2) reduces to checking it in a single case for any nonzero vector Z of Z.

In (5.8) we construct a lattice F in a 5-dimensional simply connected, 2-step nilpotent group N with 1-dimensional center such that none of the maps j ( Z ) , Z e Z, are in resonance. It follows from 2) above that the periodic vectors in S (T\7V) are not dense in

S(T\N), and the same is true for any lattice in N.

3. THE ASSOCIATED FLAT TORI TB AND Tp. - If N is a simply connected nilpotent Lie group with Lie algebra A/", then the exponential map exp : Af —> N is a diffeomorphism.

Let log : N —^ At denote the inverse of exp. Let F be a lattice in a simply connected, 2-step, nilpotent Lie group N with Lie algebra A/" = V 9 Z, and let Try : A/" —> V denote the projection map. It is elementary to show that Try log F and log F D Z are vector lattices in V and Z respectively. Define flat tori TB = VK-KV log F) and Tp = Z/(logF H Z). In (5.5) we show that there exists a Riemannian submersion of F\N onto Tp whose fibers are imbedded, flat, totally geodesic tori isometric to Tp. These fibers are also the orbits in F\N of Jo (r\^V), which acts freely on F\N. The (closed geodesic) length spectra of TB and TF are closely related to the length spectrum of F\N and in fact determine the length spectrum of F\N if N is of Heisenberg type (5.17). However, the isometry classes of TB and TF do not determine the isomorphism class of the fundamental group F of F\N (5.23).

We note that R. Palais and T. Stewart in [PS] showed that the compact 2-step nilmanifolds are precisely the total spaces of principal torus bundles over a torus.

4. LENGTH SPECTRUM OF F\N. - Let F be a lattice in N, and let C denote a free homotopy class of closed curves in F\N. Let I (C) denote the collection of lengths of closed geodesies of F\N that belong to C. The length spectrum of F\N is the collection of all ordered pairs (L, m), where L is the length of a closed geodesic in F\N and m is the multiplicity of L, L e. the number of free homotopy classes C for which L G I (C). Compact nilmanifolds F\N and r*\7V* are said to have the same marked length spectrum if there exists an isomorphism (f) of F onto F* such that I (<^ C) = I (C) for all free homotopy classes C in r\7V, where (^ denotes the bijection induced by (j) between free homotopy classes of closed curves in T\N and r*\^V*.

The set I (C) contains in general more than one number for each free homotopy class C [(4.8) and (4.11)]. However, I (C) always contains a largest number F (C), which is explicitly computable (4.5). The maximal length spectrum of T\N is the collection of all ordered pairs (£, m), where L = F (C)} for some free homotopy class C of closed curves in T\N and m is the number of free homotopy classes C for which L = Z* (C).

Compact nilmanifolds F\N and r*\7V* are said to have the same marked maximal length spectrum if there exists an isomorphism of) of F onto F* such that Z* (<^ C) = F (C) for all free homotopy classes C in T\N.

4'^ SERIE - TOME 27 - 1994 - N° 5

The closed geodesies with maximal length Z* (C) in a free homotopy class C of F\N have both geometric and dynamical significance. If 7 is a closed geodesic in C with length r (C), then 7 is the projection of a geodesic in N of the form t —> n ' exp (^) for some

^ G A/". The geodesies of A/" are rarely left translates of 1-parameter subgroups of N (see (3.9)). To explain the dynamical significance of F (C) we define for each number a; in I (C) a set 5W^ (C) consisting of those unit vectors in S (T\N) that are tangent to a closed geodesic of length uj that belongs to C. Clearly each set SN^ (C) is invariant under the geodesic flow {^}. If a;* = F (C) and if uj is any number in / (C) distinct from a;*, then we show in (4.18) that the dimension of SN^. (C) is strictly smaller than the dimension of SN^(C). Moreover, in (4.17) we show that SN^. (C) is a smooth submanifold of

S (F\7V), but we do not know if this is true for SN^ {C) with uj ^- a;*.

In (5.20) we show that there are essentially only two ways that compact, 2-step nilmanifolds F\N and r*\7V* can have the same marked maximal length spectrum (Note that if T\N and r*\7V* have the same marked length spectrum then they have the same marked maximal length spectrum.)

1. There exists an isomorphism ^ of N onto N * such that ^ is also an isometry and

^(r) = r*.

2. N = TV* and F* = ^ (F), wherre ^ is a F-almost inner automorphism of N; that is, for each element 7 of the lattice F there is an element a of N, possibly depending on 7, such that ^ (7) = a ' 7 • a~1.

The importance of F-almost inner automorphisms of N was discovered by C. Gordon and E. Wilson, who proved in [GW1] that if ^ is a F-almost inner automorphism of N for some lattice r of TV, then r\7V and ^ (r)\7V have the same spectrum of the Laplacian on functions but are not in general isometric. Later work (cf. [Gl] and [DG1]) showed that if '0 is a F-almost inner automorphism of TV, then T\N and ^ (r)\7V have the same marked length spectrum and the same Laplacian spectrum on functions and differential forms. It follows from these facts and (5.20) that if r\7V and r*\7V* are compact, 2-step nilmanifolds with the same marked maximal length spectrum, then they also have the same marked length spectum and the same Laplacian spectrum on functions and differential forms. Moreover, (5.20) shows that if r\7V and r*\7V* are compact, 2-step nilmanifolds with the same marked maximal length spectrum, then the associated sets of tori {TB, Tp}

and {TB*, TF-} are pairwise isometric (Corollary 5.22).

The relationship between the (unmarked) maximal length spectrum and the marked maximal length spectum is somewhat mysterious. In (5.23) we construct two examples that illustrate the problem:

1. There exist homeomorphic compact, 2-step nilmanifolds T\N and r*\7V* that have the same maximal length spectra but do not have the same marked maximal length spectra for any choice of isomorphism (f) of r onto F*. In this case the associated tori TB and IB* have the same length spectrum but are not isometric.

2. There exist compact, 2-step nilmanifolds r\7V and r*\.?V* with the same maximal length spectra such that F is not isomorphic to F*. Hence their marked maximal length spectra are a priori different. This also shows that the maximal length spectrum does not

ANNALES SCIENTIFIQUES DE L'ECOLE NORMALE SUPERIEURE

determine the isomorphism class of the fundamental group F of T\N. However, in this example the associated sets of tori {TB, Tp} and {TB*, Tp-} are pairwise isometric.

In a compact, 2-step nilmanifold T\N the relationship between the length spectrum and the spectrum of the Laplacian acting on functions or differential forms is unclear, even in the case that N is of Heisenberg type. This relationship deserves further study. It is also unclear to what extent the results (if not the methods) of this paper generalize to simply connected nilpotent Lie groups with a left invariant metric and an arbitrary number of steps.

We are grateful to the referee of the first version of this article, who simplified the proofs of several results and made many useful remarks. We would also like to thank W. Ballmann and C. Gordon for helpful comments.

1. Definitions and examples

Let AT denote a finite dimensional Lie algebra over the real numbers. For each integer z ^ 1 we define AT' = [AT, A/''"¹], where A/"⁰ = Af. The Lie algebra At is nilpotent if Af^ = {0} for some positive integer i. A nilpotent Lie algebra AT has a nontrivial center that contains A/^~¹ if Af'¹ = {0}. A nilpotent Lie algebra AT is k-step if A/^ = {0}

but A^-1 / 0.

Let N denote the unique simply connected nilpotent Lie group corresponding to a given nilpotent Lie algebra AT. Let exp : Af —> N denote the Lie group exponential map. It is known that exp is a diffeomorphism [R], p. 6. We let log : N —» At denote the inverse of exp.

2-step nilpotent groups and algebras.

We are primarily interested in the case that N and Af are 2-step nilpotent. In this case the Campbell-Baker-Hausdorff formula (cf. [Hel], p. 96) yields the following simple expression for the multiplication law in N.

(1.1) exp (X) . exp (V) = exp ( x + Y + 1 [X, Y}\

\ 2 )

for arbitrary elements X, Y of At.

From the expression above we obtain

(1.2) Let 0, ^ be arbitrary elements of TV, and write 0 = exp(X), ^ = exp(V) for suitable elements X, Y of AT. Then

a)W1 = e x p ( V + [ X , V])

b) [^ ^] = W1^-1 = exp([X, V])

c) (f) commutes with '0 if and only if [X, Y] = 0 d) log (0 • '0) = log (f) + log ^ + . [log (/), log '0].

The following description of the differential of the Lie group exponential map exp : At —^ N will be useful.

4e SfiRIE - TOME 27 - 1994 - N° 5

(1.3) LEMMA. - Let J\f denote a 2-step nilpotent Lie algebra, and let N denote the simply connected 2-step nilpotent Lie group with Lie algebra At. Let exp : A/" —^ N denote the exponential map. Then for any elements ^, A of AT we have d exp^ : T^J\T —^ Texp(^) N is given by

d exp^ (A^) = d£exp (Q ( A + , [A, ^] )

\ ^ /

w/^r^ A^ denotes the initial velocity of the curve t —^ ^ + t A and Z/exp (0 denotes left translation by exp(^).

Prw/. - By definition d exp^ (A^) is the initial velocity of the curve t —> exp (^ + 1 A).

By (1.1) we see that d£exp(o ( A + - [A, ^] ) is the initial velocity of the curve

^ ^ e x p ^ . e x p f J A + ^ A ^ ] } ) = e x p ( $ + t A ) . D

\ I z J /

Exampfc^.

It is easy to construct 2-step nilpotent Lie algebras. Let V, Z be any finite dimensional real vector spaces with bases {V^ . . . , Vn} for V and {Zi, . . . , Z^} for Z. Let M = V C ^ and define a bracket operation in J\f by

m

[^,V,]=^^..^

a=l

where the constants {C^} are chosen so that C^; = -C^ for 1 ^ %, j ^ n, 1 ^ a ^ m, but not all of the constants are zero. Define [Z^, C]=Ofor3i\l(,eAf,l^a^ m. The Jacobi identity is automatically satisfied since [A/\ A/"] ^ Z, and J? lies in the center ofA/\

We construct more explicit examples (cf. [K2], p. 39).

Example 1. - Heisenberg algebras

Let n ^ 1 be any integer and let { X i , . . . , Xy,, Vi, . . . , Vn} be any basis of j^2n ^ y ]^ ^ be a 1-dimensional vector space spanned by an element Z. Define [Xi, Yi\ = -[Yi, Xi\ = Z for 1 ^ i ^ n with all other brackets zero. The Lie algebra ]\[ = V 9 Z is the (2n + 1)-dimensional Heisenberg algebra.

Remark. - Let C .H^ denote the complex hyperbolic space of real dimension 2 n. The normalized sectional curvatures K (n) satisfy -4 ^ ^ (II) ^ -1. Let G = lo^CH71) and let G = KAN be an Iwasawa decomposition of G. The group N is the (2n + 1)- dimensional Heisenberg group, the simply connected, (2n + 1) -dimensional nilpotent Lie group whose Lie algebra is the (2n + 1)-dimensional Heisenberg algebra. Geometrically, AN acts transitively on C H71 and fixes a unique point x in the boundary sphere C H^ (oo).

The group N acts simply transitively on each horosphere at x (see section 6 of [Ka] and [E2]).

Example 2. - Quatemionic Heisenberg algebras.

Let n ^ 1 be any integer. For each integer i with 1 ^ i ^ n let H1 denote a 4-dimensional real vector space with basis [Xi, Y,, Vi, Wi}. Let V = ®H\ Let 2 be a 3-dimensional

ANNALES SCIENTIFIQUES DE L'ECOLE NORMALE SUPERIEURE

real vector space with basis {Zi, Z^, ^3}, and let AT = V C Z. We define a bracket operation in A/" as follows:

[Zj^] = 0 for 1 ^ j ^ 3 and all ^ G Af

[X^Y,}=Z^ [X^Vi}=Z^ [X^Wi]=Z^ for l ^ i ^ n [Y^Xi]=-Z^ [Y^Vi\=Z^ [Y^Wi}=-Z^ for l ^ i ^ n [V^Xi]=-Z^ [V^Y^=-Z^ [V^W,]=Z^ for l^i^n [W^X,}=-Z^ [W^Y^=Z^ [W^V^=-Z^ for l ^ i ^ n all other brackets are zero

The resulting Lie algebra At is the quaternionic Heisenberg algebra of dimension 4 n + 3.

Remark. - Let N denote the simply connected (4 n+3)-dimensional nilpotent Lie group whose Lie algebra is the (4n + 3)-dimensional quaternionic Heisenberg Lie algebra. Let H H71 denote the quaternionic hyperbolic space of real dimension 4 n. The remark of the previous example now applies to N if one replaces C f f ^ by H H " ' .

(1.4) DEFINITION. - A 2-step nilpotent Lie algebra Af is nonsingular ifad X : At^ Z is surjective for all X G N — Z.

Here ad X (V) = [X, Y] for all X, Y G Af. The Heisenberg and quaternionic Heisenberg algebras are nonsingular for any positive integer n. The nonsingular 2-step nilpotent Lie algebras Af form an important class of 2-step nilpotent Lie algebras, and in general one can say much more than in the general case about the geometry of the corresponding group N equipped with a left invariant metric.

Metric examples. - We now assume that our 2-step nilpotent Lie algebra A/" is equipped with a positive definite inner product ( , ). Let Z denote the center of A/", and let V denote the orthogonal complement of Z in J\f relative to { , }. For each element Z in Z we define a skew symmetric linear transformation j (Z) : V —> V by

j (Z) X = (ad X)' Z for all X G V

where (ad X)* denotes the adjoint of ad X. Equivalently one has the following more useful characterization:

(1.5) ( j { Z ) X , Y ) = ( [ X , V], Z ) forallX, V G V, all Z G Z

The transformations {j (Z) : Z G Z} capture all of the geometry of N equipped with the left invariant metric determined by ( , ). The notation j (Z) was apparently first introduced by A. Kaplan in [Kl] to study 2-step nilpotent groups N of Heisenberg type.

Given a pair of positive integers m, n, each linear map j : R71 —^ so (m] determines a metric, 2-step nilpotent Lie algebra AT. Define At to be the orthogonal direct sum J\[~ = R^ 9 R^ where each factor has the standard metric. Then equip Af with the Lie bracket determined by (1.5), where ^ = Z and IR771 = V.

461 SERIE - TOME 27 - 1994 - N° 5

(1.6) DEFINITION. - A 2-step metric nilpotent Lie algebra [J\T^ ( , )} is of Heisenberg type if

j ( Z )2 = - I Z ^ I d o n V

for every choice of Z G 2. A simply connected 2-step nilpotent Lie group {N, ( , )} with a left invariant metric is of Heisenberg type if its Lie algebra {.V, ( , )} is of Heisenberg type.

If {A/\ ( , )} is of Heisenberg type, then from the definitions we immediately obtain the following facts.

a) ( j ( Z ) X J ( Z ^ X ) = ( Z , Z ^ \ X \2 for all Z, Z* G Z, a l l X c V n ^ /&) { H Z ) X J ( Z ) Y ) = \ Z \2{ X ^ Y ) for all Z G Z, allX, Y G V

^•^ ] c ) \j(Z)X\= \X\\Z\ f o r a l I Z G Z , a l l X e V

[ri) j ( Z ) o j ( Z * ) + j ( Z * ) o j ( Z ) = - 2 ( Z , Z * ) I d f o r a l l Z , Z * G Z . The 2-step nilpotent groups [N, { , )} of Heisenberg type may be regarded as the model spaces for the class of 2-step nilpotent groups {TV, ( , )} with a left invariant metric. The groups of Heisenberg type have especially large groups of isometries and have a special status analogous to that of the Riemannian symmetric spaces in the class of Riemannian manifolds. See [Kl,2] as well as the discussion below in Example 4 of (2.11).

The Heisenberg and quatemionic Heisenberg algebras equipped with a natural inner product become 2-step nilpotent Lie algebras of Heisenberg type. In general, for any positive integer m there exist infinitely many nonisometric Lie algebras {At, ( , )} of Heisenberg type whose centers Z have dimension m. See [K2], p. 36.

We now define natural inner products on the Heisenberg and quatemionic Heisenberg Lie algebras and describe the maps {j (Z) : Z G 2} in each case.

Example 1. - Let JV = V 0 Z be the Heisenberg algebra of dimension 2 n + 1, where as above V = span{Xi, Vi, . . . , Xn, Yn} and Z = span{Z}. Identify V with C71 as follows: if zj = aj + \^i /3j G C, where aj, (3j; C R, then identify

n

(^i, . . . , Zn) G C" with V {aj Xj + f3j Yj} G V. Give Af the inner product such that the vectors {X^, Y^, Z : 1 ^ %, j ^ n} form an orthonormal basis. With these identificationsj=i it follows that for any real number a,

j(aZ)(^i, ..., Zn) = (aV^I)^!, . . . , Zn) = (aV^l^i, . . . , aV^izn).

Clearly j'(aZ)2 = -a2 Id == - | a Z | 2 l d .

Example 2. - Let A/" = V 9 ^ be the quatemionic Heisenberg algebra of dimension 4 n + 3, and let [X,, Y^ V^ W, : 1 ^ i ^ n} and {Zi, Z^ Z^} be the bases of V, Z defined above. Give J\f the inner product such that these basis vectors form an orthonormal basis for Af.

Generalizing the previous example, we show that the action on V of a map j (Z), Z G Z^

corresponds to left quaternion multiplication on H77' w V by a purely imaginary quaternion ([K2], p. 39).

ANNALES SCIENTIFIQUES DE L'ECOLE NORMALE SUPERIEURE

Recall that H is R4 with a basis {1, %, j, k} and noncommutative multiplication given by ij -==. —ji •==. fc; jk = —kj = % ; & % = = —%fc = j and %2 = j2 == fc2 = —1. Identify V with IH"' as follows: if hr = Or -h /3r i + 7rJ^ + ^r ^? 1 ^ r ^ n, then identify (/ii, . . . , hn)

n 3

with ^ {a^ Xy. + A Yr + 7r ^r + Sr Wr}. If Z = ^ c^ ^ is any element of Z, then it

r=l s=l

is routine to show that j (Z) (/ii, . . . , hn) = £, • (^i, . . . , ^n) = (^ /ii, . . . , <^ hn), where

^ = d i + C2j + C3 k <E H. It follows that j (Z)2 = -(c2 + c| + cj) Id = -| Z |2 Id for all Z (E Z.

We conclude this section with a useful characterization of nonsingular 2-step nilpotent Lie algebras.

(1.8) LEMMA. - Let AT be a 2-step nilpotent Lie algebra. Then J\f is nonsingular if and

on^ if f01" ^Y positive definite inner product { , ) on J\T the maps {j (Z) : Z G Z} are nonsingular on V = Z¹' for every nonzero Z G Z.

Proof. - If Z G Z and X € V are any elements, then it follows from the definitions that j (Z) X = 0 if and only if Z is orthogonal to ad X (A/") = [X, At]. D

2. Geometry of 2-step nilpotent groups with a left invariant metric

Let {A^, ( , )} denote a 2-step nilpotent Lie algebra with a positive definite inner product.

Let {TV, ( , )} denote the simply connected 2-step nilpotent group N with Lie algebra M and left invariant metric { , ); that is, the left translations Ln, n G N, are isometries of {TV, ( , )}. In this section we derive some basic formulas for the curvature and Ricci tensors of N. We also give examples of complete, totally geodesic submanifolds of TV.

In this section and in the sequel we shall sometimes regard the elements X of J\T as left invariant vector fields on N determined by their values at the identity e of TV.

Covariant derivative and curvature ([CE], p. 64).

If X, y are elements of M regarded as left invariant vector fields on TV, then the function n —> ( X (n), y ( n ) ) is a constant function on TV. The formula for the covariant derivative Vjc Y of smooth vector fields on a Riemannian manifold normally contains 6 terms (cf. [Hel], p. 48), but in this case 3 of them vanish since (X, Y ) is constant. One obtains

(2.1) V x V = \ {[X, V] - (adX)* (V) - (adV)* (X)}

where (ad X)*, (ad V)* denote the adjoints of ad X, ad Y. We may regard V as a bilinear mapping from A/" x A/" into At since V^ Y is a left invariant vector field if X, Y are left invariant vector fields.

From (2.1) one obtains routinely

(

^ -<

^ b) Vx Z = Vz X = -- j (Z) X for all X G V, Z € Z , c) Vz Z* = 0 for all Z, Z* C 2'.

40 SERIE - TOME 27 - 1994 - N° 5

Curvature tensor.

If $i? ^2? $3 are vector fields on At then recall that the curvature tensor is given by

R^i. 6)^3 = -v^,] ^3 + v^ (v^ 6) - v^ (v^ ^3).

If $1^2, $3 are left invariant vector fields, then R (^i, ^2) ^3 is also left invariant, and we may regard R as a multilinear map from Af x J\T x J\f to AT. From (2.2) we obtain

a) R (x, Y) x* = | j ([x, y]) x* - ^ ([y, x*]) x + ^ ([x, x*]) y

for all X, r, X* G V. In particular

^(x,y)x=^([x,y])x

b) R(X, Z ) y = -l[ X , J ( Z ) y ] forallX, V G V, all Z £ Z

(2.3) . R (x, Y) z = - ^ [x, j (z) y] + ^ [v, j (z) x]

c) ^ ( X , Z ) Z * = - ^ { j ( Z ) o j ( Z * ) X }

J Z ( Z , Z * ) X = - ^ { j ( Z * ) o j ( Z ) X } + ^ { j ( Z ) o j ( Z * ) X } for all X G V, all Z, Z* G 2^

d) ^ (Zi, ^2) ^ 3 = 0 for all Zi, ^2, ^3 e Z

The entire curvature tensor can be computed from these formulas and the Bianchi identities.

Sectional Curvature.

Let II <= Tn N be a 2-dimensional subspace, and let X, Y be orthonormal elements of A/" such that span{X(n), Y(n)} = n. The sectional curvature K (II) equals AT(X, V) = <J?(X, y)V, X ) . From (2.3) we immediately obtain

' a) If X, y are orthonormal elements of V, then

iW Y) = - 1 1 [x, y] |

(2.4) < b) If X G V and Z G Z are orthonormal, then K ( X ^ Z ) = - ^ \ j ( Z ) X \2

[c) If Z, Z* are orthonormal elements of Z, then ^ (Z, Z*) = 0.

7?rcri tensor.

For arbitrary elements X, Y of A/" we recall that the Ricci tensor of N is given by Ric (X, V) = trace {^ -^ R (^ X) V, ^ G A/"). Symmetries of the curvature tensor imply

ANNALES SCIENTIFIQUES DE L'fiCOLE NORMALE SUPfiRIEURE

that Ric is a symmetric, bilinear form on J\f x A/", and hence there exists a symmetric linear transformation T : Af -^ At such that Ric (^i, 6) = < ^1^2 ) for all ^i, ^2 ^ A^.

(2.5) PROPOSITION. - a) Ric (X, Z) = 0 /or ^ZZ X G V, Z G Z.

In particular T leaves V and Z invariant

b) If {Zi, . . . , Z-m} is an orthonormal basis of Z, then

. m

^v-^E^) A;=i 2 '

In particular, T|v /5' negative definite, and

Ric (X, X) < 0 for all nonzero X G V.

c) Ric (Z, Z*) = - , rmc6? { j (Z) o j (Z*)} /or all Z, Z* G 2. In particular T \z is positive semidefinite. The kernel ofT in M = [Z G Z : j (Z) = 0} = [Z G Z : Z is orthogonal to [At, A/]}.

(2.6) COROLLARY. - (c/ [Kl], p. 134.) - Let {N, { , ) } be a simply connected 2-step nilpotent Lie group with a left invariant metric. Then the left invariant distributions V and

Z in N are left invariant by every isometry of N.

Proof of the Corollary. - At any point n of N the distribution V is the subspace of Tn N spanned by the eigenvectors of the Ricci transformation T corresponding to negative eigenvalues of T. The distribution Z is similarly described by the nonnegative eigenvalues of T. D

Proof of the Proposition. - Assertions a) and c) follow routinely from (2.3). We prove b). Let X, V be arbitrary elements of V, and let {Vi, . . . , KJ and {Zi, . . . , Zm} be orthonormal bases of V and Z respectively. From (2.3), the skew symmetry ofj ( Z k ) and

m

the fact that [V,, X} = ^ {j (Zj,) V,, X) Zj, we obtain

k=l

n o n ^ m

(*) ^{R(V,,X)Y,Vi}=-^{j([Vi,X])Vi,Y}= - ^ { j ( Z ^ X , Y ) } .

%=1 1=1 fc=l

By (2.3)

^ ( R ( Z ^ X ) Y ^ Z , ) = -l^ ( j { Z , )2X ^ Y ) )

k=l k=l

and b) now follows from (^). D Euclidean de Rham factor of N.

The next result explains the geometric significance of the nullity of the Ricci tensor.

(2.7) PROPOSITION. - Let {TV, ( , )} be a simply connected 2-step nilpotent Lie group -with a left invariant metric. IfAf denotes the Lie algebra ofN let £ = {Z G Z : j (Z) = 0}, and let A/"* denote the orthogonal complement of£ in At relative to { , ). Then

4'^ sfiRffi - TOME 27 - 1994 - N° 5

1) £ and A/"* are commuting ideals in A/\ and N is the direct product of the subgroups TV* = exp(A/"*) and E = exp(f).

2) N is isometric to the Riemannian product of the totally geodesic submanifolds TV*, £', and E is the Euclidean de Rham factor of N.

Proof. - Assertion 1) follows immediately from the definition of £. We prove 2). Let

<f, .V* also denote the left invariant distributions in N determined by the subspaces <f, A/"*

of AV = TeN. The subgroups TV* = exp(A/"*) and E = exp (<?) are maximal integral manifolds of .V*, £.

The distributions A/"*, £ are not integrable but parallel (i.e. invariant under parallel translation along arbitrary curves). It suffices to verify this for £ since the orthogonal complement of a parallel distribution is also parallel. From the definition of £ and (2.2) we see that V^ ^ = 0 if ^ e £ and ^* C A/", and hence £ is parallel. It now follows from the de Rham theorem (see for example Theorem 6.1 of [KN], p. 187) that N is isometric to the Riemannian product TV* x E. The totally geodesic submanifold E is flat by (2.3) and the fact that £ C Z. If Z is a nonzero element of ./V* H Z, then j (Z) ^ 0 by the definition of <? and A/"*. The subgroup TV* is totally geodesic [cf. (2.9) below], and the Ricci tensor of TV* is nondegenerate by (2.5c). In particular TV* has no Euclidean de Rham factor, and we conclude that E is the Euclidean de Rham factor of TV. D

Isometry group of TV.

(2.8) PROPOSITION. - Let {TV, (, )} be a simply connected, nilpotent Lie group -with a left invariant metric, and let I (N) denote the isometry group of TV. Let A (TV) = I (TV) D Aut (TV), -where Aut (TV) denotes the automorphism group of TV. Let TV also denote the subgroup of I {N) consisting of left translations by elements of N. Then TV is a normal subgroup of I (TV); TV U A (TV) = {e} and I (TV) = TV • A (TV) = A (TV) • TV.

Proof. - See Theorem 4.2 of [Wo2] and Theorem 2 of [Wi]. A simple direct proof of this result for 2-step nilpotent groups of Heisenberg type can be found in [Kl], and this proof is valid without change in the general 2-step nilpotent case in view of Corollary (2.6) above. D

Totally geodesic submanifolds and subgroups.

Let {TV, (, )} be a simply connected 2-step nilpotent Lie group with a left invariant metric. A connected subgroup TV* of TV with Lie algebra A^* C J\f is a totally geodesic submanifold of TV if and only if it is totally geodesic at the identity; left translations by elements of TV* are isometries of TV that leave TV* invariant. Hence

(2.9) A connected subgroup TV* of TV is a totally geodesic submanifold of TV if and only if V^ ^2 ^ A/"* whenever ^i, ^2 ^ •A/'*» where A/"* C J\f is the Lie algebra of TV*.

(2.10) DEFINITION. - A Lie algebra A/"* C A/" is totally geodesic if V^ ^2 € A/'* whenever

^2 C M\

(2.11) EXAMPLES OF TOTALLY GEODESIC SUBGROUPS.- A complete, connected totally geodesic submanifold of TV need not be a connected, totally geodesic subgroup of TV, but a 2-step group {TV, (, )} admits many totally geodesic subgroups. Many of these are

ANNALES SCIENTIFIQUES DE L'ECOLE NORMALE SUP^RIEURE

flat, which is not too surprising since {TV, (, )} should be similar in some sense to flat Euclidean space. We list some basic examples.

Example 1. - Let $ G M be arbitrary. The 1-parameter subgroup exp {t^) is a geodesic of N if and only if (ad$)* ^ = 0 by (2.1). This condition holds if and only if ^ is orthogonal to [^ A/]. In particular if $ € V or ^ G Z, then ^ -^ exp (^) is a geodesic of N that starts at the identity. These are the only possibilities for a 1-parameter subgroup to be a geodesic if Af is nonsingular.

Example 2. - Let A/'* be an abelian subspace of V; that is, [X, Y] = 0 for all X, V G A/\

Then A/"* is a totally geodesic subalgebra in view of (2.1) since Vjc Y = - [X, Y] for all X, Y G V. Moreover, AT (X, V) = 0 for all X, Y e A/"* by (2.4). Hence TV* = exp (A/'*) is a complete, flat, totally geodesic subgroup of TV that contains the identity.

This example generalizes a part of the first example. Abelian subspaces A/"* of V of dimension at least 2 arise whenever dimV ^ 2 + dimZ. In such a case the map adX:

V -^ J7 has a kernel of dimension at least 2 for every nonzero X G V, and we define A/"* = span {X, Xi}, where Xi is any element of ker (ad X) that is not collinear with X.

More generally, if dim V ^ 1 + r + r dim Z, for some integer r ^ 2, then every nonzero element X of V lies in an abelian subspace A/"* of V of dimension r+1. One may construct A/"* as follows. Define Vo = ker (ad X), a subspace of V of dimension ^ v - z ^ 2, where v = dim(V) and z = dim(Z). Let Xi be any nonzero element of Vo that is linearly independent from X and define Vi = Vo D ker (adXi). Continuing in this fashion we let Xj be any nonzero element of Vj_i that is linearly independent from {X, X i , . . . , Xj_i}

and define Vj = Vj-i H ker (adXj). If 1 ^ j ^ r, then this construction is possible since dimVj-i ^ v - jz ^ v - rz ^ r + 1 ^ j + 1. Hence Xy. exists, and by construction the subspace A/"* = span {X, X i , . . . , Xy.} is abelian and has dimension r + 1.

Example 3. -If Z denotes the center of N, then it follows easily from (2.2) and (2.3) that the orbits of Z in TV under left multiplication are complete, flat, totally geodesic submanifolds of TV.

If TV is of Heisenberg type, then one can find many additional totally geodesic submanifolds of TV.

Example 4. - Let {TV, {, )} be a 2-step nilpotent Lie group of Heisenberg type. Then every unit speed geodesic 7 of TV is contained in a complete, 3-dimensional totally geodesic subgroup TV* of TV. After rescaling the metric of TV* by a positive constant depending on the geodesic 7, the group {TV, (, )} is isometric to the 3-dimensional Heisenberg group corresponding to the 3-dimensional Heisenberg algebra {A/\ (, }} constructed above in section 1.

We verify the assertions of the example above. It suffices to consider the case that the geodesic 7 satisfies 7 (0) = e, the identity of TV. Let 7' (0) = Xo + Zo, where Xo G V and ZQ e Z. We first consider the case that Xo and ZQ are both nonzero.

Let A/"* = span{Xo, ZQ, j (Zo) Xo} and let TV* = exp (A/"*). We assert that A/'* is a 3-dimensional, totally geodesic subalgebra of A/", and it will then follow that TV* is a complete, 3-dimensional, totally geodesic subgroup of TV that contains 7.

4e SERIE - TOME 27 - 1994 - N° 5

From (1.7) we obtain the following

LEMMA. - Let N be a 2-step nilpotent group of Heisenberg type. Then [X, j{Z)X}= I X I ² Z for all elements X <E V, Z G Z.

It follows from the lemma that the subspace A/"* defined above is a subalgebra of A/". From (2.2) we see that A/"* contains V^o Zo = Vzo x^ v^ J (^o) -^o = -Vj (Zo)Xo -^o and Vz, j (Zo) ^o = Vj (Zo) Xo ^o. It follows that V^ ^2 e A/"* for all ^i, ^ e A/"*, and hence A/"* is a totally geodesic subalgebra of A/", provided that both Xo and Zo are nonzero. If y (0) = Xo + Zo, where either Xo or Zo is zero.then 7' (0) lies in infinitely many totally geodesic subalgebras of the type A/"* above. The group TV* = exp (A/"*) is of Heisenberg type since A/"* H V is invariant under j (Zo). It is easy to see that there is a unique 3-dimensional, 2-step Lie algebra of Heisenberg type up to isometry and multiplication of the metric by a positive constant. This completes the discussion of Example 4.

Remark. - The property of example 4 characterizes groups of Heisenberg type in the class of all 2-step, simply connected nilpotent Lie groups N with a left invariant metric ( , ). More precisely we have

(2.12) THEOREM. - Let {N, { , )} be a simply connected 2-step nilpotent Lie group with a left invariant metric. Suppose that every unit speed geodesic through the identity e of N is tangent to at least one 3-dimensional totally geodesic submanifold of N that intersects the center Z in a submanifold of positive dimension at e. Then N is of Heisenberg type if one replaces { , ) by c² ( , ) for a suitable positive constant c.

We omit the proof, which will appear in [El].

3. Geodesies

To describe the geodesies of {N, ( , }} it suffices to describe those geodesies that begin at the identity of N. Let 7 (t) be a curve with 7 (0) = e, and let 7' (0) = Xo + Zo G A/", where Xo e V and Zo G Z. In exponential coordinates we write

7 ^ ) = e x p ( X ( t ) ) + Z ( t ) , where X (t) G V, Z (t) C Z for all t and

X'{Q}=X^ ZfW=Zo

(3.1) PROPOSITION. - The curve 7 (t) is a geodesic if and only if the following equations are satisfied:

a) X" (t) = j {Zo) X ' {t} for all t G R b) Z ' {t) + 1 [X' (t), X (t)\ =E Zo for all t G R

z^

Proof. - These equations were derived by A. Kaplan in [Kl] to study 2-step nilpotent groups N of Heisenberg type, but the proof is valid without change in the general 2-step nilpotent case. These equations can be completely integrated if N is of Heisenberg type (cf. (3.8) below, [Kl, 2] and [Ko]). In the general 2-step case the equations can also be integrated but the answer involves the eigenvalues of j (Zo) as we show in (3.5).

ANNALES SCIENTIFIQUES DE L'ECOLE NORMALE SUPERIEURE

The next result will show that the geodesic flow in {N, (, )} has dimZ linearly independent first integrals, a fact reminiscent of the geodesic flow in flat Euclidean spaces.

(3.2) PROPOSITION. - Let {N, (, )} be a simply connected 2-step nilpotent Lie group with a left invariant metric, and let 7 (t) be a geodesic ofN with 7 (0) = e. Write 7' (0) = XQ + ZQ, where XQ C V C M, ZQ G Z C M and Af = Te N. Then

Y (t) = dL^ (,) (e^'(zo) Xo + Zo) for all t G R

00

where e^'^0) = ^ ( ^ j (Zo)")/^!

71=0

Pwo/. - Write 7 (^) = exp (X (t) + Z (t)), where Z (t) e V, and Z (^) G 2 for all t C R. Using the result and the notation of (1.3) and the second equation of (3.1) we obtain

7' {t) = dexp^^z^ {X' {f} + Z ' %)x(t)+z(t)

= dL,^ ^ + Z ' + J [{X' + Z'), (X + Z)}\

= d£^) (x7 + Z ' + ^ [X7, X]') = rf£^) (Z' + Zo).

By integrating the first equation of (3.1) we obtain X' (t) = g^'^o) ^ which completes the proof. D

(3.3) COROLLARY. - Let {^} denote the geodesic flow in TN. Let n G N and XQ, ZQ G At = Te N be given, where XQ G V and ZQ e Z. Then

g^t {dL^Xo + Zo)) = ri^d) (e^^ XQ + Zo), where 7 {t) is the unique geodesic with 7' (0) = dLn (XQ + Zo).

Proof. - Straightforward. D

(3.4) COROLLARY. - Define f : TN ^ Z by f (dLn X) = H^ X, where T^z : At -^ Z is the projection map and n G N, X e Af = Te N are arbitrary. Then f o (^ = f for all t G R. If F C TV is any discrete subgroup acting on N by left translations, then f induces a function F : T (T\N) -^ Z such that F o g^t = F for all t <E R.

Proof. - These assertions follow routinely from (3.3). D

By choosing a basis for Z we can define m = dim Z linearly independent first integrals from the Z-valued first integrals /, F above. Alternatively, [Ba] if F C TV is any discrete subgroup, then each nonzero element Z of Z defines a Noether first integral (cf. [A], pp. 88-91) h: S(T\N) —^ R as follows: Extend Z to a biinvariant vector field on N and define H : SN —^ R by H (v) = (v, Z{p{v))}, where p : SN —^ N is the projection map. The flow transformations of Z are isometries of N, and hence the restriction of Z to any geodesic 7^ of N is a Jacobi vector field of constant length. It follows that for each v G SN the function H{g^t v) = (^ {t), Z (7^ t)) is a bounded affine linear function on R and must therefore be constant. The function H is invariant under dL^ for all n G N, and hence H induces a first integral h: S {T\N) -^ R for any discrete subgroup F of N.

46 sfiRffi - TOME 27 - 1994 - N° 5

Integration of the geodesic equations.

We give a solution to the equations (3.1), but the equations obtained are expressed in terms of eigenvalues of the transformation j {Zo) as well as the initial data Xo, Zo. These equations simplify if [N, ( , )} is nonsingular, especially if {TV, ( , )} is of Heisenberg type, but do not simplify further in the general case.

Again, it suffices to consider the case that 7 is a geodesic of N with 7 (0) = e. We write 77 (0) = Xo + Zo, where Xo e V and Zo ^ Z. Let J : V -^ V denote the skew symmetric transformation j (Zo)» and write V as a direct sum Vi © 1^ where Vi is the kernel of J and ^2 is the orthogonal complement of Vi in V. Note that J leaves ^2 invariant and is nonsingular on V'z.

Let {% 0i, -z 0 i , . . . , i O N , -i 07v } be the distinct nonzero eigenvalues of J, where 0^ > 0

N

for all %, and write V^ as an orthogonal direct sum (j) Wp where J leaves invariant each j=i

W, and J2 = -6^Id on Wj. We write

Xo = Xi + X2, where X, G Vi for j = 1, 2.

N

X2 = ^ ^-, where ^ (E W, for 1 ^ j ^ N j=i

(3.5) PROPOSITION. - Let {TV, ( , )} be a simply connected 2-step nilpotent Lie group with a left invariant metric. Let 7 (t) be a geodesic with 7 (0) = e. Write 7' (0) = XQ + ZQ, where Xo G V and ZQ G Z. Write 7 (t) = exp (X (t) + ^ (^)), w^^ X (^) G V and Z {t) G Zfor all t G R ^mrf X' (0) = Xo, Z ' (0) = Zo- T^i with respect to the notation above we have

1 ) X ( t ) = t X i + ^ - I d ^ -¹^ ) 2) Z(T) = ^Zi + ^2 (^), where

1 1 ^N

a) Zi % = Zo + 3 [^i, (^J + Id) (J-1 X2)] + ^ [J-1 ^, ^-]

j=i

b) Z^ (t] is a function of uniformly bounded absolute value given by Z2 {t) = [Xi, (Id - e^) (J-² X2)] + ¹ [e^ J-¹ X^ J-¹ X^\

Zt

1 N

-- E ^ / ( ^ - ^ { ^ ^ ^ e ^ J -¹^ - ^ ^ , , ^⁷^ } ' W=i

+| E { i ^l /(^ ² - ^0? )}{[ ^l7 ^- ^J " ^l ^-K-^}

^j=i

Proof. - We verify that the expressions for X (t) and Z (^) given above satisfy the equations in (3.1) together with the initial conditions X (0) = Z(0) = 0; X'(0) == Xo and Z ' (ff) == Zo- First we note

a) J commutes with eⁱ³ for all i e R and -,-(e^tJ} = Je^ = e^tJ J . dt

ANNALES SCIENTIFIQUES DE L'fiCOLE NORMALE SUP^RIEURE

b) J² = -9] Id and J == -6^ J-¹ on W, for each l ^ j ^ N .

It is now straightforward to verify that X (t) satisfies the first equation of (3.1) and the initial conditions X (0) = 0 and X' (0) = Xo. Moreover it is routine to show

c) [X' (^ X (t)} = [Xi, (e^ - Id) (J-1 ^2)] + t [e^X^ X,}

+ [ et JZ 2 , ( et t 7- I d ) ( J -lX 2 ) ]

Next we observe that the derivative of [e*17^, e^J"1^] is zero by a) and b) for all t e R and 1 $ j^ $ TV. Hence we obtain

rf) [e^ ^ e^ J"1 ^] = [Q, J~1 ^j] for all ^ <E R and 1 ^ j ^ N

N

From rf) and the fact that X^ = V^ we obtain j==i

.U^^e^J-1^]^ E [e t^-e t'l 7 - l^+E^'J - l^

i^==l j = l

From a) and Z?) we obtain

1 1 ^

y/ /-i.\ _ r y ,,tJ T—l v 1 i \otJ\r 7-1 V 1 Y^ \^tJ /- _ t J 7-! ^ i

^ ( ^ - " [ ^ l ^6 J ^2] -+- ^ [e A 2 , J 2J ~ 9 / . ^ ^5 e l/ ^J z^'=l

Combining this with e) we obtain

/) Z, {t) = -[X,, e^ J-¹ X,\ - J [e^Xa, (e*⁷ -Id) (J-¹ X,)} - J ^ [J-¹ ^ ^]

J=l

Next we compute Z ' (t) = Zi (t) + ^ Z^ (t) + Z^ (t) and from f) we obtain

^ ^ ( t ) = Z o - j [e^^^^J-¹^]

-J [Zi, e^ - Id) (J-1^)] + J [e^X^ J-1^] - J ^[6^X2, Xi]

Finally from c) and g) we see that Z' (t) + , [X7 (^), X (t)] ^ Zo, which is the second equation of (3.1). It is evident that Z (0) = 0 and from g) we see that Z ' (0) = Zo. D

(3.6) Remark. - Using assertion b) above it follows that

e^{t J}= c o s ( t ^ • ) I d + { s m ^ ^ • ) / ^ • } J on W^ ^ ^ J ^ N . From this it follows that Z^ (t) is uniformly bounded in absolute value for all t € R.

N

(3.7) Remark. - Using (3.6) and the fact that X^ = V^ ^ we can rewrite the equation j=i

for X {t} in (3.5) in the following form:

N

X(t)=tX^^X^{t)^

^l

4° SERIE - TOME 27 - 1994 - N° 5

where

X] (t) = (e^ - Id) (J-1 ^) = {cos(^) - 1} J-1^. + {sm(^)/^K,.

Each curve X* (t) is a circle in Wj with center —J~1 ^, radius | J~1 ^ | and period —.

^ (3.8) Remark. - If the Lie algebra At is nonsingular [cf. (1.4)] and if Zo / 0, then J = j'(Zo) is invertible by (1.8), and it follows that Xi = 0 and X^ = XQ in the notation of (3.5). The equations of (3.5) simplify in this case. If in addition N is of Heisenberg type, then i \ Zo \ and -i ZQ \ are the only eigenvalues of J, and with the help of (3.6) and (3.7) the equations in (3.5) become

1) X (t) = (e^-ld) (J-1 Xo) = {cos(t Zo I ) - 1} J~1 Xo+{sin (t \ Zo I ) / I Zo \} XQ 2) Z (t) == tZ^ (t) + Z2 (t), where

a) Z, {t) EE Zo + ^ [J~1 Xo, Xo] = (l + l x o 1 2) Zo by (1.7) and the lemma in

2 \ 2 Zo | /

example 4 of (2.11).

b ) Z ^ ( t ) = J [ e ^ J ^ X o ^ J ^ X o ] = J { s m ^ l Z o D / I Z o l H X o . J ^ X o ] = r - sin (t | Zo | ) | Xo | -i ^ ^ ^ ^ ^ ^ lemma in example 4 of (2.11).

I z j Z/Q -J

I "V

- K T - . L - ^ 1 - _ ^ 17- /jA • - - - • - - 1 - _ - ' ^ 1 - --—^.-.- T — 1 \r ^ - - l ' - - I T — 1 V I I 0

Note that X (t) is a circle with center J 1 XQ, radius | J 1 XQ\ = and period I ZQ

-. Moreover Z (t) is a multiple of Zo for all ^ G R.

Zo I

(3.9) Remark. - If Xo = 0 or Zo = 0» then the geodesic equations in (3.1) or (3.5) become respectively ^{t) == exp(tZo) or 7(^) = exp(tXo). More generally, it follows from the equations in (3.1) that if 7 {t) is the unique geodesic with 7' (0) = XQ + Zo, then 7 (t) = exp (t (Xo + Zo)) if and only if j {Zo) Xo = 0 if and only if Zo is orthogonal to

[Xo, At}. This fact can also be deduced from (2.1).

We conclude this section with two results about the behavior of geodesies in N that are tangent to the left invariant distributions V and Z in N.

(3.10) PROPOSITION. - Let {N, ( , )} be a simply connected 2-step nilpotent Lie group with a left invariant metric. Let 7 (t) be a unit speed geodesic of the form 7 (t) = a • exp (t Z*),

"where a G N is arbitrary and Z* is any unit vector in Z. Then J (Z*) =: 0 if and only if no two points 0/7 are conjugate. Ifj (Z*) ^ 0, then 7 (0) is conjugate to 7 (6) for some b > 0.

Remark. -If N has no Euclidean factor, then j (Z*) ^ 0 if Z* / 0 by (2.7).

(3.11) PROPOSITION. - Let {TV, ( , )} be a simply connected 2-step nilpotent Lie group with a left invariant metric. Let V* be a nonzero element ofV, and let d ( , ) denote the left invariant metric of N. Then d (e, exp (V* + Z*)) ^ | V* | for all Z* G Z "with equality if and only if Z * = 0. Hence if^y (t) is a unit speed geodesic of the form 7 (t) = a • exp (t V*), where a G N is arbitrary and V* is any unit vector in V, then 7 minimises the distance between any two of its points.

Proof of Proposition 3.10. - Since left translations are isometries it suffices to consider the case that a = e and 7 (t) = exp (t Z*), where Z* is a unit vector in Z. If j (Z*) ^ 0,

ANNALES SCIENTIFIQUES DE L'^COLE NORMALE SUPERIEURE

then the sectional curvature formulas in (2.4) imply that the sectional curvature of any 2-plane containing 7' (t) is zero for any t G R. Standard arguments then show that no two points of 7 are conjugate.

Conversely, if j ( Z * ) ^ 0, then by (2.5) Ric(Z*) = Ric(Y(0) = c > 0. It follows that Ric^'^t)) = c > 0 since 7' (t) = d^exp(tz*) V (0)» which proves that 7(0) and 7(6) are conjugate for some number b > 0 (see Theorem 1.26 of [CE]). This completes the proof of Proposition (3.10). D

Proof of Proposition 3.11. - Let V* G V and Z* G Z be arbitrary elements, where V* / 0. Let 7: [0, 1] -^ N be a shortest geodesic from e = 7 (0) to exp (V*+Z*) = 7 (1), and let 7' (0) = XQ+ZO, where Xo G V and Zo ^ ^. We write 7 (^) = exp {X (t) + Z (t)), where X (t) e V and Z (t) G ^ and X (0) = Z (0) = 0. By (1.3) or (3.2) and the geodesic equations in Proposition 3.1 we see that

Y W = dL^^ ( x ' + Z ' + J [X', X]') = d^(,) [X' + Zo).

Hence

d ( e , e x p ( V * + Z * ) ) = / | 7'M ^ = /l ( | X ' (t) 2 + \ Zo\2)1/2 dt Jo Jo

^ ( \X'(t}\ dt^d(0, V*) Jo

= | V* | since Z (0) = 0 and X (1) = V\

It is now routine to complete the proof. D

4. Isometry invariant geodesies

Let {TV, ( , )} be a simply connected 2-step nilpotent Lie group with a left invariant metric, and let (f) be an arbitrary nonidentity element of N . We say that (f) translates a unit speed geodesic 7 (t) in N by an amount uj if (/) • 7 (t) = 7 {t + ci;) for all t G R.

The number c<; is called a period of <^. If (f) belongs to a discrete group F C TV, then the periods of (/) are precisely the lengths of the closed geodesies in r\7V that belong to the free homotopy class of closed curves in r\7V determined by (/). Elements of F that are conjugate in r have the same periods and determine the same free homotopy classes in r\7V.

In this section we show that every nonidentity element (f) in N translates some geodesic of N. Moreover (f) has both a minimal and a maximal period, which coincide if J\f is nonsingular and (/) does not lie in the center of N. For each period uj of (f) let N^ {(/)) denote the union of all unit speed geodesies of TV that are translated an amount uj by (f). Let SN^ (0) denote the set of unit vectors in N that are tangent to a unit speed geodesic of TV that is translated an amount uj by (f). Each set N^ (<^) is invariant under Z (^>) = [^ e N : (f)^ == '0^)}, and N^ ((f)) is a single Z (^) orbit if and only if cc; is the maximal period ^ of (f). Moreover, for any period uj of <f) the dimension of the set SN^ (cf)) is at least equal to the dimension ofSN^ ((/)) with equality if and only if uj = a;*.

46 SfiRIE - TOME 27 - 1994 - N° 5

By the dimension of a set we mean the largest integer k such that the set contains an imbedded open fc-disk.

Remark. - Some of the results of this section were obtained earlier by C. Gordon [G3]. In particular she essentially obtained the first three equivalences of Proposition 4.3 and showed (in the notation of Proposition 4.5) that (f) = exp(V* + Z*) translates the 1-parameter group t —^ exp ( — (V* + Z**) ) .

\^* /

(4.1) PROPOSITION. - Let (j) G N be a nonidentity element, and let d^ : N —>• R be the displacement function defined by d^ (n) == d(n, (f)n). Then d^ assumes a minimum value uj > 0 on N, and (f) translates some unit speed geodesic 7 of N by an amount uj. The number uj is the smallest period of (f).

Proof. - Choose V* C V and Z* G ^ s o t h a t ^ > = e x p ( y * + Z * ) . I f Z y * = { e x p [ V * , ^]:

^ G A/'}, then Zy* is a closed subgroup of Z that equals the identity if and only if V* = 0.

Therefore the set (f) • Zy* = Zy* ' (/) is closed in N , and we may choose an element '0* e (f)'Zv- such that d (e, ^*) ^ d (e, ^) for all ^ G ^ ' Z y . . We assert that uj = d (e, '0*) is the minimum value of d^ and that d^ assumes its minimum value at exp(^*), where

^* G Af is any element such that '0* = (/) • exp ([V*, f]) = exp (V* + Z* + [V*, <^*]).

Moreover, 0 translates any minimizing geodesic from ^* to 0^*.

Let ^* G A/' be chosen as above. If ^ G .V is arbitrary, then ^ = exp(—^) • (f) ' exp (0 = exp(V* + Z* + [V*, ^]) = ^ • exp ([V*, ^]) G ^ • Zy*. Hence ^ (exp^*) =

d(e, exp(-^*)-<^exp(^*)) = d ( e , ^*) ^ d(e, ^) = ^ (exp ^) by the choice of ^*. This proves that d^ assumes its minimum value uj = d(e, ^*) at exp^*. Standard arguments now show that uj is the smallest period of (f), and (f) translates any minimizing geodesic from exp($*) to (f) • exp(^*). D

Next we define a number a;* which later will turn out to be the maximal period of (f) = exp(V* + Z*).

(4.2) PROPOSITION. - Let (/) G N be an arbitrary element and write (f) = exp (V* + Z*) for suitable elements V* G V and Z* G Z. Let Z** te ^ component of Z* orthogonal to [V\ A/], 6mJ /^ a;* = { | V* | 2 + | Z** | 2}1/2 = | V* + Z** I . Let ^ G M be chosen so that Z** = Z* + [V*, ^], ^nd /^ 7 (t) = exp (Q • exp [^- (V* + Z * * ) ] . Then 7 (t) ^ a unit speed geodesic such that (f) • 7 (t) = 7(1 + c<;*) /or all t G 1R.

Pwo/. - If a = exp(0 we define ^ = a-1 • 0 • a = exp(V* + Z* + [V*, ^]) = exp (V* + Z**). The condition that (f) • 7 (t) = 7 (t + a;*) for all 1 G R is equivalent to the condition that ^ ' 7* (t) = 7* (t + cc;*) for all 1 e R, where 7* (t) = a~1 ' 7 (t) = exp ( — (V* + Z**) ). This latter condition is routine to verify.

V^* /

y* + ^**

Note that 7*7 (0) == ————— is a unit vector by the definition of cc;*. The condition that

CJ*

Z** be orthogonal to [V*, A/I is equivalent to the condition that j (Z**) V* = 0. It follows immediately that 7* (t) satisfies the geodesic equations in (3.1), and hence 7 (t) =• 0-7* (t) is a unit speed geodesic. See also (3.9). D

Now we describe some general criteria for an element (j) to translate a geodesic 7.

ANNALES SCIENTIFIQUES DE L'ECOLE NORMALE SUPERIEURE

(4.3) PROPOSITION. - Let (f) G N be an arbitrary element and write (f) = exp (V* + Z*) for suitable elements V* G V and Z* G Z. Let 7 (t) te a unit speed geodesic with 7 (0) = a and^ (w) = <f)'a, andlet^' (0) = dLa (Xo-^-Zo) for suitable elements XQ G V and Zo G Z.

Let J denote j (Zo). Let a~1 • 7 (t) = exp (X (f) + Z (^)), where X (t) G V 6mrf Z (t) G Z /or all t ^ R and X (0) == Z (0) = 0. Then the following assertions are equivalent.

i. x(t) +y* = x(t^uo)

Z {t) + Z* + ¹ [V*, X (t)] = Z (t + uj) for all t G R 2. ^ • 7 (^) = 7 (t + a;) for all t G R

3. 7' (0) is orthogonal to the orbit Zy* • a, where Zy* = exp ([V*, A/]) C Z 4. 7⁷ (c<;) is orthogonal to the orbit Zy* ' (f) ' a

5. e^⁷ fixes XQ

Proof. - By straightforward arguments similar to those used in the proof of (4.2) it suffices to consider the case that a = e, the identity in N. Assertions 1) and 2) are equivalent by the multiplication law (1.1). To prove the other equivalences we proceed in the cyclic order 2) => 3) => 4) ^ 5) => 2).

We prove 2) => 3). Write V as an orthogonal direct sum V = Vi (B ^2^ where Vi is the kernel of J = j {Zo). Write XQ = X^ + ^2, where Xi e Vi and X^ G ^2.

LEMMA. - V* = a; Xi 6md e^J fixes X^.

Suppose for the moment that the lemma has been proved. The lemma implies that J (V*) == 0, which is equivalent to Zo being orthogonal to [V*, A/] = TeZy*. Since Y (0) = XQ+ ZQ we conclude that 7' (0) is orthogonal to Zy* at e = 7 (0), which proves the assertion 2) =^ 3).

Proof of the lemma. - By Proposition 3.5 we have

(i) X (no;) = nc<;Xi + (e^⁷ - Id)(J~¹ ^2) for every positive integer n.

By induction we obtain from the equations above in 1) of the proposition (ii) X {nuj^ = nV* for every positive integer n.

If we write V* = V^ + V^, where V^ e Vi and V^ G V^, then from (i) and (ii) we obtain (iii) n V^ = n uj X^ for every positive integer n

(iv) nV^ = (e^^3 — Id) (J""1 X^) for every positive integer n.

The right hand side of (iv) is uniformly bounded in norm for all n since en^J is an orthogonal transformation. This implies that V^ == 0 and hence V* = V^ = ujX^ by (iii). From (iv) in the case n = 1 we see that euJJ fixes J~1 X^ and hence e^J fixes ^2 since e^17 commutes with J. D

We prove 3) => 4). Since 7' (0) = Xo + Zo the hypothesis 3) is equivalent to ZQ being orthogonal to [V*, A/"] = TeZy.. By Proposition 3.2 we see that 7' (o;) = dL^ („) (e" J Xo + Zo), where J = j (Zo). Now (^ = 7 (^) and ^ (Zy* . ^) = dL^, Tg (Zy*). Since Zo is orthogonal to Tg (Zy*) we conclude that 7' (a;) is orthogonal to T^ (Zy* • (f>). D

We prove 4) => 5). Since 7(0;) = (f) = exp(V* + Z*) it follows that Z* = Z (ci;) and V* = X (a;). Write V as an orthogonal direct sum V = Vi (B ^2, where Vi is the

4e SERIE - TOME 27 - 1994 - N° 5