A Fast Nilpotency Test

There are numerous characterizations of finite nilpotent groups (cf. Exer-cise 7.9), and quite a few of them can be utilized for testing whether a given per-mutation groupGis nilpotent. Here we present anilpotency test from [R´ak´oczi, 1995; 1997], which is extremely fast even in the case when log|G|is propor-tional to the permutation degree.

First, we describe an algorithm that decides whether a group G= S ≤ Sym(),with|| =n, is a p-group for agiven prime p. A possibility is to compute|G|by the algorithm of Section 7.1, but we can do better. A permutation group is ap-group if and only if all of its transitive constituents are p-groups;

hence we start with constructing the transitive constituents ofG. This amounts to computing the orbits ofGand can be done inO(|S|n) time. IfGhas an orbit such that||is not apower ofpthen we can conclude immediately thatG is not ap-group.

For the timing of the next theorem, recall that the inverse Ackerman function A⁻¹(m,n) was defined in Section 5.5.2.

Theorem 7.4.1. Let G = S ≤Sym()be transitive, with || = n = p^k for some prime p. It can be decided whether G is a p-group by a deterministic algorithm in O(|S|nlogn A⁻¹(2|S|n,n))time.

Proof. The algorithm is based on the following simple characterization of tran-sitivep-groups. A transitive groupG≤Sym(),with|| =p^kfor somek≥1, is a p-group if and only if

(i) Ghas a block system= {1, . . . , p};

(ii) Gacts onas the cyclic groupCp; a nd

(iii) ifG1 is the setwise stabilizer of1 inG then the restriction G1|1 is a p-group.

IfG satisfies (i) and (ii) thenG1 is the kernel of action on , sinceG acts regularly on. Hence|G| =p|G₁|and (iii) implies thatGis a p-group. Con-versely, Theorem 6.2.5 and Exercise 6.14 imply that the only primitivep-group isCp in its natural action on p points, so ifGis atransitive p-group then it has a block system satisfying (i) and (ii), and (iii) holds obviously. Therefore, we can test whetherG= Sis a p-group by checking whether it has a block system satisfying (i)–(iii).

The first step is asubroutine that constructs ablock system of size pifGis a p-group and returns a block system of size por{}ifGis not a p-group.

We use the version of block computations described in Section 5.5.2. To this

end, we need to construct some subset⊆,with|| ≥2, such that ifG is ap-group then there is a nontrivial block system with a block containing all points of.

One way of finding such a setis based on the observation that ifG is a p-group andg,h∈Gare not commuting then [g,h] fixes all blocks in any block system of sizepforG, and so any nontrivial orbit of [g,h] can be used as. Another way is to take a nontrivial elementgofZ(G). By Lemma6.1.7, the orbits ofgare permuted byG. Therefore, ifghas more than one orbit on then any nontrivial orbit ofgis suitable as, whereas ifghas only one orbit then any orbit ofg^psuffices.

Letgbe the first generator ofGgiven in the generating setS. We compute the commutator ofgwith all other elementsh ∈S. If one of these commutators is nontrivial then the first method described in the previous paragraph for finding can be applied, whereas if all commutators are equal to 1 theng∈Z(G) a nd the second method can be used. Applying the algorithm of Section 5.5.2, we construct ablock system^∗such that some block of^∗contains all points of . If^∗ = {}then we conclude thatGis not a p-group, and if|^∗|>pthen we construct the action ofG on^∗ and repeat the procedure recursively for this action. Eventually, we obtain a block systemforGsatisfying (i), or we terminate with the conclusion thatGis not ap-group.

Ifis constructed then the second step of the algorithm checks whether the action ofGonsatisfies (ii). Any element ofSacting nontrivially onmust define acyclic ordering of the blocks inand it is straightforward to check whether the other generators respect this cyclic ordering.

The third step of the algorithm is the construction of generators for G1|1. We find someg∈S acting nontrivially on and construct the bijec-tions ϕj:1→j and their inverses for j∈[2,p], defined by the powers g,g², . . . ,g^p−1ofg. These bijections can be used to construct the restrictions of the permutations in the set

T := {gⁱhg^−j|i ∈[0,p−1], j ∈[0,p−1], h ∈S, gⁱhg^−j ∈G1} to1. By Lemma4.2.1, these restrictions generateG1|1.

After that, the same procedure as we just described can be applied recursively to determine whetherG1|1is a p-group. Note thatG1|1acts transitively on 1, since for anyα, β ∈ 1, there existsx ∈ Gwithα^x =β and obviously x|1 ∈G₁|1.

Now we analyze the time requirement of the algorithm. In the first step of the algorithm, the set is computed in O(|S|n) time. By Corollary 5.5.9, the block system^∗ is obtained inO(|S|n A⁻¹(2|S|n,n)) time. If|^∗| > p

then further block computations are required; however, these recursive calls of block computations are on sets of size decreasing by at least a factorp, so they contribute only aconstant multiplier to the time complexity. The second step, checking the action on, is in O(|S|p) time. In the third step, the bijections ϕjand their inverses are constructed inO(n) total time, because for each of the n/pelementsα∈1 and for each j ∈ [2,p−1], ϕj+1(α) =ϕj(α)^g can be obtained in constant time fromϕj(α). The|S|pSchreier generators forG1|1

are computed inO(|S|n) total time, since we construct permutations only on a domain of degreen/p. Hence the total time requirement of the three steps is O(|S|n A⁻¹(2|S|n,n)).

The recursive call is for the permutation group G₁|1 of degree n/p with|S|p generators, so the three steps of the algorithm for G1|1 require O((|S|p)(n/p)A⁻¹(2|S|n,n/p))=O(|S|n A⁻¹(2|S|n,n)) time. Similarly, fur-ther recursive calls are on domains of degree decreasing by a factorpand with the number of generators increasing by a factor p, so the time requirement remains the same. Since the number of recursive calls is at most logn, the total time requirement of the algorithm isO(|S|nlogn A⁻¹(2|S|n,n)), as claimed.

The nilpotency test for permutation groups is based on the p-group test described in the proof of Theorem 7.4.1. A permutation groupG = S ≤ Sym(), with|| = n, is nilpotent if and only if all of its transitive con-stituents are nilpotent, so again we start with the construction of the transitive constituents, inO(|S|n) time. From now on, we suppose thatGis transitive.

Letpbe aprime divisor ofn. IfGis nilpotent thenG=P×Q, wherePis anontrivialp-group andQis anilpotent group of order not divisible byp. By Lemma6.1.7, the orbits ofPconstitute ablock system^∗forG. SincePis a p-group, the size of blocks in^∗isp^kfor somek≥1. Moreover, asPis in the kernel of the action ofGon^∗, the action ofGon^∗has order relative prime top. So, in particular, the divisor|^∗|of this group order is relative prime top andp^kis the largest power ofpthat dividesn. The orbits ofQconstitute another block system^∗∗forG. By the same argument we used for^∗, we obtain that

|^∗∗|is relative prime to each prime divisor of|Q|. Hence|^∗∗| = p^k. Finally, we describe how can we get generators forPandQ. For agenerator s∈S, let|s| = p^lrwith (p,r)=1. Thensp:=s^r∈Pandsq :=s^pl∈Q, a nd s∈ sp,sq. HenceP= sp|s∈SandQ= sq|s∈S. Note that|s|divides nsince obviously p^l ≤ p^k, and by induction on the number of different prime divisors of|Q|we can prove thatr dividesn/p^k.

Lemma 7.4.2. Let G = S ≤Sym(),with|| =n, be transitive, let p be a prime divisor of n, and let P:= sp|s∈S,Q:= sq|s∈Swith sp,sq as

defined in the previous paragraph. Let p^kbe the largest power of p dividing n.

Then G is nilpotent if and only if it satisfies all of the following properties:

(i) For each s∈S,|s|divides n.

(ii) The orbits of P constitute a block system^∗for G.

(iii) The orbits of Q constitute a block system^∗∗for G.

(iv) |^∗| =n/p^kand|^∗∗| = p^k. (v) P acts as a p-group on its orbits.

(vi) Q acts as a nilpotent group on its orbits.

Proof. We have seen that a transitive nilpotent group satisfies (i)–(vi). Con-versely, suppose that a transitive groupGsatisfies (i)–(vi), and letN1andN2be the kernels of actions ofGon^∗ and^∗∗, respectively. We haves ∈ sp,sq for each s ∈ S, so G = P,Q. Since P≤N₁ and Q≤N₂, we also have G= N1,N2. Moreover,N1∩N2=1, since the orbits ofN1∩N2constitute ablock system that is acommon refinement of^∗and^∗∗, and so by (iv) the orbits must have size 1. HenceG=N1×N2. Using again thatG= P,Qand P ≤N1,Q≤ N2, we obtainG=P×Q. By (v) and (vi),Gis nilpotent.

Corollary 7.4.3. Let G = S ≤Sym(),with|| =n, be transitive. Then we can test whether G is nilpotent in O(|S|nlogn A⁻¹(2|S|n,n)) time by a deterministic algorithm.

Proof. We have to show that properties (i)–(vi) in Lemma 7.4.2 can be checked in the given time bound. The prime factorization of n can be obtained in O(√

nlog²n) time by brute force. For each s∈S, we can check whether sⁿ = 1 and, if the answer is yes, we can compute sp andsq in O(nlogn) time. The orbits ofP andQcan be determined, and properties (ii), (iii), and (iv) can be checked, inO(|S|n) time. By Theorem 7.4.1, (v) can be checked inO(|S|nlogn A⁻¹(2|S|n,n)) time. Finally, (vi) is checked by a recursive call.

Since the orbits ofQhave sizen/p^k≤n/2, and further recursive calls are for groups of degree decreasing at least by a factor 2, the recursive calls contribute only a constant multiplier to the overall running time.