Normal Subgroups with Nontrivial Centralizer

In practice, solving the problem (6.7) is most time consuming in the case when the input primitive groupGhas a regular normal subgroup with a nontrivial centralizer (i.e.,Ghas a regular abelian normal subgroup orGhas two regular nonabelian normal subgroups). We shall present two approaches. The first method originated in [Luks, 1987] and subsequently was refined in [Babai et al., 1993] and [Beals and Seress, 1992] to a nearly linear-time algorithm. The second method was introduced in [Neumann, 1986], and it handles the case whenGhas a regular abelian normal subgroup. Here we give a nearly linear-time version. As we shall discuss in Remarks 6.2.11 and 6.2.14, both approaches can be extended to handle all primitive groups with regular normal subgroups.

However, we cannot guarantee nearly linear running time of these extensions in the cases whenGhas a unique regular nonabelian normal subgroup orGhas a nonabelian regular normal subgroup, respectively.

Let G≤Sym() be primitive, and suppose that G has a regular normal subgroup N. For anyα, β∈, there is aunique element x_αβ∈N such that α^xαβ = β. The first method finds generators for larger and larger subgroups ofGthat containx_αβ, eventually constructing some maximal subgroupK≤G such that the action of Gon the cosets of K is not faithful and|G:K|<n.

The action on the cosets ofK is constructed, and its kernel is a solution for (6.7). The second method constructs smaller and smaller subsets containing x_αβ, eventually arriving to some subsetK⊆Gthat is small enough so that we can afford to take the normal closure of each element.

Computing the Socle of a Frobenius Group

As a preliminary step, both methods handle the case of Frobenius groups. We present Neumann’s algorithm for this task; another algorithm is described in Section 6.2.5. Some background material on Frobenius groups can be found in Exercises 6.11 and 6.12.

LetG≤Sym() be aprimitive Frobenius group, and letNdenote its unique minimal normal subgroup. Moreover, letα∈and 1 =z∈ Z(G_α). It is known (cf. Exercise 6.12(iii), (iv)) that such an elementzexists. For anyg∈G\Gα, we haveα^zzg =α^zg =α^zgz, since the only fixed point ofzisαand so, in particular, zmoves the pointα^zg. Hence the commutator [z,z^g] =1, but [z,z^g]∈ Nsince the cosetN zis in the center ofG/N. Hence[z,z^g]^G = N.

Given a base and SGS forG, the center ofG_α, and thus a nontrivial element zin it, can be constructed by a nearly linear-time deterministic algorithm, as described in Section 6.1.3. The normal closure[z,z^g]^Gcan also be computed in nearly linear time by a deterministic algorithm.

From now on, we may suppose that the input primitive group has a normal subgroup with nontrivial centralizer, but it is not a Frobenius group. First we describe a solution for (6.7) based on Luks’s approach.

Lemma 6.2.8. Suppose that G≤Sym()is primitive,|| =n, G has a regular normal subgroup with nontrivial centralizer, and there are two pointsα,β∈ such that G_αβ =1. If K is any maximal subgroup of G containing NG(G_αβ) then K contains a regular normal subgroup of G and|G:K|<n.

Proof. LetNbe an arbitrary regular normal subgroup ofG. It contains a unique elementx_αβthat mapsαtoβ. For anyg ∈ G_αβ,x_αβ^g ∈ N and it mapsαtoβ; hencex_αβ^g =x_αβ, which implies thatx_αβ∈CG(G_αβ) a ndx_αβ∈ NG(G_αβ).

Since K is a maximal subgroup of G, the action of G on the cosets of K is primitive. This action cannot be faithful, sinceG either has a regular abelian normal subgroup or two regular normal nonabelian subgroups, and in any primitive faithful action these minimal normal subgroups have to act regularly; however, in theG-action on the cosets ofK, the elementx_αβhas a fixed point (the cosetK·1).

Now, since the action of Gon the cosets of K is not faithful,K contains anormal subgroup ofGand so it contains a minimal normal subgroupM of G. Since K also containsG_αβ and M∩G_αβ=1, we have|G:K| ≤ |M||G:

G_αβ| ≤n−1.

Luks’s Algorithm

The algorithm for solving (6.7) is to computeNG(G_αβ), embed it into a maximal subgroupK, construct the action ofG on the cosets of K, and compute the kernel of the action. We have to show that these computations can be done in nearly linear time.

Lemma 6.2.9. Given G ≤Sym()andα, β ∈as in Lemma6.2.8and an SGS for G relative to some nonredundant base, NG(G_αβ)can be computed in nearly linear time by a deterministic algorithm.

Proof. Applying a base change (cf. Section 5.4), we may assume thatαandβ are the first two base points. LetR1andR2be the transversals forGmodG_α andG_αmodG_αβ, respectively, encoded in shallow Schreier trees (S₁,T₁) a nd (S2,T2). LetN denote a regular normal subgroup ofG.

Letbe the fixed point set ofG_αβ. For anyg∈G, it is clear thatg∈ NG(G_αβ) if and only ifgfixessetwise andgfixessetwise if and only ifα^g ∈and

β^g ∈. Moreover, for anyγ, δ∈, the unique elementx_{γ δ} ∈ Nmovingγto δcentralizesG_αβand sox_{γ δ} ∈ NG(G_αβ). ThereforeNG(G_αβ) acts transitively on.

First, we compute. This can be done in nearly linear time, using the genera-tors forG_αβin the SGS forG. After that, generators forNG(G_αβ) are computed in two steps. We initializeH :=G_αβand the orbitsα^H:= {α}andβ^H:= {β}.

In the first step, we computeNG(G_αβ)_α. This is done by running through the elements of. If for someγ ∈we haveγ ∈β^Hbutγis in the fundamental orbitβ^Gαthen we multiply out the productr_γ of labels along the path fromγto βin the Schreier treeT2, we replaceHbyH,r_γ, and we recompute the orbit β^H. SinceR₂is encoded in a shallow Schreier tree, the computation ofr_γ can be done in nearly linear time. Moreover,H is increased at mostlog(n−1) times, since|NG(G_αβ)_α:G_αβ| ≤n−1.

In the second step, we embedNG(G_αβ)_αinto a subgroup that acts transitively on. By the discussion in the second paragraph of the proof, this subgroup will beNG(G_αβ). Again, we run through the elements of. If we encounter someδ ∈ that is not inα^Hthen we multiply out the product of labelsr_δin the Schreier treeT₁along the path fromδtoαand computeβ^Gα∩^rδ. This intersection is nonempty, since there exist elements inGthat mapδtoαand fix setwise (and so, in particular, mapβto an element of). Any such element xcan be written in the form ¯xr_δ⁻¹ for some ¯x ∈G_αand thenβ^x¯∈β^Gα ∩^rδ. Therefore we can pickγ∈β^Gα∩^rδ, multiply out the product of labelsr_γ in the Schreier treeT₂along the path fromγ toβ, replaceHbyH,r_γ⁻¹r_δ⁻¹, a nd recompute the orbitα^H. SinceHincreases in the second step at mostlog||

times, the second step runs in nearly linear time as well.

The next step in the solution of (6.7) is the embedding ofNG(G_αβ) into a maximal subgroupK ofG. This part of the algorithm is of Monte Carlo type and requires(n) group operations, but so many permutation multiplications are not allowed in the nearly linear-time context. Therefore, we consider the isomorphismψ:G→H with the black-box group H consisting of the stan-dard words, as described in Section 5.3. We embed the subgroupψ(NG(G_αβ)) into a maximal subgroupK^∗ofH, and we finally constructK:=ψ⁻¹(K^∗). By Lemma5.3.1, group operations inH(including the construction of nearly uni-formly distributed random elements in subgroups of H) require onlyO(log^c|G|) time for an absolute constantc.

Lemma 6.2.10. Let H be a black-box group, L≤H , and suppose that we can test membership in L and that there exists a maximal subgroup of K^∗≤H containing L that has index less than n. Then generators for a subgroup L^∗

satisfying LL^∗H can be computed by a Monte Carlo algorithm, using O(log²(ε⁻¹)nlogn)group operations and membership tests in L, whereεis the desired error probability.

Proof. Letl:= nlog((ε/2)⁻¹) . We construct asequence (h1, . . . ,hl) of ele-ments of H recursively. Let h1 be arandom element of H\L and, ifhi−1 is already defined, lethi be arandom element ofL,hi−1\L. (We constructhi

as a random element ofL,hi−1; ifhi happens to be inL, then we repeat the choice ofhi, up tolog(2l/ε) times.)

We claim that with probability greater than 1−ε, the sequence (h1, . . . ,hl) is constructed andLL,hlH. By the definition of the sequence, for each iwe haveLL,hi. For afixedi, ifhi−1is defined then the probability that arandom element ofL,hi−1 is not in L is at least 1/2, so the probability that the algorithm fails to constructhi withlog(2l/ε) random selections in L,hi−1is at mostε/2l. Therefore, the sequence (h1, . . . ,hl) is constructed with probability at least 1−ε/2.

Suppose now that (h1, . . . ,hl) is constructed. IfL,hi−1 =H for somei then the probability thathi ∈K^∗is greater than 1/n, and ifL,hiis aproper subgroup ofH thenL,hjis aproper subgroup ofHfor all j≥i. Hence the probability thatL,hl =His less than (1−1/n)^l < ε/2.

We compute a shallow Schreier tree data structure forNG(G_αβ), compute L:=ψ(NG(G_αβ)), and apply Lemma 6.2.10 to obtainL^∗≤Hand the subgroup ψ⁻¹(L^∗) ≤ G properly containing NG(G_αβ). (Note that because we have a shallow Schreier tree data structure forNG(G_αβ), we can test membership in L.) Then we repeat the same steps, starting withψ⁻¹(L^∗)≤G. We stop when the subgroup returned by the algorithm of Lemma 6.2.10 is H (this fact is noticed when the next shallow Schreir tree data structure is computed); with high probability, the input subgroup of the last iteration is maximal inG. Since

|G: NG(G_αβ)|<n², the number of iterations isO(logn).

The final stage of the solution of (6.7) is the construction of the action ofG on the cosets ofK and the computation of the kernel of this action. This can be done as described in Lemma 6.2.4.

Remark 6.2.11. A similar algorithm can be used to solve (6.7) in the case whenG has a unique nonabelian regular normal subgroup. The construction of NG(G_αβ) and its embedding into a maximal subgroup K can be done in nearly linear time, as described in Lemmas 6.2.9 and 6.2.10. However, we cannot guarantee thatKcontainsN, and so the action ofGon the cosets ofK may be faithful and Lemma 6.2.4 may not be applied. We obtain a solution for

(6.7) even in the case whenK does not containN, since thenG = K N and K∩N =1, so|G:K| ≤n/2. However, it is not clear how we can construct the action on the cosets ofKin nearly linear time. We shall see in Section 6.2.6 that this is possible ifβ is chosen from the smallest orbit ofG_α in\{α}, but the proof of that uses consequences of the classification of simple groups. There is also an elementary nearly linear-time algorithm for the handling of groups with regular non-abelian socle, which we shall present in Section 6.2.4.

Neumann’s Algorithm (the EARNS Subroutine)

Now we turn to the description of theEARNS(Elementary Abelian Regular Normal Subgroup) method. Suppose thatG≤Sym() is primitive with aregu-lar abelian normal subgroupN,n:= || =p^dfor some prime p, and there are two pointsα, β∈such thatG_αβ =1. We may suppose thatβis from an orbit ofG_αin\{α}of the smallest size, since replacing anyβwith one from such an orbit does not decrease the size ofG_αβ and so the propertyG_αβ =1 still holds.

As in the proof of Lemma6.2.9, letbe the set of fixed points ofG_αβ. For any two pointsγ, δ∈, we ha veG_αβ≤G_{γ δ} and so the maximality of|Gαβ| impliesG_αβ=G_{γ δ}. Therefore the restrictionL :=NG(G_αβ)|is either regular or aFrobenius group. In either case,L has a regular normal subgroup R that consists of the restrictions of the elementsx_{γ δ} ∈Nmappingγtoδ, for all (not necessarily different)γ, δ∈. Moreover, all suchX_{γ δ}∈NcentralizeG_αβ(see the second paragraph of the proof of Lemma 6.2.9).

We start the algorithm by computingM:=NG(G_αβ), and thenCG(G_αβ)= CM(G_αβ). As we have observed in the previous paragraph,CG(G_αβ)|contains Ras a normal subgroup. We continue the algorithm by computingR, and the next step is the computation of the preimageDof RinCG(G_αβ). The group Dis abelian, because D()=Z(G_αβ) and so Dis generated by Z(G_αβ) a nd by elements of N that commute with each other and with Z(G_αβ). Next, we construct the subgroup Pof Z(G_αβ) consisting of the elements ofZ(G_αβ) of order dividing p, a ndx∈Dof order psuch thatx| =1. Then the cosetP x contains a nontrivial element of N. Therefore, we can construct N by taking the normal closure of each element ofP xinGand pick the only abelian group among these normal closures.

By Lemma 6.2.9 and by Section 6.1.4, the computation ofCG(G_αβ) ca n be done in nearly linear time. We have seen earlier in this section that the regular normal subgroup in Frobenius groups can be computed in nearly linear time, so Rcan be obtained by a nearly linear-time algorithm. The groupDis apreimage at a transitive constituent homomorphism, andZ(G_αβ) is the pointwise stabilizer

D(). Generators for P are obtained by taking the appropriate powers of the generators of Z(G_αβ), andx can be chosen as a power of an element in the transversal for DmodD_α. Hence the coset P x can be constructed in nearly linear time.

Next, we need an upper estimate for|P|. It is based on the following result [Neumann, 1986, Lemma 3.4], the proof of which we leave as Exercise 6.13.

Lemma 6.2.12. Let M be any group, let A be an abelian p-subgroup of M of order q, let C:=CM(A), and let m := |M :C|. If Op(M)=1then q≤m.

Corollary 6.2.13. |P|<n.

Proof. We claim that Op(G_α)=1. Indeed, N can be identified with a d-dimensional vector space over GF(p), withαplaying the role of the 0 vector;

with this identification,G_αacts onNby conjugation as an irreducible subgroup of GLd(p), and the fixed vectors ofOp(G_α) comprise ablock of imprimitivity ofG(cf. Exercise 6.14). Hence the only fixed vector ofOp(G_α) is the 0 vector and soOp(G_α)=1.

We can apply Lemma 6.2.12 withM :=G_αandA:=P. SinceP ≤Z(G_αβ), we haveCM(P)≥G_αβ and so|M : CM(P)| ≤ |Gα : G_αβ)| ≤ n−1. Hence

|P| ≤ |M :CM(P)|<n.

By the corollary,|P x|<n. In a permutation group, we can compute the nor-mal closure of an element in nearly linear time; however, we cannot do(n) such computations within the nearly linear time constraint. Therefore, as in Luks’s approach, we consider the isomorphismψ:G→Hwith the black-box group H= Sconsisting of the standard words. By Theorem 2.3.9 and Lemma 2.3.14, for anyh ∈Hthe normal closureh^Hcan be computed, and then the commu-tativeness ofh^Hcan be tested, usingO(log|H|(|S| +log|H|)) group oper-ations. By Lemma 5.3.1, group operations inH can be performed in log^c|G|

time, for an absolute constantc. Hence we can compute the normal closure of all standard words representing elements ofP x in nearly linear total time by a Monte Carlo algorithm. When an abelian normal subgroup ofHis found, the image of its generators underψ⁻¹can be computed in nearly linear time by a deterministic algorithm.

Remark 6.2.14. This method can also be extended to solve (6.7) for primitive groups with one or two nonabelian regular normal subgroups. Let N denote a regular normal subgroup ofG. Using the notation of Neumann’s algorithm, our observation that R consists of the restrictions of the elements ofN to is still valid, and R can be constructed in nearly linear time. For an element

x∈Dof prime orderr satisfyingx| =1 and for the subgroupPof Z(G_αβ) consisting of elements of order dividingr, it is still true that P x contains a nontrivial element of N. Corollary 6.2.13 is also true, sinceG_α has no non-trivial solvable normal subgroups (cf. Exercise 6.15 for the case of two regular normal subgroups and [Dixon and Mortimer, 1996, Theorem 4.7B(i)] for the case of a unique nonabelian regular normal subgroup). Hence we can com-pute the normal closures in H of all standard words representing elements of P x in nearly linear total time. However, we cannot decide which normal closure corresponds to aregular subgroup ofGwithin the nearly linear time constraint.

6.2.4. Groups with a Unique Nonabelian Minimal Normal Subgroup