The O’Nan–Scott Theorem

Every abstract group has a faithful transitive permutation representation, but the structure of primitive permutation groups is quite restricted. Primitive groups can be divided into classes according to their socle and the stabilizer of a point of the permutation domain in the socle. This characterization is known as the O’Nan–Scott theorem (cf. [Scott, 1980]), but there are numerous versions. In fact, we shall also state two versions: The first one is more of a group theoretic nature, whereas the second one is a rearrangement of the cases and is suited better for our algorithmic purposes.

We need some notation. IfTis a simple nonabelian group andH=T1×· · ·×

Tkis agroup such that fori=1,2, . . . ,kwe have isomorphismsϕi:T→Tithen we denote the subgroup{(ϕ1(t), . . . , ϕk(t))∈H|t∈T}by Diag(H) and call it adiagonal subgroupof H. Although Diag(H) depends on the isomorphisms ϕi, there will be no confusion from omitting these isomorphisms from the notation.

Theorem 6.2.5. Let G≤ Sym()be primitive and let|| =n. Then one of the following holds:

(I) G has a minimal normal subgroup N with CG(N) =1. Moreover, (i) if N is abelian then n= p^d for some prime p,N is regular, and it is

the only minimal normal subgroup of G;

(ii) if N is nonabelian then there are exactly two minimal normal sub-groups of G and both of them are regular.

(II) G has a unique minimal normal subgroup N=Soc(G)=T₁× · · · ×Tr, where each Ti is isomorphic to the same simple nonabelian group T and GAut(T)Sr. The group G permutes the set{T1, . . . ,Tr}by conjugation, and this permutation action is transitive. Moreover, letα ∈. Then one of the following three cases occurs:

(i) N_α =1and n= |T|^r;

(ii) N_α =(T1)_α× · · · ×(Tr)_αwith isomorphic subgroups(Ti)_αsatisfying 1 =(Ti)_α =Ti, and n= |T₁: (T₁)_α|^r; or

(iii) r=kl with k≥2 and there exists a permutation of the indices 1,2, . . . ,r such that N_α=Diag(T₁× · · · ×Tk)×Diag(Tk+1× · · · × T2k)× · · · ×Diag(Tr−k+1× · · · ×Tr).Moreover, n= |T1|^(k−1)l. Proof. Suppose first thatGhas a minimal normal subgroupNwithCG(N) =1.

By Exercise 6.4,CG(N) is also a normal subgroup ofGand its centralizer is nontrivial, since it containsN. Moreover, by Exercise 6.1, bothN andCG(N) are transitive. However, by Lemma 6.1.4,NandCG(N) are semiregular as sub-groups of the semiregular sub-groupsC_Sym()(CG(N)) andC_Sym()(N), respectively.

Hence bothN andCG(N) are regular.

IfNis abelian thenN ≤CG(N) and soN =CG(N). Moreover, sinceNis a minimal normal subgroup, it is the direct product of isomorphic simple groups, that is, an elementary abelianp-group for some primepandn= p^dfor some d ≥1. IfNis nonabelian thenNandCG(N) are two different subgroups. Both of them are minimal normal inG, since aproper subgroup of aregular group cannot be transitive. In both of the cases of abelian and nonabelian N, there are no other minimal normal subgroups ofGbecause any two minimal normal subgroups must centralize each other. Hence we are in case I(i) or I(ii) of the theorem.

Suppose now thatGhas no minimal normal subgroup with nontrivial cen-tralizer. Then, using again that any two minimal normal subgroups centralize each other, we obtain thatGhas only one minimal normal subgroup, which we may denote byN. SinceNis minimal normal, there exists a nonabelian simple groupT and an integerr≥1 such that N = T1× · · · ×Tr withTi ∼=T for alli ∈[1,r]. The only minimal normal subgroups ofN are the groupsTi for i ∈[1,r] (cf. Exercise 6.10), soGmust permute these by conjugation. Hence Aut(N)∼=Aut(T)Sr andGcan be identified with a subgroup of Aut(T)Sr. The minimality ofNalso implies that this conjugation action is transitive since any orbit corresponds to a normal subgroup ofG.

Letα ∈ . We first observe that G_α also acts by conjugation transitively on{T1, . . . ,Tr}, sinceG=N G_αand the conjugation action ofNis trivial. We consider the projectionsπi :N_α→Ti.

Ifπ1is surjective then allπiare surjective sinceπi(N_α)=π1(N_α)^gfor some g∈G_αconjugating T1toTi. In this case, N_α is asubdirect product of theTi

and, by Exercise 6.9, there is a partition = (1, . . . , l) of the index set {1,2, . . . ,r}such thatN_αis the product ofldiagonal subgroups, corresponding to this partition. SinceN_αG_α, the partsi are permuted by the conjugation

action ofG_αon{1,2, . . . ,r}, and since the conjugation action is transitive, all parts must be of the same size. Hence we are in case II(iii).

If 1π1(N_α) T1 then 1 πi(N_α)Ti for alli ∈ [1,r] and we are in case II(ii). Finally, if 1=π1(N_α) then 1=πi(N_α) for alli∈[1,r] and we are

in case II(i).

It is possible to describe the structure and action of primitive groups in much more detail than was done here, but we confine ourselves to the properties needed in the design and analysis of the composition series algorithm. A more thorough description can be found, for example, in [Dixon and Mortimer, 1996, Chap. 4]. Some further material is also listed in Exercise 6.7 concerning case I(ii) and in Exercise 6.8 concerning cases II(ii) and II(iii) withl>1. We need more details only in the case II(iii) withl=1. The proof of the following lemma can be found in [Dixon and Mortimer, 1996, Theorem 4.5A].

Lemma 6.2.6. Let G ≤ Sym()be primitive with a unique minimal normal subgroup N =T1× · · · ×Tr, such that all Tiare isomorphic to a nonabelian simple group T . Letα∈and suppose that N_α=Diag(T₁× · · · ×Tr). Then G_α=H×P, whereInn(T)HAut(T)and P is isomorphic to a primitive subgroup of Sr.

The subgroup N_αdefines an isomorphism among the Ti. So N can be iden-tified with the set of sequences(t1, . . . ,tr),ti ∈T . For g∈ P,(t1, . . . ,tr)^gis a sequence consisting of a permutation of the entries of(t₁, . . . ,tr), according to the permutation action of P in Sr.

Our second version of the O’Nan–Scott theorem follows.

Theorem 6.2.7. Let G≤Sym()be primitive and let|| =n. Then G satisfies at least one of the following properties:

(A) G has a minimal normal subgroup N with CG(N) =1.

(B) G has a proper normal subgroup of index less than n.

(C) G has a unique minimal normal subgroup N =Soc(G)=N₁× · · · ×Nm, where the Niare isomorphic groups, m!/2≥n, and G acts by conjugation on{N1, . . . ,Nm}as the full alternating group Am. Moreover, letα∈. Then one of the following three cases occurs:

(i)N_α=1;

(ii)N_α=(N₁)_α× · · · ×(Nm)_αwith1 =(Ni)_α =Ni; or

(iii)each Ni is isomorphic to the same simple nonabelian group T and N_α=Diag(N₁× · · · ×Nm).

(D) G is simple.

Proof. We have to show that the groups in case II of Theorem 6.2.5 are covered by cases B, C, and D of this theorem. Using the notation of Theorem 6.2.5, suppose thatGis in case II, with minimal normal subgroupN=T1× · · · ×Tr

and allTi isomorphic to the same simple nonabelian groupT. Since|T| ≥60 and|T:H| ≥5 for any proper subgroup HofT, we ha ver ≤log₅nin each subcase of case II.

Ifr =1 then eitherGis simple (and it is in case D) orN ∼=Inn(T)G ≤ Aut(T). In the latter case,G/Nis solvable by Schreier’s Hypothesis (which is now a theorem, as a consequence of the classification of finite simple groups);

henceGhas a cyclic factor group of orderpfor some primep. Clearly p≤n, and soGis in case B.

Suppose now thatr≥2. Letbe a minimal block system in the conjugation action ofGon{T1, , . . . ,Tr}and letm:= ||. IfGis in case II(iii) andm<

r then we also suppose that the groups Ti belonging to the same diagonal collection are in the same block of. ThenGacts primitively on; letP ≤Sm

denote the image of this action. By [Praeger and Saxl, 1980], if P≤Sm is primitive and P =Am,Sm then|P|<4^m; sincem≤r≤log₅n, the kernel of action on has index less thann andG is in case B. If P=Sm then it has a normal subgroup of index 2 and G is in case B. This leaves us with the case thatP=Am. Ifm!/2<nthen we are in case B, so we may suppose that m!/2≥n.

Now, ifGis in case II(i) thenGis in case C(i). IfGis in case II(ii) then it is in case C(ii). IfGis in case II(iii) withl>1 then it is in case C(ii). Finally, if Gis in case II(iii) withl=1 thenGis in case C(iii).

The composition series algorithm that we shall describe in the next two sections uses different methods for solving the problem (6.7) from Section 6.2.1 for inputs belonging to the different cases of Theorem 6.2.7. If it is not known to which case of Theorem 6.2.7 the input primitive groupGbelongs, then the algorithm tries all methods. Note that if the output of (6.7) is a proper normal subgroup or a faithful permutation action on at mostn/2 points then the cor-rectness of the output can be checked in nearly linear time. Hence if all methods report failure or thatGis simple then we can conclude thatGis indeed simple.

The only regular primitive groups are cyclic of prime order, and these can be easily recognized. Hence, in the next two sections, we may assume that the input primitive group is not regular.