Fixed points of endomorphisms of certain free products

FIXED POINTS OF ENDOMORPHISMS OF CERTAIN FREE PRODUCTS

Pedro V. Silva

Abstract. The fixed point submonoid of an endomorphism of a free product of a free monoid and cyclic groups is proved to be rational using automata-theoretic techniques. Maslakova’s result on the computability of the fixed point subgroup of a free group automorphism is generalized to endomorphisms of free products of a free monoid and a free group which are automorphisms of the maximal subgroup.

Mathematics Subject Classification. 20M05, 20F10.

1. Introduction

Gersten proved in the eighties that the fixed point subgroup of a free group au- tomorphism ϕis finitely generated [10]. Using a different approach, Cooper gave an alternative proof, proving also that the fixed points of the continuous exten- sion ofϕto the boundary of the free group is in some sense finitely generated [9].

Bestvina and Handel achieved in 1992 a major breakthrough through their inno- vative train track techniques, bounding the rank of the fixed point subgroup and the generating set for the infinite fixed points [3]. Their approach was pursued by Maslakova in 2003 to prove that the fixed point subgroup can be effectively computed [16].

Gersten’s result was generalized to further classes of groups and endomorphisms in subsequent years. Goldstein and Turner extended it to monomorphisms of free groups [11], and later to arbitrary endomorphisms [12]. Collins and Turner ex- tended it to automorphisms of free products of freely indecomposable groups [8], and recently Sykiotis to monomorphisms [21]. The interested reader can find more information in Ventura’s excellent survey [22].

Keywords and phrases.Endomorphisms, fixed points, free products.

1 Centro de Matem´atica, Faculdade de Ciˆencias, Universidade do Porto, R. Campo Alegre 687, 4169-007 Porto, Portugal.pvsilva@fc.up.pt

Article published by EDP Sciences c EDP Sciences 2011

PEDRO V. SILVA

Cassaigne and the author developed in [6] an approach to the study of monoids defined by special confluent rewriting systems (SC monoids) that preserves some of the features of the free group case and contains free products of cyclic groups as a particular case, as well as the partially reversible monoids introduced in [19].

In fact, the undirected Cayley graph of these monoids is hyperbolic and there exists a nice compact completion for the prefix metric. Uniformly continuous en- domorphisms, algorithmically characterized in [6], admit a continuous extension to the boundary. In [7], the same authors used this approach to study the dynam- ics of infinite periodic points for two classes of endomorphisms of the monoids in question.

In [20], the author proved that the fixed point submonoids of a large class of uniformly continuous endomorphisms of SC monoids are rational (actually finitely generated in the group case), obtaining also results of the same flavour for infinite fixed points. In the group case, new results were obtained for infinite fixed points of monomorphisms of free products of cyclic groups (c-free groups).

In the present paper, we go beyond the uniform continuity restriction of [20], which is equivalent to injectivity in the group case. Section 3 is devoted to the problem of proving that the fixed point submonoid is rational, generalizing Gold- stein and Turner’s proof for free group endomorphisms. This involves facing some technical difficulties brought by the existence of finite order elements. We obtain thus a fully automata-theoretic proof of a result that follows also from previous results by Sykiotis [21] in the particular cases of monomorphisms or symetric en- domorphisms. Another advantage of our proof is that it may offer some insight into the algorithmic aspects of the problem.

In Section 4, we discuss computability of the fixed point submonoid, the ref- erence being of course Maslakova’s result on free group automorphisms [16]. We generalize this result to endomorphisms of free products of a free monoid and a free group whose restriction to the group is an automorphism.

2. Preliminaries

Given a monoid M, we denote by RatM the set of all rational subsets of M, i.e., the smallest family of subsets ofM containing the finite sets and closed under union, product and the star operator (X^∗denotes the submonoid ofM generated byX⊆M). For details on rational languages, the reader is referred to [2,18].

In the particular case of a free monoidM =A^∗, a combinatorial description in terms of finite automata is usually preferred. We define a (finite)A-automaton to be a quadruple A = (Q, q₀, T, E) where Q is a (finite) set, q₀ ∈Q is the initial vertex,T ⊆Qare the terminal vertices andE⊆Q×A×Q. Anontrivial pathin Ais a sequence

p₀−→p^a1 1−→a2 . . .−→p^an n

with (p_i−1, a_i, p_i) ∈ E for i = 1, . . . , n. Its label is the word a₁. . . a_n ∈ A⁺ = A^∗\ {1}. It is said to be a successful path if p₀ = q₀ and p_n ∈ T. We consider

also thetrivial path p−→p¹ for p∈Q. It is successful ifp=q₀ ∈T. The language recognized byAis

L(A) ={w∈A^∗|wlabels a successful path inA}.

The classical Kleene’s Theorem states thatL⊆A^∗is rational if and only ifL= L(A) for some finiteA-automatonA. If we replace letters by rational languages as labels of edges (the resulting language being the union of the languages obtained by taking all the possible choices for each edge label), we remain within the realm of rational languages. Note that, if we fix a homomorphism π : A^∗ → M, then RatM = (RatA^∗)πand so the rational subsets ofM can be defined through finite automata.

LetA= (Q, q₀, T, E) be anA-automaton. We say thatAis

• deterministicif (p, a, q),(p, a, r)∈E impliesq=r;

• completeif there exist edges with arbitrary label starting at every vertex;

• accessibleif there exist paths fromq₀to any arbitrary vertex.

Another case that will be relevant for us is the case ofM being a group, when we have the following result of Anissimov and Seifert:

Proposition 2.1 ([18] Prop. II.6.2). Let H be a subgroup of a group G. Then H ∈RatGif and only ifH is finitely generated.

A finite rewriting system is a formal expressionA | R, where A is a finite alphabet and R a finite subset of A^∗×A^∗. The elements of R are called rules.

OnceR is fixed, for givenu, v∈A^∗, we write u−→v if u=xry, v=xsy

for somex, y ∈A^∗ and (r, s)∈R. We denote by −→^∗ the reflexive and transitive closure of the relation−→. ThecongruenceonA^∗generated byRwill be denoted byR. Note thatRcoincides with the relation−→^∗ defined with respect toR∪R⁻¹. The quotient M = A^∗/R is said to be the monoid defined by the rewriting systemR. We denote byπthe canonical homomorphismA^∗→M.

A rewriting systemA|Ris said to be

• specialifR⊆A⁺× {1};

• confluentif, wheneveru−→v^∗ andu−→w, there exists^∗ z∈A^∗ such thatv−→z^∗ andw−→^∗ z:

u ^∗ //

∗

v ∗

w_ ^∗_ _//z

We shall refer to a monoid defined by a finite special confluent rewriting system as anSC monoid. An important case is given by free groups. Indeed, the free group onA, denoted byF G_A, is defined by

A∪A⁻¹|aa⁻¹→1, a⁻¹a→1 (a∈A), whereA⁻¹ denotes a set of formal inverses ofA.

PEDRO V. SILVA

LetA|Rbe a special confluent rewriting system. We say thatw∈A^∗isirre- ducible(with respect toR) ifw /∈ ∪_(r,s)∈RA^∗rA^∗. For everyu∈A^∗, there is exactly one irreduciblev ∈ A^∗ such that u−→v: existence follows from^∗ R being length- reducing, and uniqueness from confluence. We denote this unique irreducible word byu. It is well known (see [5]) that the equivalence

uπ=vπ⇔u=v

holds for allu, v∈A^∗, henceA^∗={u|u∈A^∗}constitutes a set of normal forms for the monoidM =A^∗/R.

A generalized version of the classical Benois’ Theorem states that rational lan- guages are preserved by reduction:

Theorem 2.2 ([1]). LetA|Rbe a finite special confluent rewriting system and letL⊆A^∗ be rational. Then Lis rational and effectively constructible from L.

If M and M are defined respectively by the rewriting systems A | R and A|R, then the free productM∗M is defined byA∪A|R∪R, takingA disjoint fromA. We can now introduce the crucial concept of c-free group: a group is said to bec-freeif it is a free product of cyclic groups. Since every free group is a free product of infinite cyclic groups, the class of c-free groups extends the class of free groups.

We denote the monoid of endomorphisms (respectively group of automorphisms) ofM by EndM (respectively AutM). Given ϕ∈EndM, let

Fixϕ={u∈M |uϕ=u}.

We say that Fixϕis the submonoid offixed pointsofϕ. Note that Fixϕis a group ifM is a group.

3. Rationality

It is easy to determine which SC monoids can be embedded into some group.

Following [17], we say that a monoidM isdirectly finiteif

∀x, y∈M xy= 1⇒yx= 1.

We recall that the bicyclic monoid is the SC monoid defined by the rewriting systema, b|ab→1.

Proposition 3.1. Let M be an SC monoid. Then the following conditions are equivalent:

(i) M is embeddable into some group;

(ii) M contains no bicyclic submonoid;

(iii) M is directly finite;

(iv) M is a free product of a free monoid and cyclic groups.

Proof. (i) ⇒ (ii). Since a bicyclic monoid contains infinitely many idempotents (bⁿaⁿ for the standard rewriting system).

(ii)⇒(iii). By [15] Section VI.3.

(iii)⇒(iv). Assume thatM is defined by the finite special confluent rewriting systemA|R. LetA₁ be the set of generators which occur in some relator ofR, and writeA₀=A\A₁. Leta∈A₁. ThenRhas some relator of the formuav→1 for someu, v∈A^∗. SinceM is directly finite, it follows that (vua)π= 1 = (avu)π, henceaπ is invertible inM and soA^∗₁π is a subgroup ofM. SinceA^∗₁πis defined by the finite special confluent rewriting system A₁ | R, we get a free product decompositionM =A^∗₀∗A^∗₁π. By [20] Proposition 6.1, every SC group is c-free and so (iv) holds.

(iv) ⇒ (i). We can extend the canonical embedding ofA^∗ in F G_A to an em-

bedding ofA^∗∗GintoF G_A∗G.

We adapt now Goldstein and Turner’s proof [11] to c-free groups.

Theorem 3.2. Let ϕ be an endomorphism of a finitely generated c-free group.

Then Fixϕis finitely generated.

Proof. LetGbe a finitely generated c-free group. Clearly,Gcan be defined by a finite rewriting system of the formA|R, where A=A₀∪A₁∪A⁻¹₁ and there existsm_a≥2 for everya∈A₀ such that

R={(a^ma,1)|a∈A₀} ∪ {(aa⁻¹,1),(a⁻¹a,1)|a∈A₁}.

Since the unique overlapping of relators we can get proceeds from the free group relatorsaa⁻¹, a⁻¹a, it follows easily that this special rewriting system is confluent, henceGis an SC-monoid.

Let ϕ be an endomorphism of G and let π : A^∗ → G denote the canonical morphism. For everya∈A₀, writea⁻¹=a^ma−1. Write also (a⁻¹)⁻¹=afor every a∈A₁. Define u⁻¹inductively for everyu∈A^∗ through

1⁻¹= 1, (va)⁻¹=a⁻¹v⁻¹(v∈A⁺, a∈A).

Clearly, (u⁻¹)π= (uπ)⁻¹ for everyu∈A^∗.

Clearly,A^∗acts onGthroughga=g(aπ). For everyg∈G, letQ(g) =g⁻¹(gϕ).

Note that g ∈ Fixϕ if and only if Q(g) = 1. We define an A-automatonAϕ = (Q,1,1, E) by

Q={Q(g)|g∈G};

E={(Q(g), a, Q(ga))|g∈G, a∈A}.

Clearly,Aϕ is a complete accessible deterministic automaton and L(Aϕ) = (Fixϕ)π⁻¹.

We define a subautomatonA_ϕ= (Q,1,1, E) by taking E ={(p, a, q)∈E|aqis irreducible}.

PEDRO V. SILVA

Letd_ϕ= max{|aϕ|; a∈A} andm_R= max({2} ∪ {m_a|a∈A₀}). Note that, for allu∈A^∗ anda∈A, the suffix ofuinvolved in the reduction ofu(aϕ) has length at most (m_R−1)d_ϕ(since each letter ofaϕcan erase at mostm_R−1 letters ofu).

We show that

∀p∈Q |p|>(m_R−1)d_ϕ⇒phas outdegree ≤1 inA_ϕ. (3.1) Letp∈Qbe such that |p|>(m_R−1)d_ϕ. Suppose that (p, a, q),(p, b, q)∈E are distinct edges. Since A_ϕ is deterministic, we have a =b. It suffices to show thatp∈aA^∗. By symmetry, alsop∈bA^∗and we reach the required contradiction.

Suppose thatp=cu withc∈A\ {a}. Thenq=a⁻¹cu(aϕ). Sincec=a, then a⁻¹cuis irreducible. Now|u| ≥(m_R−1)d_ϕ implies thata⁻¹cremains untouched in the reduction ofa⁻¹cu(aϕ). Hence q=a⁻¹cu(aϕ) and soaqis reducible, con- tradicting (p, a, q)∈E. Thusp∈aA^∗ and so (3.1) holds.

Givenq∈Q, let thedepthofq, denoted by dep(q), be the length of the shortest path 1−→qin Aϕ. Since Aϕis accessible, dep(q) is well defined.

Fixs₀>1 + (m_R−1)d_ϕ such thats₀≥m_a+ (m_R−1)|aⁱϕ|for alla∈A₀and i∈ {1, . . . , m_a−1}, and let

s=m_R+ max{dep(p)|p∈Qand|p|< s₀}. It follows that

dep(p)> s−m_R⇒ |p| ≥s₀>1 + (m_R−1)d_ϕ (3.2) holds for everyp∈Q.

Our proof of Theorem3.2requires the following intermediate result:

Lemma 3.3. If (p, a, q)∈E\E and dep(p),dep(q)> s−m_R, then there exists a pathq−→p^a−1 in A_ϕ.

Proof. Clearly, we have a pathq−−→p^a−1 inAϕ. We must show that all the edges in it are inE. First, we note thataq must be reducible by definition ofE, and so q=a⁻¹ufor some u∈A^∗\aA^∗.

Assume first that a ∈A₁∪A⁻¹₁ . Suppose that (q, a⁻¹, p)∈/ E. Then a⁻¹pis reducible and so p = abv for some b ∈ A\ {a⁻¹} and v ∈ A^∗. We have |p| >

1 + (m_R−1)d_ϕ by (3.2), hencea⁻¹u=q=a⁻¹p(aϕ) =bv(aϕ), yielding b=a⁻¹ and contradictingp=abv. Thus (q, a⁻¹, p)∈E in this case.

Assume now thata∈A₀. Fori= 1, . . . , m_a−1, we have an edge a^ma−iu(aⁱ⁻¹ϕ)−→a^{a ma−i−1}u(aⁱϕ)

in A_ϕ. Since |q| ≥ s₀ > 1 + (m_R −1)d_ϕ by (3.2), we get |u| > s₀ −m_a ≥ (m_R−1)|aⁱ⁻¹ϕ|,(m_R−1)|aⁱϕ|. It follows thata^ma−iu(aⁱ⁻¹ϕ)−→^a a^ma−i−1u(aⁱϕ) is an edge of Aϕ, indeed of A_ϕ, for i = 1, . . . , m_a −1. Thus there is a path q−−→u(a^{a−1 ma−1}ϕ) in A_ϕ. SinceA_ϕ is deterministic, it follows thatu(a^ma−1ϕ) =p

and the lemma is proved.

Back to the proof of Theorem3.2, letB_ϕ be the (finite) full subautomaton of Aϕ induced by the subset of vertices of depth ≤ s (that is, Bϕ contains all the edges ofAϕ connecting vertices of depth≤s).

Givenq∈Qof depth> s−m_R, by (3.1) and (3.2) there exists inA_ϕ a unique maximal path α_q : q−→. . . where every vertex has depth > s−m_R. Let Q₀ (respectively Q₁) denote the set of allq∈Qwiths−m_R<dep(q)≤ssuch that the set of vertices occurring inα_q is finite (respectively infinite). Given p, q∈Q₁ distinct, let p∧q denote the first vertex inα_p to appear inα_q (if such a vertex exists, otherwisep∧qremains undefined). Thenp∧q=q∧p, otherwise we would have a cycle

p∧qll ,,q∧p contradictingp∈Q₁.

We define Cϕ to be the automaton obtained by adding to Bϕ all vertices and edges in the following paths ofA_ϕ:

(C1) α_q forq∈Q₀;

(C2) initial segmentsq−→pofα_q forq∈Q₁ and dep(p)≤s;

(C3) p−→p∧qfor allp, q∈Q₁such thatp∧q is defined.

Clearly,C_ϕis finite. Finally,C_ϕ is obtained by adding toC_ϕ, for every edge (p, a, q) ofCϕ, all the edges in the pathq−→^a−1pinAϕ. Note thatC_ϕ is a finite subautomaton ofAϕ. Moreover, if (p, a, q) is an edge ofC_ϕ, there exists a pathq−→p^a−1 inC_ϕ.

We prove now that

Fixϕ⊆L(C_ϕ). (3.3)

Recall that (Fixϕ)π⁻¹=L(Aϕ). Since Bϕis a subautomaton ofC_ϕ, it suffices to show that every path p−→q^u in Aϕ such that u∈A^∗, dep(p) = s, dep(q) ≤s and all intermediate vertices have depth > s, is also a path inC_ϕ.

Letp−→q^u be such a path. We can factor this path as p=p₀−→r^u0 1−→pv1 1−→u1 . . .−→p^vn n−→run n+1=q,

where the r_i−→p^vi _i group the edges in E\E, and u_i, v_j = 1 for 1 ≤i ≤n−1 and 1 ≤j ≤n. Note that, when (r, a, t) ∈ E, then dep(t) > dep(r)−m_R: this is clear ifa∈ A₁∪A⁻¹₁ since (t, a⁻¹, r)∈E; on the other hand, if a∈A₀, then there is a path t−→r^a−1 of length < m_R and our claim holds too. It follows that dep(q)> s−m_R and so we can apply Lemma 3.3and get paths

p=p₀−→r^u0 ₁←−p^v1−1 ₁−→^u1 . . .^v

n−1

←−p_n−→r^un _n+1=q

inA_ϕ. Assume first thatn= 0. Thenp−→q^u is an initial segment of some pathα_q in (C1) or (C2), hence is a path in C_ϕ and therefore inC_ϕ. Thus we assume that n >0. Takei∈ {1, . . . , n}. We show thatu_i is a proper prefix ofv_i⁻¹.

Indeed, we have paths

r_i^v

−1i

←−p_i−→r^ui _i+1

PEDRO V. SILVA

in A_ϕ and all our vertices are too deep to have outdegree>1. Hence one of this two paths must be an initial segment of the other. Write v_i =xa witha∈ A. If a∈ A₁∪A⁻¹₁ , then u_i = 1, otherwiseu_i ∈a⁻¹A^∗, and aa⁻¹ would be a factor of v_iu_i, contradicting the irreducibility of u. Hence we may assume that a∈ A₀ and so v_i⁻¹ = a^ma−1x⁻¹. Similarly to the preceding case, u_i ∈ a^ma−1A^∗ would contradict the irreducibility ofu, hence u_i must be a proper prefix of a^ma−1 and therefore ofv⁻¹_i .

Writev⁻¹_i =u_iw_i. We have paths

p−→r^u0 1←−rw1 2←−w2 . . .←−r^wn n+1=q

inA_ϕ. Letw=w_nw_n−1. . . w₁. We claim thatp−→r^u0 1←−qw are paths inC_ϕ. This is immediate ifp∈Q₀ (which is equivalent to q∈Q₀), hence we assume thatp, q∈Q₁.

Suppose thatp=q. Thenu₀=wand sowv₁u₁is a factor ofu, hence irreducible.

Decompose in lettersv₁=a₁. . . a_k. Sinceu₁w₁=v₁⁻¹=a⁻¹_k . . . a⁻¹₁ andw₁v₁u₁is irreducible, it follows thatk= 1, otherwisea⁻¹_k is a prefix ofu₁ ora⁻¹₁ is a suffix ofw₁. Ifa₁∈/A₀, thenu₁w₁=a⁻¹₁ produces immediately a contradiction. On the other hand, ifa₁ ∈ A₀, then u₁w₁ =a^ma−1 and so w₁v₁u₁ =a^ma is reducible, another contradiction. Hencep=q.

Writeu₀=u₀h,w=wh, wherehdenotes the longest common suffix of the two words. Then we have pathsp^u

0

−→p∧q←−q^w inCϕ. So we are done ifh= 1. Assume then thath = 1 and write h =hb with b ∈ A. Note that, since u₁ is a proper prefix ofv₁⁻¹, thenw₁ is a nontrivial suffix ofw. If b /∈A₀, thenw₁∈A^∗b and so v₁∈b⁻¹A^∗. Hencebb⁻¹would be a factor ofu₀v₁, contradicting the irreducibility ofu. Henceb∈A₀. Writeh=hbⁱwithh∈/A^∗b. Thenw₁∈A^∗band sov₁∈bA^∗. Suppose first that v₁ = bvc with c ∈ A. Then u₁w₁ = v⁻¹₁ = c⁻¹v⁻¹b^mb−1. If c⁻¹is a prefix ofu₁, thenv₁u₁would not be irreducible, henceb^mb−1 is a suffix of w₁. Now, sinceu₀v₁ is irreducible andv₁ starts byb, we have i≤m_a−2 and so h=bⁱ. Moreover, since b^mb−1 is a suffix ofw, the last edge ofq−→p^w ∧q belongs to theb-labelled cycle ofAϕcontainingp∧q−→r^h 1, and so the edges in this path must belong toC_ϕ.

Assume now that v₁ = b. Then u₁w₁ = b^mb−1. Since u₀v₁u₁ = u₀hbⁱbu₁ is irreducible, we have|w₁|> iand so once again the last edge ofq−→^w p∧qbelongs to the b-labelled cycle of Aϕ containing p∧q−→r^h 1. Therefore the edges in this path must belong toC_ϕ.

Now, since p−→^u0 r₁←−^w q are paths in C_ϕ, also r₁−→^w−1q is a path in C_ϕ. Hence v⁻¹_i = u_iw_i yields v_i = w⁻¹_i u⁻¹_i and so w⁻¹_i = v_iu_i = v_iu_i since v_iu_i is irreducible. Sincew⁻¹ =w₁⁻¹. . . w⁻¹_n , it follows that there exists a path inC_ϕ of the formr₁−→q^z forz=v₁u₁. . . v_nu_n. Therefore p−→q^u is a path inC_ϕ and (3.3) holds.

Now Fixϕ⊆L(C_ϕ)⊆L(Aϕ) = (Fixϕ)π⁻¹ yields Fixϕ ⊆L(C_ϕ)⊆Fixϕand so Fixϕ = L(C_ϕ). By Theorem 2.2, Fixϕ is rational and so Fixϕ is a rational subset ofG. By Proposition2.1, Fixϕis then a finitely generated subgroup ofG.

Corollary 3.4. Letϕ be an endomorphism of a finitely generated free product of a free monoid and cyclic groups. ThenFixϕis rational.

Proof. LetM =A^∗₀∗G₁, whereG₁is the c-free group defined by the finite special confluent rewriting system A₁ | R₁. Let G be the c-free group defined by the finite special confluent rewriting system A | R, where A = A₀ ∪A⁻¹₀ ∪A₁ and R = R₁∪ {(aa⁻¹,1),(a⁻¹a,1) | a ∈A₀}. Let u(respectively u) denote the irreducible form ofuin the rewriting system ofM (respectivelyG).

Nowϕextends to an endomorphismΦofGthrougha⁻¹₀ Φ= (a₀ϕ)⁻¹(a₀∈A₀).

By Theorem3.2, FixΦis a finitely generated and therefore rational subgroup ofG.

Denoting the canonical homomorphismA^∗ →Gbyπ, it follows that FixΦ=Lπ for some rationalL⊆A^∗. In view of Theorem 2.2,FixΦ=L is a rational subset ofA^∗. Since Fixϕ=FixΦ∩(A₀∪A₁)^∗, it follows that Fixϕis a rational subset of (A₀∪A₁)^∗ and so Fixϕis a rational subset ofM.

4. Computability

We start this section by considering the simple case of free monoid endomor- phisms, whose discussion is essential to the follow-up. The first proof was given by Head [14], and an alternative one was later provided by Hamm and Shallit [13].

We include here a very short proof.

LetA₀ be a finite alphabet and letϕ∈EndA^∗₀. Writem=|A0|and define A₂={a∈A₀|aϕⁿ= 1 for somen≥1},

A₃=A₀\A₂,

A₄={a∈A₃|aϕ∈A^∗₂aA^∗₂}.

LetΓ be the directed graph with vertex setA₀and edgesa−→bwheneverboccurs in aϕ. Thena∈A₂ if and only if there exists no infinite patha−→ · · · inΓ. This is equivalent to say there is no patha−→ · · · in Γ of lengthm, hence

A₂={a∈A₀|aϕ^m= 1} (4.1)

and is therefore effectively computable, and so areA₃ andA₄.

GivenB ⊆A, we denote byθ_A,B the homomorphismA^∗→B^∗ defined by aθ=

a ifa∈B 1 otherwise

Lemma 4.1 ([14]). Let ϕ∈EndA^∗₀ andm=|A₀|. Then Fixϕ= (A₄ϕ^m)^∗. Proof. Leta∈A₄. Thenaϕ=uavfor someu, v∈A^∗₂. It follows from (4.1) that

aϕ^m+1= (uav)ϕ^m= (uϕ^m)(aϕ^m)(vϕ^m) =aϕ^m, henceA₄ϕ^m⊆Fixϕand so (A₄ϕ^m)^∗⊆Fixϕ.

To prove the opposite inclusion, letθ=θ_A0,A3. Takeu∈Fixϕ. Thenuϕθ=uθ.

SinceA₂ϕ⊆A^∗₂, we get uθϕθ=uϕθ=uθand souθ∈Fix (ϕθ). Since 1∈/A₃ϕθ, it follows easily that uθ∈ A^∗₄. Hence u∈ (A₂∪A₄)^∗ and so u= uϕ^m ∈ (A₂∪ A₄)^∗ϕ^m=A^∗₄ϕ^m= (A₄ϕ^m)^∗ as required.

PEDRO V. SILVA

Note that, given an endomorphismϕofA^∗₀∗G, whereGis a group, the restriction ϕ|G is an endomorphism of G. Clearly,G is the (unique) maximal subgroup of A^∗₀∗G. To proceed with our task, we must be able to articulate the processes taking pace at the group components and the free monoid components. That is achieved through the next result, in circumstances which are far from general:

Theorem 4.2. Let M =A^∗₀∗Gbe finitely generated, where Gis a c-free group.

Let ϕ∈EndM be such that the equation

x=v(xϕ|_G)w (x∈G)

has an effectively constructible rational solution set for allv, w∈G. ThenFixϕis an effectively constructible rational submonoid of M.

Proof. LetGbe defined by the finite special confluent rewriting system A1 |R.

WriteA=A₀∪A₁ andθ=θ_A,A0. For everyu∈A^∗\A^∗₁, letuξdenote the longest factor of u in A₀A^∗∩A^∗A₀. Let A₂, A₃ and A₄ be defined as in the beginning of the section, replacing A by A₀ and ϕ by ψ = ϕ|A^∗₀θ. Write m = |A₀|. Let u=a₁. . . a_n∈A₄ψ^m= Fixψ witha₁, . . . , a_n∈A₀.

Our proof of Theorem4.2requires the following intermediate result:

Lemma 4.3. There exist effectively constructible rational subsetsL₁, . . . , L_n−1of Gsuch that the solution set of the equation

a₁x₁a₂. . . x_n−1a_n= (a₁x₁a₂. . . x_n−1a_n)ϕξ (x_i∈G) (4.2) is preciselyL₁×. . .×L_n−1.

Proof. Let 1≤ i₁ < . . . < i_k ≤ n denote all i ∈ {1, . . . , n} such that a_iψ = 1.

Then there exist 1 =j₁< . . . < j_k+1=n+ 1 such that a_irψ=a_jr. . . a_jr+1−1

for everyr∈ {1, . . . , k}. Moreover, there exist wordsp_s, w_s, q_s ∈A^∗₁ (for all ade- quate values ofs) such that

a_irϕ=p_jr−1a_jrw_jra_jr+1. . . w_jr+1−2a_jr+1−1q_jr+1−1

forr∈ {1, . . . , k}. We claim that equation (4.2) is equivalent to the system ofn−1 equations

⎧⎪

⎪⎨

⎪⎪

⎩

x_jr−1=q_jr−1(x_ir−1a_ir−1+1x_ir−1+1. . . a_ir−1x_ir−1)ϕp_jr−1 forr∈ {2, . . . , k}

x_i=w_i

wheneverj_r≤i < j_r+1−1 for somer∈ {1, . . . , k}.

Indeed, sincea₁. . . a_n ∈Fixψ, all we need is to find out necessary and sufficient conditions that thex_i must satisfy in equation (4.2). Let i∈ {1, . . . , n−1}, and

consider x_i in the left hand side of (4.2). Clearly, if j_r ≤i < j_r+1−1 for some r∈ {1, . . . , k}, thenx_i is fully determined bya_jrϕthroughx_i =w_i.

Otherwise,i=j_r−1 for some r∈ {2, . . . , k}. In this case, we need to compute which part of the right hand side of (4.2) eventually determinesx_jr−1: the longest suffix ofa_ir−1ϕinA^∗₁, the longest prefix ofa_irϕinA^∗₁, and the image of the factor betweena_ir−1 anda_ir. This proves the claim.

Next we show that our system is equivalent to a system of equations of the form (x_i =y_i)i=1,...,n−1, where

y_i∈A^∗₁ ∪ A^∗₁(x_iϕ)A^∗₁

for i= 1, . . . , n−1. For that purpose, we define a directed graph Λ with vertex set{1, . . . , n−1} and edges

(j_r−1)−→i wheneveri_r−1≤i < i_r(r= 2, . . . , k).

Clearly, ifj has outdegree 0, thenx_j =w_j is an equation in the original system.

It follows easily that, if there is no infinite pathj−→ · · · inΛ, thenx_j is uniquely determined in the solution set of (4.2) (if it is nonempty). Thus we need to discuss the structure of infinite paths in Λ. Since Λ is finite, such an infinite path must always contain a cycle. We show next that all cycles inΛhave length 1.

Suppose that

(j_r1−1)−→(j_r2 −1)−→ · · · −→(j_rs−1) = (j_r1−1)

is a cycle of lengths−1 inΛ. We claim that there exists a loop (j_r1−1)−→(j_r1−1) in Λ, which is equivalent to

i_r1−1≤j_r1−1< i_r1. (4.3) Suppose first thati_r1−1> j_r1−1. It suffices to show that

i_rt−1> j_rt−1 implies (r_t< r_t+1 andi_rt+1−1> j_rt+1−1) fort= 1, . . . , s−1 to derive a contradiction fromr_s=r₁. Indeed, assume thati_rt−1> j_rt−1. Since (j_rt −1)−→(j_rt+1 −1) is an edge of Λ, we have i_rt−1 ≤ j_rt+1 −1 < i_rt and so i_rt−1 < j_rt+1 ≤ i_rt. Hence j_rt ≤ i_rt−1 < j_rt+1 and so r_t < r_t+1. It follows that i_rt+1−1 ≥ i_rt > j_rt+1−1 and so our implication holds, yielding the desired contradiction.

Suppose now that j_r1 −1 ≥i_r1. Similarly to the preceding case, it suffices to show that

j_rt−1≥i_rt implies (r_t> r_t+1 andj_rt+1−1≥i_rt+1) fort= 1, . . . , s−1 to derive a contradiction fromr_s =r₁. Indeed, assume thatj_rt−1 ≥i_rt. Since i_rt−1≤j_rt+1−1< i_rt, we getj_rt−1≥i_rt > j_rt+1−1 and so r_t> r_t+1. It follows thatj_rt+1−1≥i_rt−1≥i_rt+1, yielding the desired contradiction.

PEDRO V. SILVA

Thus (4.3) holds and so there exists a loop (j_r1 −1)−→(j_r1 −1) in Γ. It is immediate that no vertex of Λ can have indegree>1, hencej_rs−1 −1 = j_r1−1 and sor_s−1=r₁, yieldings= 2. Therefore all cycles inΛ have length 1.

In view of the indegree property, it follows that the unique infinite pathj−→ · · · inΛ, if such a path exists, consists of infinitely many tours of the loopj−→j: you cannot enter a loop unless you have always been there.

Now take an equation of the form

x_jr−1=q_jr−1(x_ir−1a_ir−1+1x_ir−1+1. . . a_ir−1x_ir−1)ϕp_jr−1

forr∈ {2, . . . , k}in our original system. If there is no loopj_r−1−→j_r−1 inΛ, then the equationsx_i =w_i will eventually determine a unique possible value forx_jr−1 and we can replace the above equation by some other of the formx_jr−1=w_jr−1 for somew_jr−1∈A^∗₁.

Finally, assume that there is a loop j_r −1−→jr −1 in Λ. Then x_jr−1 ∈ {x_ir−1, . . . , x_ir−1} and all the other variables are bound to be eventually deter- mined by the equationsx_i =w_i. Since

q_jr−1, a_ir−1+1ϕ, . . . , a_ir−1ϕ, p_jr−1∈A^∗₁

are constants, we can replace our equation by another of the form x_jr−1 = v_jr−1(x_jr−1ϕ)z_jr−1 for somev_jr−1, z_jr−1∈A^∗₁. Note also that this construction of the new system is an effective procedure since it consists of successively replacing somex_i by w_i in the other equations. Therefore we can now concentrate on the new system (x_i =y_i)i=1,...,n−1. Since each variable occurs just in one equation, and no equation contains more than one single variable, the solution set will come out as a direct productL₁×. . .×Ln−1, whereL_iis the solution set of the equation containingx_i. If this equation is of the formx_i=w_i, thenL_i={wi}is (trivially) an effectively constructible rational subset of G. If the equation is of the form x_i=v_i(x_iϕ)z_i, then the claim follows from the Lemma’s hypothesis onϕ|G. Back to the proof of Theorem4.2, we remark that there existp, q∈Gsuch that

(a₁x₁a₂. . . x_n−1a_n)ϕ=p((a₁x₁a₂. . . x_n−1a_n)ϕξ)q

for every solution (x₁, . . . , x_n−1)∈L₁×. . .×L_n−1. Indeed, the prefix erased byξ is (a₁x₁. . . a_i1−1x_i1−1)ϕv_i1 and is uniquely determined sincex₁, x₂, . . . , x_i1−1 are uniquely determined as well. A symmetric argument applies to suffixes.

Back to the proof of Theorem 4.2, take u ∈A^∗. Note thatu ∈Fixϕ implies uθ∈Fixψ, hence we may restrict our attention to this latter condition. In view of Lemma4.1, we may writeuθ=u₁. . . u_t for someu₁, . . . , u_t∈A₄ψ^m. Ift= 0, thenϕ|_G∈EndGimplies thatu∈Fixϕif and only ifu∈Fixϕ|_G, and we can use the theorem’s hypothesis withv =w= 1. Therefore we only need to concentrate on the caset >0.

Write

u_i=a_i1a_i2. . . a_ini (a_ij ∈A₀)