Deciding the value 1 problem for probabilistic leaktight automata

HAL Id: hal-01140262

https://hal.archives-ouvertes.fr/hal-01140262

Submitted on 8 Apr 2015

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Open licence - etalab|

automata

Nathanaël Fijalkow, Hugo Gimbert, Edon Kelmendi, Youssouf Oualhadj

To cite this version:

Nathanaël Fijalkow, Hugo Gimbert, Edon Kelmendi, Youssouf Oualhadj. Deciding the value 1 problem

for probabilistic leaktight automata. Logical Methods in Computer Science, Logical Methods in

Computer Science Association, 2015, pp.37. �10.2168/LMCS-2014-994�. �hal-01140262�

NATHANA¨ EL FIJALKOW, HUGO GIMBERT, EDON KELMENDI, AND YOUSSOUF OUALHADJ LIAFA, Universit´e Denis Diderot - Paris 7, France, and University of Warsaw, Poland.

e-mail address: nath@liafa.univ-paris-diderot.fr LaBRI, CNRS, Bordeaux, France.

e-mail address: hugo.gimbert@labri.fr LaBRI and Universit´e de Bordeaux, France.

e-mail address: edon.kelmendi@labri.fr Universit´e Paris-Est, LACL, France.

e-mail address: youssouf.oualhadj@lacl.fr

Abstract. The value 1 problem is a decision problem for probabilistic automata over finite words: given a probabilistic automaton, are there words accepted with probability arbitrarily close to 1? This problem was proved undecidable recently; to overcome this, several classes of probabilistic automata of different nature were proposed, for which the value 1 problem has been shown decidable. In this paper, we introduce yet another class of probabilistic automata, called leaktight automata, which strictly subsumes all classes of probabilistic automata whose value 1 problem is known to be decidable.

We prove that for leaktight automata, the value 1 problem is decidable (in fact, PSPACE- complete) by constructing a saturation algorithm based on the computation of a monoid abstracting the behaviours of the automaton. We rely on algebraic techniques developed by Simon to prove that this abstraction is complete. Furthermore, we adapt this saturation algorithm to decide whether an automaton is leaktight.

Finally, we show a reduction allowing to extend our decidability results from finite words to infinite ones, implying that the value 1 problem for probabilistic leaktight parity automata is decidable.

1998 ACM Subject Classification: Probabilistic computation.

Key words and phrases: Probabilistic automata, Value 1 problem, Algebraic Techniques in Automata Theory.

A preliminary version appeared in LiCS’2012 [FGO12]. The sections about probabilistic automata over infinite words and the comparisons with structurally simple automata are new. The latter is mostly due to Edon Kelmendi. This project was supported by the french ANR project ”Stoch-MC” as well as ”LaBEX CPU” of Universit´e de Bordeaux.

LOGICAL METHODS

IN COMPUTER SCIENCE DOI:10.2168/LMCS-???

c

Fijalkow, Gimbert, Kelmendi, and Oualhadj Creative Commons

1

Introduction

Probabilistic automata. Rabin invented a very simple yet powerful model of probabilistic machine called probabilistic automata, which, quoting Rabin, “are a generalization of finite deterministic automata” [Rab63]. A probabilistic automaton has a finite set of states and reads input words from a finite alphabet. The computation starts from the initial state and consists in reading the input word sequentially; the state is updated according to transition probabilities determined by the current state and the input letter. The probability to accept a finite input word is the probability that the computation ends in one of the final states.

Probabilistic automata, and more generally partially observable Markov decision pro- cesses and stochastic games, are a widely studied model of probabilistic machines used in many fields like software verification [BBG12, CDHR07], image processing [CK97], compu- tational biology [DEKM99] and speech processing [Moh97]. As a consequence, it is crucial to understand which decision problems are algorithmically tractable for probabilistic au- tomata. From a language-theoretic perspective, several algorithmic properties of probabilis- tic automata are known: while language emptiness is undecidable [Paz71, Ber74, GO10], functional equivalence is decidable [Sch61, Tze92] as well as other properties [CMRR08].

Our initial motivation for this work comes from control and game theory: we aim at solving algorithmic questions about partially observable Markov decision processes and stochastic games. For this reason, we consider probabilistic automata as machines controlled by a blind controller, who is in charge of choosing the sequence of input letters in order to maximize the acceptance probability. While in a fully observable Markov decision process the controller can observe the current state of the process to choose adequately the next input letter, a blind controller does not observe anything and its choice depends only on the number of letters already chosen. In other words, the strategy of a blind controller is an input word of the automaton.

The value of a probabilistic automaton. With this game-theoretic interpretation in mind, we define the value of a probabilistic automaton as the supremum acceptance probability over all input words, and we would like to compute this value. Unfortunately, as a consequence of Paz undecidability result, the value of an automaton is not computable in general. However, the following decision problem was conjectured by Bertoni [Ber74] to be decidable:

Value 1 problem: Given a probabilistic automaton, does it have value 1? In other words are there input words whose acceptance probability is arbitrarily close to 1?

Recently, the second and fourth authors of the present paper proved that the value 1 problem is undecidable [GO10].

Our result. We introduce a new class of probabilistic automata, called leaktight au- tomata, for which the value 1 problem is decidable. This subclass strictly subsumes all known subclasses of probabilistic automata sharing this decidability property and has good closure properties. Our algorithm to decide the value 1 problem computes in polynomial space a finite monoid whose elements are directed graphs and checks whether it contains a certain type of elements that are value 1 witnesses.

Related works. Introducing subclasses of probabilistic automata to cope with un-

decidability results has been a fruitful and lively topic recently. We discuss some of them

here.

The first subclass which was introduced specifically to decide the value 1 problem are the

♯-acyclic automata [GO10]. Later on, Chatterjee and Tracol [CT12] introduced structurally simple automata, which are probabilistic automata satisfying a structural property (related to the decomposition-separation theorem from probability theory), and proved that the value 1 problem is decidable for structurally simple automata. At the same time, a subset of the authors introduced leaktight automata, and proved a similar result. As we shall see, both ♯-acyclic and structurally simple automata are leaktight, hence our results extend both [GO10] and [CT12].

Quite recently, Chadha, Sistla and Viswanathan introduced the subclass of hierarchical automata [CSV11], and showed that over infinite words, they recognize exactly the class of ω-regular languages. As we shall see, hierarchical automata are leaktight, hence as a consequence of our result, the value 1 problem is decidable for hierarchical automata.

Proof techniques. Our proof techniques totally depart from the ones used in [CSV11, CT12, GO10]. We make use of algebraic techniques and in particular Simon’s factorization forest theorem, which was used successfully to prove the decidability of the boundedness problem for distance automata [Sim94], and extended models as desert automata and B- automata [Kir05, Col09]

Outline. Basic definitions are given in Section 1.

In Section 2, we introduce the Markov monoid and the Markov monoid algorithm for the value 1 problem; since the problem is in general undecidable, the algorithm is incomplete:

a positive answer implies that the automaton has value 1, but a negative answer gives no guarantee.

In Section 3, we define the class of leaktight automata and show that the leaktight property is a sufficient condition for this algorithm to be complete; in particular, this implies that the value 1 problem is decidable for leaktight automata.

In Section 4, we show that the Markov monoid algorithm runs in polynomial space, and obtain as a corollary that the value 1 problem for leaktight automata is PSPACE-complete.

Furthermore, we extend the Markov monoid algorithm to check at the same time whether an automaton is leaktight and whether in such case it has value 1.

In Section 5, we further investigate the class of leaktight automata: we provide examples of leaktight automata and show that all subclasses of probabilistic automata whose value 1 problem is known to be decidable are leaktight.

In Section 6, we give a general theorem allowing to extend the decidability results from finite words to infinite words.

1. Definitions

1.1. Probabilistic automata. We fix A a finite alphabet. A (finite) word u is a (possibly empty) sequence of letters u = a ₀ a ₁ · · · a _n−1 , the set of finite words is denoted by A ^∗ . For i ≤ j we denote by u[i, j] the subword a i · · · a _j−1 , and u <p = u[0, p] = a ₀ a ₁ · · · a _p−1 .

Let Q be a finite set of states. A probability distribution over Q is a function δ : Q → [0, 1] such that P

q∈Q δ(q) = 1; we often see δ as a row vector of size |Q|. We denote by

1

3 ·q + ² ₃ · q ^′ the distribution that picks q with probability ¹ ₃ and q ^′ with probability ² ₃ , and by

q the trivial distribution picking q with probability 1. For a subset R of states, the uniform

distribution over R picks each state in R with probability _|R| ¹ . The support of a distribution

δ is the set of states picked with positive probability, i.e. Supp(δ) = {q ∈ Q | δ(q) > 0}.

Finally, the set of probability distributions over Q is D(Q).

Definition 1.1 (Probabilistic automaton). A tuple A = (Q, q ₀ , ∆, F ) represents a proba- bilistic automaton, where Q is a finite set of states, q ₀ ∈ Q is the initial state, ∆ defines the transitions and F ⊆ Q is the set of accepting states.

The transitions of a probabilistic automaton are given by a function ∆ : Q×A → D(Q), where ∆(q, a) is the probability distribution obtained by reading the letter a from the state q. The function ∆ induces the function ∆ ^′ : D(Q) × A → D(Q), where ∆ ^′ (δ, a) = P

q∈Q δ(q) · ∆(q, a). Going further, ∆ naturally extends to ∆ ^∗ : D(Q) × A ^∗ → D(Q) by induction: for a letter a ∈ A, we set ∆ ^∗ (δ, a) = ∆ ^′ (δ, a), and for an input word u = av, we set ∆ ^∗ (δ, u) = ∆ ^∗ (∆ ^′ (δ, a), v). Intuitively, ∆ ^∗ (δ, u) is the probability distribution obtained by reading the word u starting at the initial probability distribution δ. From now on, we will make no difference between ∆, ∆ ^′ and ∆ ^∗ , and denote the three of them by ∆.

We denote by P _A (s − → ^u t) the probability to go from state s to state t reading u on the automaton A, i.e. ∆(s, u)(t). Then P _A (s − → ^u T ) is defined as P

t∈T P _A (s − → ^u t). Finally, the acceptance probability of a word u ∈ A ^∗ by A is P _A (q ₀ − → ^u F ), which we denote by P _A (u).

For computational purposes, we assume that each value is a rational number given by two integers in binary decomposition.

Definition 1.2 (Value). The value of a probabilistic automaton A, denoted by val(A), is the supremum acceptance probability over all input words:

val(A) = sup

u∈A

P _A (u). (1.1)

1.2. The value 1 problem. We are interested in the following decision problem:

Problem 1.3 (Value 1 Problem). Given a probabilistic automaton A, decide whether val(A) = 1.

The value 1 problem can be reformulated using the notion of isolated cut-point intro- duced by Rabin in his seminal paper [Rab63]: an automaton has value 1 if and only if the cut-point 1 is not isolated.

0 L

L

R

R

a

b,

a, 1 − x b

a, x a, b

b,

a, x b

a, 1 − x a, b

Figure 1: This automaton has value 1 if and only if x > ¹ ₂ .

The automaton depicted on figure 1 has value 1 if and only if x > ¹ ₂ (a similar example appears in [BBG12]). The input alphabet is A = {a, b}, the initial state is the central state 0 and the unique final state is L ₂ .

We describe the behaviour of this automaton. After reading one b, the distribution is uniform over L ₁ , R ₁ . To reach L ₂ , one needs to read a b from the state L ₁ , but on the right-hand side this leads to the non-accepting absorbing state R 2 . In order to maximize the probability to reach L ₂ , one tries to “tip the scales” to the left.

If x ≤ ¹ ₂ , there is no hope to achieve this: reading a letter a gives more chance to stay in R 1 than in L 1 thus all words are accepted with probability at most ¹ ₂ , and val(A) = ¹ ₂ .

However, if x > ¹ ₂ then we show that A has value 1.

We have:

P _A (0 −−→ ^ba

L ₁ ) = 1

2 · x ⁿ and P _A (0 −−→ ^ba

R ₁ ) = 1

2 · (1 − x) ⁿ

We fix an integer N and analyse the action of reading (ba ⁿ ) ^N · b: there are N “rounds”, each of them corresponding to reading ba ⁿ from 0. In a round, there are three outcomes:

winning (that is, remaining in L ₁ ) with probability p n = ¹ ₂ · x ⁿ , losing (that is, remaining in R ₂ ) with probability q n = ¹ ₂ · (1 − x) ⁿ , or going to the next round (that is, reaching 0) with probability 1 − (p _n + q _n ). If a round is won or lost, then the next b leads to an accepting or rejecting sink; otherwise it goes on to the next round, for N rounds. Hence:

P _A ((ba ⁿ ) ^N · b) = P N

k=1 (1 − (p n + q n )) ^k−1 · p n

= p n · ^1−(1−(p _1−(1−(p

^+q

⁾⁾

n

+q

))

= ₁₊ ¹

pn

· 1 − (1 − (p _n + q _n )) ^N

We now set N = 2 ⁿ . A simple calculation shows that the sequence ((1−(p n +q n )) ²

) _n∈ N

converges to 0 as n goes to infinity. Furthermore, if x > ¹ ₂ then ^1−x _x < 1, so _p ^q

n

= ^1−x _x n

converges to 0 as n goes to infinity. It follows that the acceptance probability converges to 1 as n goes to infinity. Consequently:

lim n P _A ((ba ⁿ ) ²

· b) = 1.

This example witnesses two surprising phenomena:

• the value is discontinuous with respect to the transition probabilities, as for x = ¹ ₂ the value is ¹ ₂ , and for x > ¹ ₂ the value is 1;

• the sequence of words ((ba ⁿ ) ²

·b) n∈ N witnessing the value 1 involves two convergence speeds: indeed, the words a ⁿ b are repeated an exponential number of times, namely 2 ⁿ . One can show that repeating only n times does not lead to words accepted with arbitrarily high probability.

1.3. Recurrent states and idempotent words. We fix A a probabilistic automaton, and define two main notions: recurrent states and idempotent words.

Definition 1.4 (Induced Markov chain). Let u be a finite word, it induces a Markov chain M A,u whose state space is Q and transition matrix M A,u is defined by:

M _A,u (s, t) = P _A (s − → ^u t).

We rely on the classical notion of recurrent states in Markov chains.

Definition 1.5 (Recurrent state). A state s is u-recurrent if it is recurrent in M A,u . A finite word u is idempotent if reading once or twice the word u does not change qualitatively the transition probabilities.

Definition 1.6 (Idempotent word). A Markov chain is idempotent if its transition matrix M satisfies that for all states s, t:

M(s, t) > 0 ⇐⇒ M ² (s, t) > 0.

A finite word u is idempotent if M A,u is idempotent.

In the case of idempotent words, recurrence of a state is easily characterized, relying on simple graph-theoretical arguments:

Lemma 1.7. Let u be an idempotent word. A state s is u-recurrent if and only if for all states t we have:

M A,u (s, t) > 0 = ⇒ M A,u (t, s) > 0.

2. An (incomplete) algorithm for the value 1 problem

In this section, we present an algebraic algorithm for the value 1 problem, called the Markov monoid algorithm. Since the problem is undecidable, this algorithm does not solve the problem on all instances; we will show that it is correct, i.e. if it answers that an automaton has value 1, then the automaton does have value 1, but not complete, i.e. the converse does not hold. In the next section, we shall show that this algorithm is complete for the class of leaktight automata.

2.1. The Markov monoid algorithm. Our algorithm for the value 1 problem computes iteratively a set G of directed graphs called limit-words. Each limit-word is meant to represent the asymptotic effect of a sequence of input words, and some particular limit- words can witness that the automaton has value 1.

Definition 2.1 (Limit-word). A limit-word is a function u : Q ² → {0, 1}, such that for all states s, there exists a state t such that u(s, t) = 1.

In proofs and examples, we will adopt either of the two equivalent views for limit-words:

graphs over the set Q or square matrices over Q × Q.

We now explain the algorithm in detail. For the remainder of this section, we fix A a probabilistic automaton. Initially, G only contains those limit-words a that are induced by input letters a ∈ A :

∀s, t ∈ Q, (a(s, t) = 1 ⇐⇒ P _A (s − → ^a t) > 0), plus the limit-word 1 which is induced by the empty word:

∀s, t ∈ Q, (1(s, t) = 1 ⇐⇒ s = t).

The algorithm repeatedly adds new limit-words to G. There are two ways for that:

concatenating two limit-words or iterating an idempotent limit-word.

ALGORITHM 1: The Markov monoid algorithm.

Data: A probabilistic automaton.

G ← {a | a ∈ A} ∪ {1}.

repeat

if there is u , v ∈ G such that u · v ∈ G / then add u · v to G

end

if there is u ∈ G such that u is idempotent and u

∈ G / then add u

to G

end

until there is nothing to add;

if there is a value 1 witness in G then return true;

else

return false;

end

Concatenation of two limit-words The concatenation of two limit-words u and v is the limit-word u · v such that:

(u · v)(s, t) = 1 ⇐⇒ ∃q ∈ Q, u(s, q) = 1 and v(q, t) = 1.

In other words, concatenation corresponds to the multiplication of matrices with coefficients in the boolean semiring ({0, 1}, ∨, ∧). Intuitively, the concatenation of two limit-words corresponds to the concatenation of two sequences (u _n ) _n∈ N and (v _n ) _n∈ N of input words into the sequence (u n · v n ) _n∈ N .

We say that a limit-word u is idempotent if u ·u = u. The following lemma gives simple properties of idempotent limit-words.

Lemma 2.2. For all limit-words u:

• the limit-word u ^|Q|! is idempotent,

• if u is idempotent, then for all states r ∈ Q, there exists a state r ^′ ∈ Q such that u(r, r ^′ ) = 1 and r ^′ is u-recurrent.

The proof is omitted and relies on simple graph-theoretical arguments.

Iteration of an idempotent limit-word The iteration u ^♯ of a limit-word u is only defined when u is idempotent. It relies on the notion of u-recurrent state.

Definition 2.3 (u-recurrence). Let u be an idempotent limit-word. A state s is u-recurrent if for all states t, we have:

u(s, t) = 1 = ⇒ u(t, s) = 1.

Note that this echoes Lemma 1.7. The iterated limit-word u ^♯ removes from u any edge that does not lead to a recurrent state:

u ^♯ (s, t) = 1 ⇐⇒ u(s, t) = 1 and t is u-recurrent.

Intuitively, if a limit-word u represents a sequence (u n ) _n∈ ^N then its iteration u ^♯ repre- sents the sequence

u ^f(n) n

n∈ N for some increasing function f : N → N .

2.2. The Markov monoid and value 1 witnesses. The set of limit-words G computed by Algorithm 1 is called the Markov monoid.

Definition 2.4 (Markov monoid). The Markov monoid associated with A is the smallest set of limit-words containing {a | a ∈ A}∪{1} and closed under concatenation and iteration.

Two key properties, consistency and completeness, ensure that the limit-words of the Markov monoid reflect exactly every possible asymptotic effect of a sequence of input words.

Definition 2.5 (Reification). A sequence (u n ) _n∈ N of words reifies a limit-word u if for all states s, t, ( P _A (s −→ ^u

t)) _n∈ ^N converges and:

u(s, t) = 1 ⇐⇒ lim

n

P _A (s −→ ^u

t) > 0. (2.1) Note that if (u n ) _n∈ N reifies u, then any subsequence of (u n ) _n∈ N also does. We will use this simple observation several times.

Definition 2.6 (Consistency). A set of limit-words G is consistent with A if for every limit-word u ∈ G , there exists a sequence of input words (u n ) _n∈ N which reifies u.

Definition 2.7 (Completeness). A set of limit-words G is complete for A if for each se- quence of input words (u n )

, there exists u ∈ G such that for all states s, t ∈ Q:

lim sup

n

P _A (s −→ ^u

t) = 0 = ⇒ u(s, t) = 0. (2.2) Limit-words are useful to decide the value 1 problem because some of these are witnesses that the automaton has value 1.

Definition 2.8 (Value 1 witness). A value 1 witness is a limit-word u such that for all states t:

u(q ₀ , t) = 1 = ⇒ t ∈ F, (2.3)

where q ₀ is the initial state of the automaton.

Thanks to value 1 witnesses, the answer to the value 1 problem can be read in a consistent and complete set of limit-words:

Lemma 2.9 (A criterion for value 1). If G is consistent with A and complete for A, then A has value 1 if and only if G contains a value 1 witness.

Specifically:

• If G is consistent with A and contains a value 1 witness, then A has value 1,

• If G is complete for A and A has value 1, then A contains a value 1 witness.

Proof. We prove the first item. Assume that G is consistent with A and contains a value 1 witness u. Since G is consistent, there exists a sequence (u n ) _n∈ N reifying u. It follows from (2.1) and (2.3) that for t / ∈ F , we have lim n P _A (q ₀ −→ ^u

t) = 0. Thus lim n P _A (u n ) = P

t∈F lim n P _A (q 0 u

−→ t) = 1, so A has value 1.

We now prove the second item. Assume that G is complete for A and that A has value 1. Then there exists a sequence of words (u _n ) _n∈ N such that lim _n P _A (u _n ) = 1, i.e.

lim n P

t∈F P _A (q ₀ −→ ^u

t) = 1. Since for all n ∈ N , we have P

q∈Q P _A (q ₀ −→ ^u

q) = 1, then

for all t / ∈ F , lim sup _n P _A (q ₀ −→ ^u

t) = 0. Since G is complete, there exists a limit-word u

such that (2.2) holds. Then u is a value 1 witness: let t ∈ Q such that u(q ₀ , t) = 1, then

according to (2.2), lim sup _n P _A (q ₀ −→ ^u

t) > 0, hence t ∈ F .

2.3. Correctness of the Markov monoid algorithm.

Theorem 2.10. The Markov monoid associated with A is consistent.

This implies that if the Markov monoid algorithm outputs “true”, then for sure the input automaton has value 1. This positive result holds for every automaton (leaktight or not).

To prove Theorem 2.10, recall that the Markov monoid is the smallest set of limit- words containing {a | a ∈ A} ∪ {1} and closed under concatenation and iteration, hence it suffices to prove that the initial elements form a consistent set, and the closure under the two operations.

First, a is reified by the constant sequence (a) n∈ N , and 1 by the constant sequence (ε) _n∈ N . We state the closure under the two operations in the following proposition:

Proposition 2.11. Let (u n ) _n∈ N and (v n ) _n∈ N be two sequences that reify the limit-words u and v respectively. Then:

(1) the sequence of words (u _n · v _n ) _n∈ N reifies u · v,

(2) if u is idempotent, then there exists an increasing function f : N → N such that for all increasing functions g : N → N satisfying g ≥ f , the sequence

u ⁿ _g(n)

n∈ N reifies the limit-word u ^♯ .

The statement about iteration is stronger than required: the existence of f such that (u ⁿ _f(n) ) n∈ N reifying the limit-word u ^♯ is enough to prove Theorem 2.10. However, we will use this stronger result later on (in Section 5.4).

Proof.

(1) Let w _n = u _n · v _n . Then (w _n ) _n∈ N reifies u · v, since:

P _A (s −−→ ^w

t) = X

r∈Q

P _A (s −→ ^u

r) · P _A (r −→ ^v

t).

(2) Consider the Markov chain M with state space Q and transition matrix M defined by M (s, t) = lim n P _A (s −→ ^u

t). Since (u n ) _n∈ N reifies u, we have u(s, t) = 1 if and only if M(s, t) > 0. First observe that since u is idempotent, the Markov chain M is aperiodic. According to standard results about finite Markov chains, this implies that the sequence of matrices (M ^k ) _k∈ N has a limit which we denote by M ^∞ , satisfying the following:

∀s, t ∈ Q, M ^∞ (s, t) > 0 = ⇒ t is recurrent in M. (2.4) By definition the sequence of matrices (M _A,u

) _{n∈ N} converges to M . Since the matrix product operation is continuous, for every k ∈ N ,

M _A,u ^k

n∈ N converges to M ^k . So for every k ≥ 1, there exists N k ∈ N such that for all p ≥ N k , ||M ^k − M _A,u ^k

|| _∞ ≤ ¹ _k . We define f : N → N by induction, so that f (k) is the maximum of f (k − 1) + 1 and of N k , ensuring that f is increasing. Then for any increasing function g : N → N satisfying g ≥ f , the sequence of matrices

M _A,u ⁿ

n∈ N converges to M ^∞ . We

prove that u ⁿ _g(n)

n∈ N reifies u ^♯ :

u ^♯ (s, t) = 1 ⇐⇒ u(s, t) = 1 and t is u-recurrent

⇐⇒ M (s, t) > 0 and t is recurrent in M

⇐⇒ M ^∞ (s, t) > 0

⇐⇒ lim

n

P _A (s ^u

n

−−−→

t) > 0,

where the first equivalence is by definition of the iteration, the second holds be- cause (u n ) _n∈ N reifies u, the third by definition of M ^∞ , and the fourth because M _A,u

g(n)

n∈ N converges to M ^∞ . This concludes the proof.

Note that completeness is not true in general; for instance, one can show that the Markov monoid of the automaton represented in figure 1 is not complete. The next section gives a sufficient condition for completeness: the leaktight property.

3. Decidability of the value 1 problem for leaktight automata In this section we establish our main result:

Theorem 3.1. The value 1 problem is decidable for leaktight automata.

The definition of leaktight automata is given in the next subsection. For now (in this section), we are only interested in decidability issues; we will actually prove in Section 4 that the value 1 problem is PSPACE-complete for leaktight automata.

Note that as observed in the literature [BBG12, CSV13, Fij14], the value 1 problem for probabilistic automata over finite words is equivalent to the emptiness problem for proba- bilistic B¨ uchi automata with positive semantics, hence we obtain the following corollary:

Corollary 3.2. The emptiness problem is decidable for probabilistic B¨ uchi leaktight au- tomata with positive semantics.

The following theorem proves that the Markov monoid of a leaktight automaton is complete; since it is always consistent, by Lemma 2.9, the Markov monoid algorithm solves the value 1 problem for leaktight automata.

Theorem 3.3. If a probabilistic automaton is leaktight then its Markov monoid is complete.

The remainder of this section is devoted to the proof of Theorem 3.3. We first define

the leaktight property, and extend the Markov monoid. This extended version allows to

state an algebraic characterization of the leaktight property. Then, the technical core of the

proof relies on a subtle algebraic argument based on the existence of ♯-factorization trees of

bounded height [Sim90, Sim94, Col09, Tor11].

3.1. Leaks. The undecidability of the value 1 problem comes from the necessity to compare parallel convergence rates in order to track down vanishing probabilities. Comparing two convergence rates may require to compare the decimals of the rates up to an arbitrary pre- cision, which in turn can encode a Post correspondence problem, hence the undecidability.

One of the phenomena that makes tracking vanishing probabilities difficult are leaks.

A leak occurs in an automaton when a sequence of words turns a set of states C ⊆ Q into a recurrence class C on the long run, but on the short run, some of the probability of the recurrence class is “leaking” to a different recurrence class.

L

L

0 b

a

b a

a, b

a a

· b

L

L

0 ε

Figure 2: (a ⁿ · b) _n∈ N is a leak from L ₁ to L ₂ .

Such leaks occur in the automaton depicted in the left hand side of figure 2 with the input sequence (a ⁿ b) _n∈ N . As n grows large, the probability to reach L ₂ from L ₁ while reading the input word a ⁿ b vanishes, thus the sets {L ₁ } and {L ₂ } are two different recurrence classes on the long run (i.e. asymptotically), however on the short run remains a small yet positive probability to reach L 2 from L 1 .

The right hand side of figure 2 shows the asymptotic behaviour of reading (a ⁿ b) _n∈ N . Since the automaton in figure 1 contains two symmetric parts identical to figure 2, it features one leak on the left hand side and another in the right hand side. As a consequence, the real asymptotic behaviour is complex and depends on the compared speeds of these leaks.

An automaton without leak is called a leaktight automaton. In this section we prove that the value 1 problem is decidable when restricted to the subclass of leaktight automata.

The formal definition of a leak is as follows:

Definition 3.4 (Leak). Let (u n ) _n∈ N be a sequence of idempotent words. Assume that the sequence of matrices P _A (u n ) converges to a limit M, that this limit is idempotent and denote M the assocaited Markov chain.

The sequence (u _n ) _n∈ N is a leak if there exist r, q ∈ Q such that the following three conditions hold:

(1) r and q are recurrent in M, (2) lim _n P _A (r −→ ^u

q) = 0,

(3) for all n ∈ N , P _A (r −→ ^u

q) > 0.

Definition 3.5 (Leaktight automata). A probabilistic automaton is leaktight if it has no leak.

Several examples of leaktight automata are given in Section 5.

3.2. The extended Markov monoid. The existence of leaks can be decided by a slight extension of the Markov monoid algorithm which keeps track of strictly positive transition probabilities.

Definition 3.6 (Extended limit-word). An extended limit-word is a couple (u, u ₊ ) of two limit-words, such that for all s, t ∈ Q, we have u(s, t) = 1 = ⇒ u ₊ (s, t) = 1.

As for limit-words, extended limit-words can be seen either as graphs over the set Q, or couples of square matrices over Q × Q. Such a graph has two different kind of edges: an edge (s, t) is “normal” if u(s, t) = 1, and is a +-edge if u(s, t) = 0 but u ₊ (s, t) = 1.

We define the concatenation and iteration operations for extended limit-words. The concatenation of two extended limit-words (u, u ₊ ) and (v, v ₊ ) is the component-wise con- catenation, i.e. (u · v, u ₊ · v ₊ ). The iteration of an extended limit-word (u, u ₊ ) is only defined when it is idempotent (i.e. component-wise idempotent), by (u, u ₊ ) ^♯ = (u ^♯ , u ₊ ).

Definition 3.7 (Extended Markov monoid). The extended Markov monoid is the smallest set of extended limit-words containing {(a, a) | a ∈ A} ∪ {(1, 1)} and closed under concate- nation and iteration.

Note that if (u, u ₊ ) is in the extended Markov monoid, then u is in the Markov monoid.

The essential difference between the Markov monoid and its extended version is that the extension keeps track of those edges that are deleted by successive iteration operations.

This serves two purposes: first, to characterize the leaktight property in algebraic terms, and second, to prove Theorem 3.3.

We state a consistency result for the extended Markov monoid, extending Theorem 2.10.

The proofs of both these results are similar and given only once.

Lemma 3.8. For each (u, u ₊ ) in the extended Markov monoid, there exists a sequence (u _n ) _n∈ N such that for all states s, t ∈ Q, ( P _A (s −→ ^u

t)) _n∈ N converges and:

u(s, t) = 1 ⇐⇒ lim

n P _A (s −→ ^u

t) > 0, (3.1)

for all n ∈ N ,

u ₊ (s, t) = 1 ⇐⇒ P _A (s −→ ^u

t) > 0

. (3.2)

3.3. Leak witnesses.

Definition 3.9 (Leak witness). An idempotent extended limit-word (u, u ₊ ) is a leak witness if there exist r, q ∈ Q such that the following three conditions hold:

(1) r and q are u-recurrent, (2) u(r, q) = 0,

(3) u ₊ (r, q) = 1.

Lemma 3.10. If a probabilistic automaton is leaktight, then its extended Markov monoid does not contain any leak witness.

Proof. Suppose that there is a leak witness (u, u ₊ ) in the extended Markov monoid: u and u ₊ are idempotent and there exists r, q ∈ Q such that r and q are u-recurrent, u(r, q) = 0 and u ₊ (r, q) = 1. We prove that there exists a leak.

Thanks to Lemma 3.8, there exists a sequence (u _n ) _n∈ N satisfying (3.1) and (3.2). Note

that since u ₊ is idempotent, (3.2) implies that for all n ∈ N , u n is idempotent.

Consider the Markov chain M with state space Q and transition matrix M defined by M (s, t) = lim n P _A (s −→ ^u

t). M is idempotent since u is idempotent and thanks to (3.1).

We show that (u _n ) _n∈ N is a leak. There are three conditions to be met.

First, r and q are recurrent in M: this follows from (3.1) and the fact that r and q are u-recurrent. Second, lim n P _A (r −→ ^u

q) = 0: this follows from (3.1) and the fact that u(r, q) = 0. Third, for all n ∈ N , P _A (r −→ ^u

q) > 0: this follows from (3.2) and the fact that u ₊ (r, q) = 1.

As we will show in the next section, the converse of Lemma 3.10 is also true, which gives an algebraic characterization of the leaktight property using the extended Markov monoid. However, the proof of the converse implication is more involved and requires the lower bound lemma (Lemma 3.15), which is the object of the next subsection.

3.4. Stabilization monoids and ♯-factorization trees. We now introduce the technical material required to state and prove the lower bound lemma. The key notions here are stabilization monoids and ♯-factorization trees.

Factorization trees for monoids have been introduced by Simon [Sim90]. Roughly speak- ing, Simon’s factorization theorem states that given a morphism φ : A ^∗ → M from the set of finite words over A to a finite monoid M , the following holds: for all words u, the com- putation of φ(u) can be factorized in a tree whose depth is bounded independently of the length of the word.

Simon later developed the notion of decomposition trees to solve the limitedness prob- lem for distance automata [Sim94]. To this end, he defined an iteration operation ♯ for monoids over the tropical semiring ( N ∪ {∞}, min, +). Then Kirsten extended this tech- nique to desert automata and the nested distance desert automata [Kir05]. After him, Colcombet generalized this approach by defining stabilization monoids [Col09], which are monoids equipped with an iteration operation, and proved the existence of ♯-factorization trees of bounded depth. The formal definition is as follows:

Definition 3.11 (Stabilization monoid). A stabilization monoid (M, ·, ♯) is a finite monoid (M, ·) equipped with an iteration operation ♯ : E(M) → E(M ), where E(M ) is the set of idempotents of M, such that:

(a · b) ^♯ · a = a · (b · a) ^♯ for a · b ∈ E(M) and b · a ∈ E(M ), (3.3)

(e ^♯ ) ^♯ = e ^♯ for e ∈ E(M), (3.4)

e ^♯ · e = e ^♯ for e ∈ E(M). (3.5)

Lemma 3.12. The extended Markov monoid is a stabilization monoid.

Proof. To start with, the extended Markov monoid is a monoid for the concatenation: 1 is the neutral element, and the concatenation is associative.

Now, let us prove the three properties required for the iteration operation ♯.

Proof of (3.3). Let (u, u ₊ ), (v, v ₊ ) such that (u · v, u ₊ · v ₊ ) and (v · u, v ₊ · u ₊ ) are idempotent. By definition ((u, u ₊ ) · (v, v ₊ )) ^♯ ·(u, u ₊ ) is equal to (u · v) ^♯ · u, u ₊ · v ₊ · u ₊

, and (u, u ₊ ) · ((v, v ₊ ) · (u, u ₊ )) ^♯ to u · (v · u) ^♯ , u ₊ · v ₊ · u ₊

. Let s, t ∈ Q, we have the following equivalence: (u · v) ^♯ · u

(s, t) = 1 if and only if:

there exists r, q ∈ Q, u(s, r) = 1 ∧ v(r, q) = 1 ∧ u(q, t) = 1 ∧ q is (u · v)-recurrent, (3.6)

and similarly, u · (v · u) ^♯

(s, t) = 1 if and only if:

there exists r, q ∈ Q, u(s, r) = 1 ∧ v(r, q) = 1 ∧ u(q, t) = 1 ∧ t is (v · u)-recurrent. (3.7) We show that (3.6) and (3.7) are equivalent. Assume (3.6), and prove that t is (v · u)- recurrent. Let p ∈ Q such that (v · u)(t, p) = 1. Since v is a limit-word, there exists ℓ ∈ Q such that v(p, ℓ) = 1. Observe that u(q, t) = 1, (v · u)(t, p) = 1 and v(p, ℓ) = 1, so (u · v) ² (q, ℓ) = 1. As u · v is idempotent, this implies (u · v)(q, ℓ) = 1. Since q is (u · v)- recurrent, we have (u · v)(ℓ, q) = 1. Altogether, v(p, ℓ) = 1, (u · v)(ℓ, q) = 1 and u(q, t) = 1 imply that (v · u) ² (p, t) = 1. As v · u is idempotent, this implies (v · u)(p, t) = 1, so t is (v · u)-recurrent, and (3.7) is proved. Conversely, assume (3.7), and prove that q is (u · v)- recurrent. Note that from v(r, q) = 1, u(q, t) = 1 and the fact that t is (v · u)-recurrent, we obtain that (v · u)(t, r) = 1. Let p ∈ Q such that (u · v)(q, p) = 1. Since u is a limit-word, there exists ℓ ∈ Q such that u(p, ℓ) = 1. Observe that v(r, q) = 1, (u · v)(q, p) = 1 and u(p, ℓ) = 1, so (v · u) ² (r, ℓ) = 1, and with (v · u)(t, r) = 1 this implies (v · u) ³ (t, ℓ) = 1.

As v · u is idempotent, this implies (v · u)(t, ℓ) = 1. Since t is (v · u)-recurrent, we have (v · u)(ℓ, t) = 1. Altogether, u(p, ℓ) = 1, (v · u)(ℓ, t) = 1, (v · u)(t, r) = 1 and v(r, q) = 1 imply that (u · v) ³ (p, q) = 1. As u · v is idempotent, this implies (u · v)(p, q) = 1, so q is (u · v)-recurrent, and (3.6) is proved. The property (3.3) follows.

Proof of (3.4). This boils down to proving (u ^♯ ) ^♯ = u ^♯ . This is clear from the definition of u ^♯ , since the notions of u-recurrence and u ^♯ -recurrence coincide.

Proof of (3.5). This boils down to proving u ^♯ · u = u ^♯ . It follows from the observation that if r ∈ Q is u-recurrent and u(r, t) = 1, then t is u-recurrent (under the assumption that u is idempotent).

Definition 3.13. Let A be a finite alphabet, (M, ·, ♯) a stabilization monoid and φ : A ^∗ → M a morphism into the submonoid (M, ·). A ♯-factorization tree of a word u ∈ A ^∗ is a finite unranked ordered tree, whose nodes have labels in A ^∗ × M and such that:

i) the root is labelled by (u, u), for some u ∈ M,

ii) every internal node with two children (called concatenation nodes) labelled by (u ₁ , u ₁ ) and (u ₂ , u ₂ ) is labelled by (u ₁ · u ₂ , u ₁ · u ₂ ),

iii) every internal node with three or more children (called iteration nodes) is labelled by (u 1 . . . u n , e ^♯ ) for some e ∈ E(M), and its children are labelled by (u 1 , e), . . . , (u n , e).

iv) every leaf is labelled by (a, a) where a is a letter, or (ε, 1).

Note that in a factorization tree, the second label is not always the image of the first component under φ; indeed, it is an element of the stabilization monoid (M, ·, ♯) whereas the image of a finite word under φ is an element of the submonoid (M, ·). However, the projection of second label into this submonoid (which consists in ignoring the operation ♯) is indeed the image of the first component under φ.

The following theorem was stated for the tropical semiring in [Sim94], and generalized in [Col09]. A simple proof can be found in [Tor11].

Theorem 3.14. Let A be a finite alphabet, (M, ·, ♯) a stabilization monoid and φ : A ^∗ → M

a morphism into the submonoid (M, ·). Every word u ∈ A ^∗ has a ♯-factorization tree whose

depth is less than 3 · |M |.

3.5. The lower bound lemma. We are ready to state and prove the lower bound lemma, which is the central argument in the proof of completeness of leaktight Markov monoids.

Lemma 3.15 (Lower bound lemma). Let A be a probabilistic automaton whose extended Markov monoid contains no leak witness. Let p _min the smallest non-zero transition proba- bility of A. Then for all words u ∈ A ^∗ , there exists (u, u ₊ ) in the extended Markov monoid such that, for all states s, t:

u ₊ (s, t) = 1 ⇐⇒ P _A (s − → ^u t) > 0, (3.8) u(s, t) = 1 = ⇒ P _A (s − → ^u t) ≥ p ² _min

. (3.9) Proof. Consider a finite word u ∈ A ^∗ ; by Theorem 3.14 applied to the extended Markov monoid G + associated with A (which is a stabilization monoid thanks to Lemma 3.12) and the morphism φ : A → M defined by φ(a) = (a, a), there exists a ♯-factorization tree of depth at most 3 · |G + |, whose root is labelled by (u, (u, u ₊ )) for some extended limit-word (u, u ₊ ).

The depth of a node in this tree is defined in a bottom-up fashion: the leaves have depth zero, and a node has depth one plus the maximum of the depths of its children.

We prove by a bottom-up induction (on h) that for every node (u, (u, u ₊ )) of this tree at depth h, for all states s, t:

u ₊ (s, t) = 1 ⇐⇒ P _A (s − → ^u t) > 0, (3.10) u(s, t) = 1 = ⇒ P _A (s − → ^u t) ≥ p ² _min

. (3.11) The case h = 0 is for leaves. Here, either u is a letter a and u = u ₊ = a, or u is the empty word ε and u = u ₊ = 1. Then both (3.10) and (3.11) hold.

Assume h > 0, there are two cases.

First case: a concatenation node labelled by (u, (u, u ₊ )) with two children labelled by (u ₁ , (u ₁ , u _+,1 )) and (u ₂ , (u ₂ , u _+,2 )). By definition u = u ₁ · u ₂ , u = u ₁ · u ₂ and u ₊ = u _+,1 · u _+,2 .

We first prove that (3.10) holds. Indeed, for s, t ∈ Q, u ₊ (s, t) = 1 if and only if there exists r ∈ Q such that u _+,1 (s, r) = 1 and u _+,2 (r, t) = 1. On the other side, since:

P _A (s − → ^u t) = X

r∈Q

P _A (s −→ ^u

r) · P _A (r −→ ^u

t),

then P _A (s − → ^u t) > 0 if and only if there exists r ∈ Q such that P _A (s −→ ^u

r) · P _A (r −→ ^u

t) > 0, which is equivalent to P _A (s −→ ^u

r) > 0 and P _A (r −→ ^u

t) > 0. We conclude with the induction hypothesis.

Now we prove that (3.11) holds. Let s, t ∈ Q such that u(s, t) = 1. Then there exists r ∈ Q such that u ₁ (s, r) = 1 and u ₂ (r, t) = 1. So:

P _A (s − → ^u t) ≥ P _A (s −→ ^u

r) · P _A (r −→ ^u

t) ≥ p ² _min

· p ² _min

= p ² _min

,

where the second inequality is by induction hypothesis. This completes the proof of (3.11).

Second case: an iteration node labelled by (u, (u ^♯ , u ₊ )) with k sons labelled by (u ₁ , (u, u ₊ )), . . . , (u _k , (u, u ₊ )). By definition, u = u ₁ · · · u _k , and (u, u ₊ ) is idempotent.

The proof that (3.10) holds is similar to the concatenation node case.

Now we prove that (3.11) holds. Let s, t ∈ Q such that u ^♯ (s, t) = 1. Since k ≥ 3:

P _A (s − → ^u t) ≥ P _A (s −→ ^u

t) · X

q∈Q

P _A (t −−−−−−→ ^u

^···u

q) · P _A (q −→ ^u

t). (3.12) To establish (3.11) we prove that:

P _A (s −→ ^u

t) ≥ p ² _min

, (3.13)

for all q ∈ Q, P _A (t −−−−−−→ ^u

^···u

q) > 0 = ⇒ P _A (q −→ ^u

t) ≥ p ² _min

. (3.14) We prove (3.13). Since u ^♯ (s, t) = 1, by definition u(s, t) = 1 and t is u-recurrent. The induction hypothesis for the node (u ₁ , (u, u ₊ )) implies that P _A (s −→ ^u

t) ≥ p ² _min

, i.e. (3.13).

Now we prove (3.14). For that we use the hypothesis that (u, u ₊ ) is not a leak witness.

Let q ∈ Q such that P _A (t −−−−−−→ ^u

^···u

q) > 0. By induction hypothesis for each child, (3.10) implies that u ^k−2 ₊ (t, q) = 1. Since u ₊ is idempotent, u ₊ (t, q) = 1. We argue that u(q, t) = 1.

Let ℓ ∈ Q a u-recurrent state such that u(q, ℓ) = 1. Then u ₊ (t, ℓ) = 1, and t, ℓ are u- recurrent. Since (u, u ₊ ) is not a leak witness, it follows that u(t, ℓ) = 1, which implies that u(ℓ, t) = 1 since t is u-recurrent. Together with u(q, ℓ) = 1, this implies u(q, t) = 1. Thus, by induction hypothesis and according to (3.11), P _A (q −→ ^u

t) ≥ p ² _min

, so (3.14) holds.

Now, putting (3.12), (3.13) and (3.14) altogether:

P _A (s − → ^u t) ≥ P _A (s −→ ^u

t) · X

q∈Q

P _A (t −−−−−−→ ^u

^···u

q) · P _A (q −→ ^u

t)

≥ p ² _min

· X

q∈Q

P _A (t −−−−−−→ ^u

^···u

q) · p ² _min

= p ² _min

, where the last equality holds because P

q∈Q P _A (t −−−−−−→ ^u

^···u

q) = 1. This completes the proof of (3.11).

To conclude, note that G + has less than 3 ^|Q|

elements.

3.6. Completeness of the Markov monoid algorithm for leaktight automata. In this subsection we rely on the lower bound lemma (Lemma 3.15) to prove Theorem 3.3.

Let A be a leaktight automaton. By Lemma 3.10, its extended Markov monoid does not contain any leak witness, hence Lemma 3.15 applies.

We prove the completeness of the Markov monoid associated with A. Let (u n ) _n∈ ^N be a sequence of finite words. By Lemma 3.15, for each word u _n there exists (u _n , u _+,n ) in the extended Markov monoid such that for all states s, t:

u _n (s, t) = 1 = ⇒ P _A (s −→ ^u

t) ≥ p ² _min

.

Since the set of limit-words is finite, there exists N ∈ N such that {n ∈ N | u _N = u _n } is infinite. To complete the proof, we prove that u N satisfies, for all states s, t:

lim sup P _A (s −→ ^u

t) = 0 = ⇒ u _N (s, t) = 0.

Assume lim sup P _A (s −→ ^u

t) = 0, then lim sup P _A (s −→ ^u

t) < p ² _min

for n sufficiently large.

Since u N = u n for infinitely many n ∈ N , this implies u N (s, t) = 0, which completes the proof of Theorem 3.3.

4. Properties of leaktight automata

In this section, we extend the algorithm presented in Section 2, and investigate its running complexity. The extended algorithm has two features: first, it checks at the same time whether an automaton is leaktight and whether it contains a value 1 witness, second, it runs in polynomial space.

We present an algebraic characterization of the leaktight property based on the extended Markov monoid, allowing the extended algorithm to check the leaktight property. For the complexity, one needs a deeper understanding of the Markov monoid; in this section, we will show a linear bound on the ♯-height, allowing to compute the extended Markov monoid in polynomial space. As a corollary, we obtain that the value 1 problem for leaktight automata is PSPACE-complete.

4.1. Characterization of the leaktight property. In this subsection, we show the con- verse of Lemma 3.10, which implies the following theorem, characterizing the leaktight property in algebraic terms.

Theorem 4.1. An automaton A is leaktight if and only if its extended Markov monoid does not contain any leak witness.

Lemma 3.15 is instrumental in the proof of this lemma.

Proof. We prove that if the extended Markov monoid of an automaton A does not contain any leak witness, then A is leaktight. The converse was proved in Lemma 3.10.

Assume A has a leak (u n ) _n∈ ^N , we show that its extended Markov monoid contains a leak witness. Consider the Markov chain M with state space Q and transition matrix M defined by M (s, t) = lim n P _A (s −→ ^u

t). By assumption M is idempotent.

By definition of a leak:

r and q are recurrent in M, (4.1)

M (r, q) = 0, (4.2)

for all n ∈ N , P _A (r −→ ^u

q) > 0. (4.3) Assume towards contradiction that the extended Markov monoid does not contain any leak witness, then Lemma 3.15 applies. For each word u n , there exists (u n , u _+,n ) in the extended Markov monoid such that for all states s, t:

u _+,n (s, t) = 1 ⇐⇒ P _A (s −→ ^u

t) > 0, (4.4) u n (s, t) = 1 = ⇒ P _A (s −→ ^u

t) ≥ p ² _min

. (4.5) Since the extended Markov monoid is finite, there exists N ∈ N such that for infinitely many n ∈ N , we have (u _N , u _+,N ) = (u _n , u _+,n ).

Note that since each u n is idempotent, (4.4) implies that each u _+,n is idempotent as

well.

Let (v, v ₊ ) = (u N , u _+,N ) ^|Q|! . The power |Q|! ensures that u ^|Q|! _N is idempotent, by Lemma 2.2. Since u _+,N is idempotent, v ₊ = u _+,N . Also, since v is idempotent, there exists r ^′ and q ^′ which are v-recurrent, such that v(r, r ^′ ) = 1 and v(q, q ^′ ) = 1, again thanks to Lemma 2.2.

Now, we prove that (v, v ₊ ) is a leak witness:

r ^′ and q ^′ are v-recurrent, (4.6)

v(r ^′ , q ^′ ) = 0, (4.7)

v ₊ (r ^′ , q ^′ ) = 1. (4.8)

Let η = p ² _min

and K = |Q|!.

Observe that for all states s, t, we have v(s, t) = 1 = ⇒ M (s, t) > 0:

v(s, t) = 1

= ⇒ u ^K _N (s, t) = 1 (by definition of v)

= ⇒ u ^K _n (s, t) = 1 for infinitely many n (by definition of N )

= ⇒ P _A (s ^u

Kn

−−→ t) ≥ η ^K for infinitely many n (by (4.5))

= ⇒ lim

n P _A (s ^u

K

−−→

t) ≥ η ^K

= ⇒ M ^K (s, t) > 0 (by definition of M)

= ⇒ M (s, t) > 0 (since M is idempotent).

First, (4.6) is by definition of r ^′ and q ^′ .

We prove (4.7). Towards contradiction, assume that v(r ^′ , q ^′ ) = 1. Then M (r ^′ , q ^′ ) > 0, so together with M(r, r ^′ ) > 0 (which follows from v(r, r ^′ ) = 1) this implies M ² (r, q ^′ ) > 0, so M(r, q ^′ ) > 0 as M is idempotent. Since M(q, q ^′ ) > 0 (which follows from v(q, q ^′ ) = 1) and q is recurrent in M, we have M (q ^′ , q) > 0. This implies M ² (r, q) > 0, and M (r, q) > 0 because M is idempotent, which contradicts (4.2).

We prove (4.8). Thanks to (4.3) and (4.4), we have u _+,N (r, q) = 1, i.e. v ₊ (r, q) = 1. Since M(r, r ^′ ) > 0 and r is recurrent in M , we have M (r ^′ , r) > 0, so (4.4) implies that u _+,N (r ^′ , r) = 1, i.e. v ₊ (r ^′ , r) = 1. Similarly, M (q, q ^′ ) > 0, so (4.4) implies that u _+,N (q, q ^′ ) = 1, i.e. v ₊ (q, q ^′ ) = 1. The three equalities v ₊ (r ^′ , r) = 1, v ₊ (r, q) = 1 and v ₊ (q, q ^′ ) = 1 imply v ³ ₊ (r ^′ , q ^′ ) = 1, and since v ₊ is idempotent v ₊ (r ^′ , q ^′ ) = 1.

It follows that (v, v ₊ ) is a leak witness, which completes the proof.

The immediate corollary of Theorem 4.1 is that checking whether an automaton is leaktight can be done by computing the extended Markov monoid and looking for leak witnesses, hence it is decidable.

4.2. The extended Markov monoid algorithm. Algorithm 2 computes the extended

Markov monoid, and looks for value 1 witnesses, which in the extended Markov monoid is

an extended limit-word (u, u ₊ ) such that u is a value 1 witness (in the Markov monoid). If

there is a value 1 witness, then the automaton has value 1, even if it is not leaktight, thanks

to Theorem 2.10. Otherwise, the algorithm looks for a leak witness; if there is no leak

witness, then the automaton is leaktight thanks to Theorem 4.1, and it does not have value

1 thanks to Theorem 3.3. In case there is a leak witness, the automaton is not leaktight, and nothing can be said.

ALGORITHM 2: The extended Markov monoid algorithm.

Data: A probabilistic automaton.

G

← {(a, a) | a ∈ A} ∪ {(1, 1)}.

repeat

if there is (u, u

), (v, v

) ∈ G

such that (u · v, u

· v

) ∈ G /

then add (u · v, u

· v

) to G

end

if there is (u, u

) ∈ G

such that (u, u

) is idempotent and (u

, u

) ∈ G /

then add (u

, u

) to G

end

until there is nothing to add;

if there is a value 1 witness in G

then return true;

else

if there is no leak witness in G

then return false;

else

return fail: the automaton is not leaktight;

end end

4.3. Parallel composition and PSPACE-hardness. The objective of this subsection is to prove the PSPACE-hardness of the value 1 problem for leaktight automata. To this end, we give a reduction from the emptiness problem of n deterministic automata. To prove that the reduction indeed constructs leaktight automata, we need to show that deterministic automata are leaktight, and the closure under parallel composition.

Proposition 4.2. Deterministic automata are leaktight.

Proof. For all limit-words u ∈ {a | a ∈ A} ∪ {1}, for all states s, there exists a unique state t such that u(s, t) = 1. In particular, each recurrence class is formed of only one state with a self-loop. This property is preserved by concatenation, and implies that the iteration operation is trivial, i.e. u ^♯ = u. Consequently, for all extended limit-words (u, u ₊ ) in the extended Markov monoid, we have u = u ₊ , which implies that there are no leak witnesses.

Definition 4.3 (Parallel composition). Consider two probabilistic automata, denoted A = (Q ^A , q ^A ₀ , ∆ ^A , F ^A ) and B = (Q ^B , q ₀ ^B , ∆ ^B , F ^B ). We assume that Q ^A and Q ^B are disjoint.

The parallel composition of A and B is:

A || B = (Q ^A ⊎ Q ^B , δ 0 , ∆ , F ^A ∪ F ^B ), where δ ₀ = ¹ ₂ · q ^A ₀ + ¹ ₂ · q ₀ ^B , and:

∆(q, a) =

( ∆ _A (q, a) if q ∈ Q _A ,

∆ _B (q, a) if q ∈ Q _B .

By definition, for u ∈ A ^∗ , we have P _A||B (u) = ¹ ₂ · P _A (u) + ¹ ₂ · P _B (u). Note that in this definition, we allowed an initial probability distribution rather than only one initial state. This could be avoided by adding a new initial state that leads to each previous initial state with probability half, but we do it here for technical convenience in the proof of the following proposition.

Proposition 4.4. The leaktight property is stable by parallel composition.

Proof. The extended Markov monoid G ₊ ^A||B of the parallel composition embeds into the direct product G ^A ₊ × G ₊ ^B of the extended Markov monoids of each automaton.

Note that for (u, u ₊ ) ∈ G ₊ ^A||B , if u(s, t) = 1, then either s, t ∈ Q ^A or s, t ∈ Q ^B , and similarly for u ₊ . Relying on this, we map (u, u ₊ ) ∈ G ₊ ^A||B to ((u, u ₊ )[A] , (u, u ₊ )[B]), where (u, u ₊ )[A] is the restriction to A and similarly for B. An easy induction on (u, u ₊ ) shows that this map is an embedding into G ^A ₊ × G ₊ ^B .

Consequently, the extended Markov monoid of the parallel composition contains a leak witness if and only if one of the extended Markov monoid contains a leak witness.

Now that we proved that deterministic automata are leaktight, and the closure under parallel composition, the PSPACE-hardness of the value 1 problem for leaktight automata is easy.

Proposition 4.5. The value 1 problem for leaktight automaton is PSPACE-hard.

Proof. We give a reduction from the following problem: given n deterministic automata over finite words, decide whether the intersection of the languages they accept is empty.

This problem is PSPACE-hard [Koz77].

The reduction is as follows: given n deterministic automata, we construct the parallel composition of the n automata, where each copy is reached with probability _n ¹ . This au- tomaton has value 1 if and only if the intersection of the languages is not empty, and is leaktight by Proposition 4.2 and Proposition 4.4.

4.4. Bounding the ♯-height in the Markov monoid. We now consider the running complexity of the extended Markov monoid algorithm. A na¨ıve argument shows that it terminates in less than 3 ^|Q|

iterations, since each iteration adds a new extended limit-word in the monoid and there are at most 3 ^|Q|

different limit-words. This gives an EXPTIME upper bound.

A better complexity can be achieved by looking for a value 1 witness or a leak witness in a non-deterministic way. The algorithm guesses the witness by its decomposition into concatenations and iterations. The key observation, made by Kirsten [Kir05] in the context of distance desert automata, is that the ♯-height, that is the number of nested applications of the iteration operation, can be restricted to at most |Q|.

Note that when dealing with ♯-height, it suffices to consider limit-words instead of extended limit-words, as by definition the second component of an extended limit-word does not contain any ♯.

Formally, we define the ♯-hierarchy inside the Markov monoid as follows:

S ₀ = h{a | a ∈ A} ∪ {1}i,

S _p+1 = hS p ∪ {u ^♯ | u ∈ E(S p )}i,

where hT i is the set of limit-words obtained as concatenation of limit-words in T .

Definition 4.6 (♯-height of a limit-word). The ♯-height of a limit-word u is the minimal p such that u ∈ S p .

Theorem 4.7. Every limit-word has ♯-height at most |Q|, i.e. the ♯-hierarchy collapses at level |Q|.

In the following, we adapt Kirsten’s proof from [Kir05] to the setting of probabilistic automata. Roughly speaking, the proof consists in associating a quantity to each idempotent element of the Markov monoid, and to show the following:

• the quantity is bounded above by |Q|.

• the quantity strictly decreases when iterating an unstable limit-word (i.e. if u ^♯ 6= u),

• the quantity does not increase when concatenating.

Let u be an idempotent limit-word, we define ∼ ^u the relation on Q by s ∼ ^u t if u(s, t) = 1 and u(t, s) = 1. Clearly, ∼ ^u is symmetric, and since u is idempotent, ∼ ^u is transitive. If for some state s there exists a state t such that s ∼ ^u t, then s ∼ ^u s since u is idempotent. Consequently, the restriction of ∼ ^u to the set

Z ^u = {s ∈ Q | s ∼ ^u s}

is reflexive, i.e. ∼ ^u is an equivalence relation on Z ^u . From now on by equivalence class of ∼ ^u we mean an equivalence class of ∼ ^u on Z u . We denote by [s] u the equivalence class of s, and by Cl(u) the set of equivalence classes of ∼ ^u . The quantity associated with u is

|Cl(u)|, the number of equivalence classes of ∼ ^u , that is the number of non-trivial connected components in the underlying graph of u. Note that |Cl(u)| ≤ |Q|.

Here are two useful observations.

Lemma 4.8.

• Let u, v be two limit-words and s, t, r ∈ Q. Then (u · v)(s, t) ≥ u(s, r) · v(r, t).

• Let u be an idempotent limit-word and s, t ∈ Q. There exists r ∈ Q such that u(s, t) = u(s, r) · u(r, r) · u(r, t).

Proof. The first claim is clear and follows from the equality:

(u · v)(s, t) = X

r∈Q

u(s, r) · v(r, t).

Consider now the second claim. For all states r ∈ Q, since u is idempotent we have:

u(s, t) = u ³ (s, t) = X

p,q∈Q

u(s, p) · u(p, q) · u(q, t) ≥ u(s, r) · u(r, r) · u(r, t).

Since u is idempotent, we have u = u ⁿ⁺² , so there exist s = r ₀ , . . . , r _n+2 = t such that u(s, t) = u(r ₀ , r ₁ ) · · · u(r _n+1 , r _n+2 ). By a counting argument, there exist i, j such that 1 ≤ i < j ≤ (n + 1) and r i = r j , denote it by r. We have:

u(s, r) = u ⁱ (s, r) ≥ u(r ₀ , r ₁ ) · · · u(r _i−1 , r _i ), u(r, r) = u ^j−i (r, r) ≥ u(r i , r _i+1 ) · · · u(r _j−1 , r j ), u(r, t) = u ^n+2−j (r, t) ≥ u(r _j , r _j+1 ) · · · u(r _n+1 , r _n+2 ).

Hence, u(s, r) · u(r, r) · u(r, t) ≥ u(r 0 , r 1 ) · · · u(r n+1 , r n+2 ) = u(s, t), and the second claim

follows.