*−∞* ⩾ *x**k* for all *k* *∈* [n]). Hence, we have S = *∪**K**⊂*[n]S_{∩}X*K* = tconv(*∪**K**⊂*[n]S_{∩}X*K*).

Therefore, the claim follows from Lemma 5.51. To prove the implication Theorem 5.5 (d)
to Theorem 5.5 (e), let S _{⊂}_{T}* ^{n}* be a projection of a tropical Metzler spectrahedron

*S ⊂*T

^{n}*×*T

^{n}*. By Proposition 4.14, there is a spectrahedron*

^{′}

**S***⊂*K

^{n+n}_{⩾}

_{0}

*such that val(S) =*

^{′}*S.*

Let * π*:K

^{n+n}

^{′}*→*K

^{n}*, π*:T

^{n+n}

^{′}*→*T

*denote the projections on the first*

^{n}*n*coordinates. Then val(π(

*)) =*

**S***π(val(*)) =

**S***π(S*) = S. The implication Theorem 5.5 (e) to Theorem 5.5 (a) follows trivially from the fact that projected spectrahedra are semialgebraic (Proposition 2.18) and convex.

**5.6** **Extension to general fields**

In this section we partially extend Theorem 5.5 to general real closed valued fields and prove Theorem 5.2. We want to do so using a model-theoretic argument. However, this requires to control the size of the spectrahedra constructed in the proof of Theorem 5.5. As a consequence, the proof is rather technical. We start by presenting a more eﬀective version of Proposition 5.34.

This variant allows us to give a uniform bounds on the complexity of describing the pieces
involved in the piecewise description of the operator *F*. To formalize this uniformity, we fix
rational matrices *B*^{(1)}*, . . . , B*^{(q)}, *B*^{(s)} *∈* Q^{m}^{s}^{×}* ^{n}* and, for every real vector

*c*= (c

^{(1)}

*, . . . , c*

^{(q)}),

*c*

^{(s)}

*∈*R

^{m}*, consider the set S*

^{s}

_{c}

_{⊂}_{R}

*defined as S*

^{n}*:=*

_{c}^{∪}

^{q}

_{s=1}*{x∈*R

*:*

^{n}*B*

^{(s)}

*x*⩽

*c*

^{(s)}

*}*.

**Lemma 5.52.** *There exists a number* *N* ⩾ 1 *that depends on* *B*^{(1)}*, . . . , B*^{(q)}*, but not on* *c,*
*such that if* S_{c}*is a nonempty real tropical cone, then the operator* *F*: R^{n}*→* R^{n}*,* *F** _{k}*(x) :=

max{y*k*: *y* *∈* S_{c}*, y* ⩽ *x}* *defined in the proof of Proposition 5.34 has a piecewise description*
(*W*^{(s)}*, A*^{(s)}*, b*^{(s)})_{s}_{∈}_{[p]} *such that* *p* ⩽ *N, the matrices* *A*^{(s)} *are rational and stochastic, and the*
*common denominator of all the numbersA*^{(s)}_{kl}*is not greater than* *N.*

*Proof.* Fix *c*such thatS* _{c}* is a nonempty real tropical cone. We will show how to construct the
piecewise description of the operator

*F*that has the desired properties. Given

*x∈*R

*we denote by*

^{n}*Q(x)*

*⊂*[q] the set of all

*s*

*∈*[q]such that the polyhedron

*{y*

*∈*R

*:*

^{n}*B*

^{(s)}

*y*⩽

*c*

^{(s)}

*, y*⩽

*x}*is nonempty. SinceS

*is a nonempty real tropical cone, the the set*

_{c}*Q(x)*is never empty (take an arbitrary

*z∈*S

*and consider*

_{c}*λ*+

*z*for

*λ∈*R small enough). Therefore, by the strong duality of linear programming ([Sch87, Corollary 7.1g]) we have

*F** _{k}*(x) = max

*s**∈**Q(x)*max*{y** _{k}*:

*B*

^{(s)}

*y*⩽

*c*

^{(s)}

*, y*⩽

*x}*(5.8)

= max

*s**∈**Q(x)*min*{z*^{⊺}*c*^{(s)}+*w*^{⊺}*x*: (B^{(s)})^{⊺}*z*+*w*=*ϵ*_{k}*, z*⩾0, w⩾0*},* (5.9)
where *ϵ** _{k}* denotes the

*kth vector of the standard basis. The polyhedron*

*W*

_{k}^{(s)}:=

*{*(z, w)

*∈*R

^{m}

^{s}*×*R

*: (B*

^{n}^{(s)})

^{⊺}

*z*+

*w*=

*ϵ*

_{k}*, z*⩾ 0, w ⩾ 0

*}*is pointed, and hence the minimum in (5.9) is attained in some vertex of this polyhedron and these vertices are rational ([Sch87, Chapter 8]).

Let*V*_{k}^{(s)} denote the set of vertices of *W*_{k}^{(s)}. Therefore, we have
*F**k*(x) = max

*s**∈**Q(x)* min

(z,w)*∈**V*_{k}^{(s)}

*{z*^{⊺}*c*^{(s)}+*w*^{⊺}*x}.* (5.10)

Moreover, by Farkas’ lemma (Theorem 4.36), the polyhedron *{y* *∈* R* ^{n}*:

*B*

^{(s)}

*y*⩽

*c*

^{(s)}

*, y*⩽

*x}*is nonempty if and only if for all (z, w) such that (B

^{(s)})

^{⊺}

*z*+

*w*= 0,

*z*⩾ 0,

*w*⩾ 0, we have

104 Chapter 5. Tropical analogue of the Helton–Nie conjecture
*z*^{⊺}*c*^{(s)}+*w*^{⊺}*x*⩾0. As previously, if*U*^{(s)} consists of precisely one rational representative of every
extreme ray of the polyhedral cone *{*(z, w) *∈*R^{m}^{s}*×*R* ^{n}*: (B

^{(s)})

^{⊺}

*z*+

*w*= 0, z ⩾0, w ⩾0

*}*, then

*{y∈*R

*:*

^{n}*B*

^{(s)}

*y*⩽

*c*

^{(s)}

*, y*⩽

*x}*is nonempty if and only if

*z*

^{⊺}

*c*

^{(s)}+w

^{⊺}

*x*⩾0for all(z, w)

*∈U*

^{(s)}. Let

*U*denote the disjoint union of

*U*

^{(1)}

*, . . . , U*

^{(q)}. For every possible choice♢

*∈ {<, >,*=}

*we define*

^{U}*W*˜

^{♢}:=

*{x∈*R

*:*

^{n}*∀s∈*[q],

*∀*(z, w)

*∈U*

^{(s)}

*, z*

^{⊺}

*c*

^{(s)}+

*w*

^{⊺}

*x*♢(s,z,w)0

*}*. Note that

*Q(x)*is constant on each

*W*˜

^{♢}. By fixing the terms achieving the maximum and minimum in (5.10) we subdivide the sets

*W*˜

^{♢}into smaller sets,

*W*˜

^{♢}=

^{∪}

^{N}

_{j=1}^{♢}

*W*˜

_{j}^{♢}such that every

*W*˜

_{j}^{♢}is an intersection of half-spaces (both open and closed) and

*F*is aﬃne on

*W*˜

_{j}^{♢}. Since

*F*is continuous (by Lemma 5.31), we can then restrict ourselves to these sets

*W*˜

_{j}^{♢}that are full dimensional, and this gives the piecewise description of

*F*. To see that

*F*has the expected property, note that the definition of

*U*

^{(s)}does not depend on

*c. Therefore, the number of sets*

*W*˜

^{♢}does not depend on

*c. Furthermore, the*definition of

*V*

_{k}^{(s)}also does not depend on

*c. Hence, the numbersN*

_{♢}are bounded by a quantity that does not depend on

*c. Moreover, by (5.10), the matrices*

*A*

^{(s)}are of the form

*A*

^{(s)}

*=*

_{k}*w*

^{⊺}. Hence, the common denominator of the numbers

*A*

^{(s)}

*is bounded by a quantity that does not depend on*

_{kl}*c. The fact thatA*

^{(s)}must be stochastic follows from Lemma 5.32.

Theorem 5.2 follows from Lemma 5.52 by a careful examination of the proofs presented in Section 5.4. We split the proof into a series of lemmas. The first lemma is a uniform version of Proposition 5.44.

**Lemma 5.53.** *There exists a number* *N* ⩾1 *that depends onB*^{(1)}*, . . . , B*^{(q)}*, but not onc, such*
*that if*S_{c}*is a real tropical cone, then it is a projection of a real tropical Metzler spectrahedron*
*S(Q*^{(0)}*|Q*^{(1)}*, . . . , Q*^{(n}^{′}^{)}), *Q*^{(k)}*∈*T^{m}_{±}^{×}^{m}*, satisfying* *n** ^{′}*⩽

*N*

*andm*⩽

*N.*

*Proof.* Suppose that S* _{c}* is a real tropical cone. We will examine all the steps involved in the
proof of Proposition 5.44. By Lemma 5.52 and the proof of Proposition 5.34, there exists

*N*

_{1}⩾1 independent of

*c*such that we haveS

*=*

_{c}*{x∈*R

*:*

^{n}*x*⩽

*F*(x)}, where the operator

*F*:R

^{n}*→*R

*is semilinear, monotone, homogeneous, and has a piecewise description (*

^{n}*W*

^{(s)}

*, A*

^{(s)}

*, b*

^{(s)})

_{s}

_{∈}_{[p]}

satisfying *p* ⩽ *N*1. Moreover, the matrices *A*^{(s)} are stochastic, rational, and the common
denominator of all the numbers *A*^{(s)}* _{kl}* is not greater than

*N*

_{1}. By Theorem 5.27, the operator

*F*can be written as

*∀k, F*

*(x) = min*

_{k}

_{i∈[M}

_{k}_{]}max

_{s}

_{∈}

_{S}*(A*

_{ki}^{(s)}

_{k}*x*+

*b*

^{(s)}

*), where 1⩽*

_{k}*M*

*⩽2*

_{k}

^{N}^{1}for all

*k*⩾1and every set

*S*

*is a subset of[p].*

_{ki}Given the operator *F*, the proof of Lemma 5.38 constructs a directed graph *G⃗* satisfying
Assumption C. By the construction presented in the proof of Lemma 5.38, the vertices, edges,
and probability distributions of *G⃗* depend on (n,(M* _{k}*)

_{k}*,*(S

*)*

_{ki}

_{ki}*,*(A

^{(s)})

*s*), but not on (b

^{(s)})

_{s}

_{∈}_{[p]}. The vector(b

^{(s)})

_{s}

_{∈}_{[p]}is used only to define the weights

*r*

*e*associated with the edges of the graph.

Subsequently, the proof of Proposition 5.44 proceeds by applying Lemmas 5.41 to 5.43 to the
graph *G⃗* and transforming it into a graph*G⃗** ^{′}*. Let

*n*

*denote the number of Min vertices of*

^{′′}*G⃗*

*and*

^{′}*m*

*denote the number of Max vertices of*

^{′}*G⃗*

*. By the constructions given in Lemmas 5.41 to 5.43, the numbers*

^{′}*n*

^{′′}*, m*

*do not depend on the weights*

^{′}*r*

*e*associated with the edges of

*G⃗*. Hence, they do not depend on(b

^{(s)})

_{s}

_{∈}_{[p]}. Given

*G⃗*

*, we apply Proposition 5.40 to construct some symmetric tropical matrices*

^{′}*Q*˜

^{(1)}

*, . . . ,Q*˜

^{(n}

^{′′}^{)}such that

*Q*˜

^{(k)}

*∈*T

^{m}

_{±}

^{′}

^{×}

^{m}*. As a final step, the proof of Proposition 5.44 transforms these matrices into matrices*

^{′}*Q*

^{(0)}

*, Q(1), . . . , Q*

^{(n}

^{′}^{)}

*∈*T

^{m}

_{±}

^{×}*by adding some additional variables and2*

^{m}*×*2blocks to the matrices

*Q*˜

^{(1)}

*, . . . ,Q*˜

^{(n}

^{′′}^{)}. However, the number of these additional variables and blocks depends only on

*n*and

*n*

*. As a consequence, the numbers*

^{′′}*n*

^{′}*, m*depend on (n,(M

*)*

_{k}

_{k}*,*(S

*)*

_{ki}

_{ki}*,*(A

^{(s)})

*), but not on (b*

_{s}^{(s)})

_{s}

_{∈}_{[p]}.

5.6. Extension to general fields 105
Since the number of possible tuples((M*k*)*k**,*(S*ki*)*ki**,*(A^{(s)})*s*)is finite and bounded by a
quan-tity that depends only on *n, N*_{1} (not on *c), the numbers* *n*^{′}*, m*are also bounded by a quantity
that does not depend on*c.*

The claim of the previous lemma can be extended to tropically convex sets using the proof of Lemma 5.49.

**Lemma 5.54.** *There exists a number* *N* ⩾1 *that depends onB*^{(1)}*, . . . , B*^{(q)}*, but not onc, such*
*that if* S_{c}*is tropically convex, then it is a projection of a real tropical Metzler spectrahedron*
*S*(Q^{(0)}*|Q(1), . . . , Q*^{(n}^{′}^{)}), *Q*^{(k)}*∈*T^{m}_{±}^{×}^{m}*, satisfying* *n** ^{′}*⩽

*N*

*and*

*m*⩽

*N.*

*Proof.* Let *e* = (1,1, . . . ,1). For every *s* *∈* [q] we denote *B*˜^{(s)} = [−B^{(s)}*e|B*^{(s)}] *∈* Q^{m}^{s}^{×}^{(n+1)}.
Then, the real homogenization ofS* _{c}* is given by

S_{c}* ^{rh}*=

∪*q*
*s=1*

*{(x*0*, x)∈*R* ^{n+1}*: ˜

*B*

^{(s)}(x0

*, x)*

^{⊺}⩽

*c*

^{(s)}

*}.*(5.11) Indeed, if we take any

*x*0

*∈*R and

*x*

*∈*S

*such that*

_{c}*B*

^{(s)}

*x*⩽

*c*

^{(s)}, then

*B*˜

^{(s)}(x0

*, x*0 +

*x)*

^{⊺}=

*−x*_{0}*B*^{(s)}*e+B*^{(s)}(x_{0}+x) =*B*^{(s)}*x*⩽*c*^{(s)}. Conversely, if(x_{0}*, x)*is such that*B*˜^{(s)}(x_{0}*, x)*^{⊺}⩽*c*^{(s)}, and
we denote*y*=*−x*0+*x, thenB*^{(s)}*y*=*−x*0*B*^{(s)}*e*+*B*^{(s)}*x*= ˜*B*^{(s)}(x0*, x)*^{⊺}⩽*c*^{(s)} and*y∈*S* _{c}*. Given
S

*, the proof of Lemma 5.49 considers its real homogenizationS*

_{c}

_{c}*, describes it as a projection of a real tropical Metzler spectrahedron*

^{rh}*S*( ˜

*Q*

^{(0)}

*|Q*˜

^{(1)}

*, . . . ,Q*˜

^{(n}

^{′}^{)}), and adds a2

*×*2 block to the matrices

*Q*˜

^{(0)}

*,Q*˜

^{(1)}

*, . . . ,Q*˜

^{(n}

^{′}^{)}. Therefore, the claim follows from Lemma 5.53 and (5.11).

The next lemma is a more precise version of Lemma 5.51.

**Lemma 5.55.** *Suppose that* S_{1} _{⊂}_{T}^{n}*is a projection of a tropical Metzler spectrahedron* *S*1 =
*S*( ˜*Q*^{(0)}*|Q*˜^{(1)}*, . . . ,Q*˜^{(n}^{1}^{)}), *Q*˜^{(k)} *∈* T^{m}_{±}^{1}^{×}^{m}^{1} *and that*S_{2} _{⊂}_{T}^{n}*is a projection of a tropical Metzler*
*spectrahedron* *S*2 = *S*( ˆ*Q*^{(0)}*|Q*ˆ^{(1)}*, . . . ,Q*ˆ^{(n}^{2}^{)}), *Q*ˆ^{(k)} *∈* T^{m}_{±}^{2}^{×}^{m}^{2}*. Then, there exists a number* *N*
*that depends only on* (n, n1*, m*1*, n*2*, m*2) *such that* tconv(S_{1} * _{∪}*S

_{2})

*is a projection of a tropical*

*Metzler spectrahedron*

*S*(Q

^{(0)}

*|Q*

^{(1)}

*, . . . , Q*

^{(n}

^{′}^{)}),

*Q*

^{(k)}

*∈*T

^{m}

_{±}

^{′}

^{×}

^{m}

^{′}*satisfying*

*n*

^{′}*, m*

*⩽*

^{′}*N.*

*Proof.* The proof follows from an examination of the construction given in Lemma 5.51. More
precisely, the construction is as follows. The first step is to consider the homogenizations
S_{1}^{h}*,*S_{2}* ^{h}* and describe them as projections of tropical Metzler spectrahedra

*S*˜1

*,S*˜2. The matrices that describe

*S*˜1

*,S*˜2 are constructed using Lemma 5.50, by taking the formal homogenizations

*S*

_{1}

^{f h}*,S*

_{2}

*and adding variables and 2*

^{f h}*×*2 blocks to the matrices that describe these formal homogenizations. The number of these variables and blocks depends only on(n, n1

*, n*2). Then, the construction presented in Lemma 5.51 considers the set

*S*˜1

*×S*˜2, which is a tropical Metzler spectrahedron described by block diagonal matrices obtained by stacking the matrices that describe

*S*˜1

*,S*˜2. Afterwards, the construction adds some 2

*×*2 blocks to the matrices that describe

*S*˜1

*×S*˜2and the number of these blocks depends only on(n, n1

*, n*2). This gives a tropical Metzler spectrahedron

*S*˜3 such that the homogenization of tconv(S

_{1}

*S*

_{∪}_{2}) is its projection.

Finally, the other part of Lemma 5.50 is used: we add a 2*×*2 block to the matrices that
describe*S*˜3 and find the matrices *Q*^{(0)}*, Q*^{(1)}*, . . . , Q*^{(n}^{′}^{)}. Hence, the numbers*n*^{′}*, m** ^{′}* depend only
on (n, n

_{1}

*, m*

_{1}

*, n*

_{2}

*, m*

_{2}).

We are now ready to give a stratified version of Lemma 5.54 and prove Theorem 5.2. Suppose
that(f* _{K,i}*)

_{K⊂[n],i∈[q}

_{K}_{]}is a collection of homogeneous polynomials with integer coeﬃcients,

*f*

_{K,i}*∈*Z[X

*], where*

_{K}*X*

*= (x*

_{K}

_{k}_{1}

*, . . . , x*

_{k}*) for*

_{ℓ}*K*=

*{k*

_{1}

*, . . . , k*

_{ℓ}*}*, and we use the convention that

*f*

_{∅}*is*

_{,i}106 Chapter 5. Tropical analogue of the Helton–Nie conjecture
the zero polynomial. Given a real vector *c*= (c^{(K,i)})*K,i*, *c*^{(K,i)} *∈*R we denote by S_{c}_{⊂}_{T}* ^{n}* the
set

∪

*K**∈*[n]

∪

*i**∈*[q*K*]

*{x∈*T* ^{n}*:

*∀k /∈K, x*

*=*

_{k}*−∞ ∧ ∀k∈K, x*

_{k}*̸*=

*−∞ ∧*

*f*

*K,i*(x

*K*)⩽

*c*

^{(K,i)}

*}.*(5.12)

**Lemma 5.56.** *There exists a number* *N* ⩾1 *that depends on* (f*K,i*)_{K}_{⊂}_{[n],i}_{∈}_{[q}_{K}_{]}*, but not on* *c,*
*such that if* S_{c}*is tropically convex, then it is a projection of a tropical Metzler spectrahedron*
*S*(Q^{(0)}*|Q(1), . . . , Q*^{(n}^{′}^{)}), *Q*^{(k)}*∈*T^{m}_{±}^{×}^{m}*, satisfying* *n** ^{′}*⩽

*N*

*and*

*m*⩽

*N.*

*Proof.* The claim follows from the previous lemmas by analyzing the proof of Theorem 5.5.

DenoteS* _{c}*=Sand take any

*K*

*⊂*[n]. If

*K*is nonempty, then the stratum ofSassociated with

*K, denoted*S

*, is tropically convex and given by*

_{K}S* _{K}* =

^{∪}

*i**∈*[q*K*]

*{x∈*T* ^{K}*:

*f*

*K,i*(x)⩽

*c*

^{(K,i)}

*}.*

Therefore, by Lemma 5.54, there exists*N**K*(independent on*c) and such that*S* _{K}* is a projection
of a real tropical Metzler spectrahedron

*S*

*K*

*⊂*R

^{n}*,*

^{K}*n*

*⩽*

_{K}*N*

*, described by matrices of size*

_{K}*m*

*⩽*

_{K}*N*

*. Moreover, as noted in the proof of Theorem 5.5, the set*

_{K}S˜* _{K}* =

^{∪}

*i**∈*[q* _{K}*]

*{x∈*T* ^{n}*:

*∀k /∈K, x*

*=*

_{k}*−∞ ∧ ∀k∈K, x*

_{k}*̸*=

*−∞ ∧*

*f*

*(x*

_{K,i}*)⩽*

_{K}*c*

^{(K,i)}

*}*

is a projection of a tropical Metzler spectrahedron *S*˜*K*. This spectrahedron is constructed by
adding variables and 1*×*1 blocks to the matrices that describe *S**K*, and the number of these
blocks and variables does not depend on *c. Similarly, the set*S* _{∅}* is either empty or reduced to

*−∞* and thus it is a tropical Metzler spectrahedron. We conclude by applying Lemma 5.55 to
tconv(^{∪}S˜* _{K}*).

*Proof of Theorem 5.2.* Suppose that *S* *⊂*K* ^{n}* is convex and semialgebraic. Fix a

*L*or-formula

*ψ*and a vector

*a*

*∈*K

*such that*

^{n}*S*=

*{x*

*∈*K

*:K*

^{n}*=*

_{|}*ϕ(x, a)}. Then, by Theorem 3.1, the*set val(S) has closed semilinear strata. In particular, by Lemma 3.12, there exists a collection (f

*)*

_{K,i}

_{K}

_{⊂}_{[n],i}

_{∈}_{[q}

_{K}_{]}and a vector

*c*= (c

^{(K,i)})

*,*

_{K,i}*c*

^{(K,i)}

*∈*

*Γ*such that val(S) is of the form given in (5.12),val(S) =S

*(we note that this form is also correct for*

_{c}*K*=

*∅*because

*Γ*is nontrivial).

For every *b* *∈* K* ^{m}* we denote

*S*

*:=*

_{b}*{x*

*∈*K

*:K*

^{n}*=*

_{|}*ϕ(x, b)}*. If we fix

*N*⩾ 1, then the statement “for all

*b, c*such that

*S*

*is convex and val(S*

_{b}*) =S*

_{b}*there exist symmetric matrices*

_{c}*Q*

^{(0)}

*, . . . , Q*

^{(n}

^{′}^{)},

*Q*

^{(k)}

*∈*K

^{m}

^{′}

^{×}

^{m}*such that*

^{′}*n*

^{′}*, m*

*⩽*

^{′}*N*,

*n*

*⩾*

^{′}*n, and*

*S*

*has the same image by valuation as the projection of the spectrahedron associated with*

_{b}*Q*

^{(0)}

*, . . . , Q*

^{(n}

^{′}^{)}” is a sentence in

*L*rcvf. This sentence is true ifK =Kand

*N*is large enough. Indeed, in this case we can apply Lemma 5.56 to S

*to describe it as a projection of a tropical Metzler spectrahedron and lift this spectrahedron into K by Proposition 4.14. This spectrahedron fulfills the claim (as noted in the proof of Theorem 5.5). Therefore, if we fix*

_{c}*N*such that the sentence above in true over K, then the completeness result of Theorem 2.120 shows that this sentence is true inK.

**Part II**

**Relation to stochastic mean payoﬀ**

**games**

**CHAPTER** *6*

**Introduction to stochastic mean payoﬀ** **games**

In this chapter we introduce the notion of stochastic mean payoﬀ games and we show that these games can be analyzed using monotone homogeneous operators (called Shapley operators).

Our presentation follows [AGG12]. The reference [AGG12] considered only deterministic mean payoﬀ games, but numerous proofs of [AGG12] extend immediately to the case of stochastic mean payoﬀ games. However, we add our two contributions. First, in Theorem 6.16 we show that the supports of tropical cones defined as sublevels sets of Shapley operators correspond to winning dominions of the underlying game (which is a nontrivial extension of the analogous result of [AGG12]). Second, in Theorem 6.30 we combine the analysis of this chapter with the previous one, showing the structural equivalence between tropicalizations of closed convex semialgebraic cones and stochastic mean payoﬀ games. The chapter is organized as follows. In Section 6.1 we define the notion of a stochastic mean payoﬀ game and the associated Shapley operator. We also show how Kohlberg’s theorem [Koh80] can be used to prove that these games have optimal policies. In Section 6.2 we study the dominions and show a simple version of the Collatz–Wielandt property of [Nus86, AGG12]. In Section 6.3 we specify the analysis to the case of bipartite games and give the relationship with the results of the previous chapter. A simplified version of the results presented here is included in the paper [AGS18].

110 Chapter 6. Introduction to stochastic mean payoﬀ games

**6.1** **Optimal policies and Shapley operators**

A *stochastic mean payoﬀ game* involves two players, Max and Min, who control disjoint sets of
states. The states owned by Player Max and Player Min are respectively denoted as*V*_{Max}:= [m]

and *V*_{Min} := [n]. The space of states of the games is a disjoint union of *V*_{Max} and *V*_{Min},
*V* :=*V*Max*⊎V*Min. We will use the symbols*i, j* to refer to states of Player Max, and*k, l*to states
of Player Min. Every state*k∈*[n]of Player Min is equipped with a finite and nonempty set*A*^{(k)}
(called the set of*actions). Similarly, every statei∈*[m]of Player Max is equipped with a finite
and nonempty set*B*^{(i)}. Every action*a∈*^{⊎}_{k}*A*^{(k)} is equipped with a probability distribution*p*^{a}* _{k}*
over

*V*,

*p*

^{a}*= (p*

_{k}

^{a}*)*

_{kv}*v*

*∈*

*V*

*∈*[0,1]

*,*

^{V}^{∑}

_{v}

_{∈}

_{V}*p*

^{a}*= 1and a real number*

_{kv}*r*

*a*

*∈*R. Likewise, every action

*b*

*∈*

^{⊎}

*i*

*B*

^{(i)}is equipped with a probability distribution

*p*

^{b}*= (p*

_{i}

^{b}*)*

_{iv}

_{v}

_{∈}

_{V}*∈*[0,1]

*,*

^{V}^{∑}

_{v}

_{∈}

_{V}*p*

^{b}*= 1 over*

_{iv}*V*and a real number

*r*

*b*

*∈*R. Both players alternatively move a pawn over these states by the rules described as follows. When the pawn is on a state

*k∈*[n], Player Min chooses an action

*a*

*∈*

*A*

^{(k)}. Then, Player Min moves the pawn to state

*v*

*∈*

*V*with probability

*p*

^{a}*and pays*

_{kv}*r*

*to Player Max. Similarly, once the pawn is on a state*

^{a}*i*

*∈*[m], Player Max picks an action

*b∈B*

^{(i)}. Then, Player Max moves the pawn to the state

*v∈V*with probability

*p*

^{b}*and Player Min pays him*

_{iv}*r*

*.*

^{b}A *policy for Player Min* is a function *σ*: [n]*→* ^{⊎}_{k}*A*^{(k)} mapping every state *k* *∈* [n] to an
action *σ(k)* in *A*^{(k)}. Analogously, a*policy for Player Max* is a function *τ*: [m]*→* ^{⊎}*i**B*^{(i)} such
that *τ*(i) *∈* *B*^{(i)} for all *i* *∈*[m]. Suppose that the game starts from a state *v** ^{∗}*. When players
play according to a couple(σ, τ)of policies, the movement of the pawn is described by a Markov
chain on the space

*V*. The average payoﬀ of Player Max in the long-term is then defined as the average payoﬀ of the controller in this Markov chain, see Definition 2.136. In other words, the payoﬀ of Player Max is given by for

*i∈*[m]. This payoﬀ is well defined by Theorem 2.137. The goal of Player Max is to find a policy that maximizes his average payoﬀ, while Player Min aims at minimizing this quantity.

The main theorem of stochastic mean payoﬀ games ensures that this game has a well-defined optimal policies. This was shown by Liggett and Lippman [LL69].

**Theorem 6.1** ([LL69]). *There exists a pair of policies*(σ, τ) *and a unique vector* *χ∈*R^{V}*such*
*that for all initial states* *v∈V, the following two conditions are satisfied:*

*•* *for each policyσ* *of Player Min,* *χ**v* ⩽*g**v*(σ, τ);

*•* *for each policyτ* *of Player Max,* *χ**v* ⩾*g**v*(σ, τ).

*The vectorχ* *called the* value *of the game, while the policies*(σ, τ) *are called* optimal. We note
*that we have* *χ**v*=*g**v*(σ, τ) *for all* *v∈V.*

The first condition in Theorem 6.1 states that, by playing according to the policy *τ*, Player
Max is certain to get an average payoﬀ greater than or equal to the value *χ**v* associated with
the initial state. Symmetrically, Player Min is ensured to limit her average loss to the quantity
*χ** _{v}* by following the policy

*σ.*

*Example* 6.2. Consider the game is depicted in Fig. 6.1. The states of Player Min are depicted
by circles. The states of Player Max are depicted by squares. Moreover, all probabilities in

6.1. Optimal policies and Shapley operators 111

Figure 6.1: Stochastic mean payoﬀ game considered in Example 6.2.

this example belong to*{*0,1/2,1*}*. More precisely, edges represent the actions of players. Edges
without full dots represent deterministic actions and full dots indicate actions where a coin
toss is made. The corresponding payments received by Player Max are indicated on the edges.

Observe that both players in this game have only two policies: at state 2 Player Max can
choose the move that goes to 1 or the one that goes to 3 , whereas at state 3 Player Min can
choose the move ^{{}1*,* 3^{}} or the move ^{{}2*,* 3^{}}. Both players in this example game have only
two policies: at state 2 Player Max can choose the action that goes to 1 or the action that
goes to 3 , whereas at state 3 Player Min can choose the action that goes to ^{{}1*,*3^{}} or the
action that goes to ^{{}2*,* 3^{}}. Suppose that Player Max chooses the action that goes to 1 and
that Player Min chooses the action that goes to ^{{}1*,* 3^{}}. The Markov chain obtained in this
way has the transition matrix of form

where *Y* describes the probabilities of transition from circle states to square states, and *Z*
describes the probabilities of transition from square states to circle states, i.e.,

*Y* =

This chain has only one recurrent class and all states belong to this class. Moreover, it is easy to verify that

*π*= 1

10(2,1,2,2,2,1)*∈*R^{V}^{Min}*×*R^{V}^{Max}

is the stationary distribution of this chain. Therefore, by Theorem 2.137, the payoﬀ of Player Max is equal to

for every initial state. Similarly, if Player Max chooses the action that goes to 1 and Player
Min chooses the action that goes to^{{}2*,* 3^{}}, then the transition matrix of the induced Markov
chain has the form*P** ^{′}* =

^{[}

_{W}^{0}

^{Y}_{0}

^{′}^{]}, where

112 Chapter 6. Introduction to stochastic mean payoﬀ games
and *Z* is the same as previously. In this case the chain also has only one recurrent class and
every state belongs to this class. Furthermore,

*π** ^{′}*= 1

14(4,1,2,2,4,1)*∈*R^{V}^{Min}*×*R^{V}^{Max}

is the stationary distribution of this chain. Hence the payoﬀ of Player Max is equal to
*g*^{({}2*,* 3^{}}*,{*1 *}*^{)}= 1

14(2*−*4 +9
4) = 1

56

for every initial state. In both cases, the payoﬀ of Player Max is positive. Therefore, if Player Max chooses the action that goes to 1 , then he is guaranteed to obtain a nonnegative payoﬀ.

One can check, by doing the same calculations for the remaining policies, that^{({}2*,* 3^{}}*,{* 1*}*^{)}
is the unique couple of optimal policies in this game and the value of the game is equal to
*χ*= _{56}^{1}(1,1,1,1,1,1).

In this section we prove Theorem 6.1 using the associated Shapley operator. This proof
was given in [AGG12] in the case of deterministic mean payoﬀ games (i.e., when all probability
distributions are reduced to vectors in*{0,*1}* ^{V}*). However, the proof in the stochastic case is the
same as the one given in [AGG12], and we reproduce it here for the sake of completeness.

**Definition 6.3.** Given a stochastic mean payoﬀ game, we define its*Shapley operatorT*:T^{V}*→*
T* ^{V}* as

*∀v∈V,*(T(x))* _{v}* :=

{min_{a}_{∈}* _{A}*(v)(r

*a*+

^{∑}

_{w}

_{∈}

_{V}*p*

^{a}

_{vw}*x*

*w*) if

*v∈V*

_{Min}

max_{b}_{∈}* _{B}*(v)(r

*+*

_{b}^{∑}

_{w}

_{∈}

_{V}*p*

^{b}

_{vw}*x*

*) if*

_{w}*v∈V*

_{Max}

*,*(6.2) where we use the convention that

*−∞ ·*0 = 0 and

*−∞ ·x*=

*−∞*for all

*x >*0.

The basic properties of Shapley operators are summarized in the next lemma.

**Lemma 6.4.** *The Shapley operatorT* *is monotone and homogeneous. Furthermore, it preserves*
R^{V}*(i.e., if* *x∈*R^{V}*, then* *T*(x)*∈*R^{V}*) and its restrictionT*_{|R}*V*:R^{V}*→*R^{V}*is piecewise aﬃne.*

*Proof.* The facts that*T* is monotone, homogeneous, and preservesR* ^{V}* follow immediately from
the definition. Its restriction to R

*is piecewise aﬃne because it is defined as a coordinatewise minimum or maximum of aﬃne functions.*

^{V}Our proof of Theorem 6.1 is based on the notion of invariant half-line and the following result of Kohlberg [Koh80].

**Definition 6.5.** Given a function *f*: R^{n}*→* R* ^{n}*, we say that a pair (u, η)

*∈*R

^{n}*×*R

*is an*

^{n}*invariant half-line*of

*f*if there exists

*γ*0⩾0such that the equality

*f*(u+γη) =

*u+(γ*+1)ηholds for all

*γ*⩾

*γ*

_{0}. Given a function

*f*:T

^{n}*→*T

*that preservesR*

^{n}*, we say that(u, η)*

^{n}*∈*R

^{n}*×*R

*is an invariant half-line of*

^{n}*f*if it is an invariant half-line of

*f*

_{|R}*:R*

^{n}

^{n}*→*R

*.*

^{n}**Theorem 6.6** ([Koh80]). *Suppose that function* *f*:R^{n}*→*R^{n}*is piecewise aﬃne and *
*nonexpan-sive in any norm. Then, it admits and invariant half-line. Furthermore, if* (u_{1}*, η*_{1}) *and*(u_{2}*, η*_{2})
*are invariant half-lines, then* *η*1 =*η*2*.*

Later on, we will need a more precise formulation of Kohlberg’s theorem. Therefore, we present its proof in Appendix B.3.

6.1. Optimal policies and Shapley operators 113
**Corollary 6.7.** *Suppose that* *f*: R^{n}*→* R^{n}*is a monotone, homogeneous, and piecewise aﬃne*
*function. Then, it admits and invariant half-line*(u, η, γ_{0}). Furthermore, if*x∈*R^{n}*is any point,*
*then we have the equality*

*N*lim*→∞*

1

*Nf** ^{N}*(x) =

*η ,*

*where* *f** ^{N}* =

*f◦f*

*◦ · · · ◦f*

*(N*

*times). In particular, this limit exists.*

*Proof.* The function *f* is nonexpansive in the supremum norm by Lemma 5.31. As a
conse-quence, it admits an invariant half-line by Theorem 6.6. In particular, for every *x* *∈* R* ^{n}* we
have

*∥f** ^{N}*(x)

*−N η−u−γ*0

*η∥*

*∞*=

*∥f*

*(x)*

^{N}*−f*

*(u+*

^{N}*γ*0

*η)∥*

*∞*⩽

*∥x−u−γ*0

*η∥*

*∞*

and lim*N**→∞* 1

*N**f** ^{N}*(x) =

*η.*

In order to prove Theorem 6.1, we introduce the following notation. If*σ* is a policy of Player
Min, then we denote by*T** ^{σ}* the Shapley operator of a game in which Player Min uses

*σ*(i.e., a game in which every state

*k∈*[n]controlled by Player Min is equipped with exactly one action and this action is

*σ(k)). Analogously, ifτ*is a policy of Player Max, then we denote by

*T*

*the Shapley operator of a game in which Player Max uses*

^{τ}*τ*. Furthermore, we denote by

*T*

*the Shapley operator of a game in which Player Min uses*

^{σ,τ}*σ*and Player Max uses

*τ*. In an explicit way, these operators are given by The properties of these operators are given in the next lemmas.

**Lemma 6.8.** *If we fix* *x∈*R^{V}*, then the set* *{T** ^{σ}*(x) :

*σ*

*is a policy of Player Min} ⊂*R

^{V}*has a*

*well defined minimum in the partial order given by*

*y*⩽

*z*

*⇐⇒ ∀v∈V, y*

*v*⩽

*z*

*v*

*. Moreover, this*

*minimum is equal to*

*T(x). Analogously, the set*

*{T*

*(x) :*

^{τ}*τ*

*is a policy of Player Max} ⊂*R

^{V}*has a well defined maximum in this partial order and this maximum is equal to*

*T*(x).

*Proof.* By definition we have (T* ^{σ}*(x))

*v*⩾(T(x))

*v*for all

*v*

*∈*

*V*. Furthermore, for every state

*k*

*∈*[n] we can take an action

*a*

^{∗}*∈*

*A*

^{(k)}that satisfies

*r*

*a*

*+*

^{∗}^{∑}

_{w}

_{∈}

_{V}*p*

^{a}

_{vw}

^{∗}*x*

*w*= min

*(k)(r*

_{a∈A}*a*+

∑

*w**∈**V* *p*^{a}_{vw}*x** _{w}*)and define

*σ*

*(k) :=*

^{∗}*a*

*. The policy*

^{∗}*σ*

*satisfies(T*

^{∗}

^{σ}*(x))*

^{∗}*= (T(x))*

_{v}*for all*

_{v}*v∈V*. The other case is analogous.

**Lemma 6.9.** *If Player Min uses a policy* *σ* *and Player Max uses a policyτ, then the expected*
*total payoﬀ of Player Max after the* *Nth move of the pawn is equal to* (T* ^{σ,τ}*)

*(0).*

^{N}*Proof.* We prove the claim by induction. Let *g*^{N}*∈* R* ^{V}* denote the expected payoﬀ of Player
Max after the

*N*th move of the pawn. We want to show that

*g*

*= (T*

^{N}*)*

^{σ,τ}*(0) for all*

^{N}*N*⩾1.

*Proof.* We prove the claim by induction. Let *g*^{N}*∈* R* ^{V}* denote the expected payoﬀ of Player
Max after the

*N*th move of the pawn. We want to show that

*g*

*= (T*

^{N}*)*

^{σ,τ}*(0) for all*

^{N}*N*⩾1.