Coding Theorems for Empirical Coordination

HAL Id: hal-01865569

https://hal.archives-ouvertes.fr/hal-01865569

Preprint submitted on 30 Jan 2020

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Mael Le Treust

To cite this version:

Mael Le Treust. Coding Theorems for Empirical Coordination: Technical Report. 2015. �hal- 01865569�

Coding Theorems for Empirical Coordination – Technical Report

Maël Le Treust

ETIS, UMR 8051 / ENSEA, Université Cergy-Pontoise, CNRS, 6, avenue du Ponceau,

95014 CERGY-PONTOISE CEDEX, FRANCE

Email: [email protected]

C

I Non-Causal Encoding and Decoding 3

I-A Achievability Proof . . . . 5 I-B Converse Proof . . . . 6

II Perfect Channel 8

II-A Achievability Proof . . . . 9 II-B Converse Proof . . . 11

III Lossless Decoding 14

III-A Achievability Proof . . . 16 III-B Converse Proof . . . 18

IV Separation between Source and Channel 22

IV-A Achievability Proof . . . 24 IV-B Converse Proof . . . 26

V Causal Decoding 30

V-A Achievability Proof . . . 32

VI Causal Encoding 38 VI-A Achievability Proof . . . 40 VI-B Converse Proof . . . 46

VII Strictly Causal Decoding 49

VII-A Achievability Proof . . . 51 VII-B Converse Proof . . . 54

VIIIStrictly Causal Encoding 56

VIII-AAchievability Proof . . . 58 VIII-BConverse Proof . . . 60

Appendix 65

References 76

I. N

-C

E

D

U

S

Z

X

Y

V

P

C T D

Fig. 1. Non-Causal Encoding functionf:Uⁿ× Sⁿ→ Xⁿand Decoding functiong:Yⁿ× Zⁿ→ Vⁿ.

Theorem I.1 (Non-Causal Encoding and Decoding)

1) Joint probability distribution Q(u, s, z, x, y, v) is achievable if and only if it decomposes as follows:





















Q(u, s, z) = P

(u, s, z), Q(y|x, s) = T (y|x, s), Y − − (X, S) − − (U, Z), Z − − (U, S ) − − (X, Y ),

(1)

and P

(u, s, z) ⊗ Q(x|u, s) ⊗ T (y|x, s) ⊗ Q(v|u, s, z, x, y) is achievable.

2) Joint probability distribution P

(u, s, z) ⊗ Q(x|u, s) ⊗ T (y|x, s) ⊗ Q(v|u, s, z, x, y) is achievable if:

max

I(W

, W

; Y, Z) − I(W

, W

; U, S)

> 0, (2)

3) Joint probability distribution P

(u, s, z) ⊗ Q(x|u, s) ⊗ T (y|x, s) ⊗ Q(v|u, s, z, x, y) is not achievable if:

max

I(W

; Y, Z|W

) − I(W

; U, S |W

)

< 0, (3)

where

is the set of distributions Q ∈ ∆(U × S × Z × W

× W

× X × Y × V) with auxiliary random variables (W

, W

) that satisfies:



























 P

(w1,w2)∈W1×W2Q(u, s, z, w1, w2, x, y, v)

=Pusz(u, s, z)⊗ Q(x|u, s)⊗ T(y|x, s)⊗ Q(v|u, s, z, x, y), Y −−(X, S)−−(U, Z, W1, W2),

Z−−(U, S)−−(X, Y, W1, W2), V −−(Y, Z, W1, W2)−−(U, S, X).

The probability distribution Q ∈

decomposes as follows:

P

(u, s, z) ⊗ Q(x|u, s) ⊗ Q(w

, w

|u, s, x) ⊗ T (y|x, s) ⊗ Q(v|y, z, w

, w

).

The supports of the auxiliary random variables (W

, W

) are bounded by max(|W

|, |W

|) ≤ (|B| + 1) · (|B| + 2) with B = U × S × Z × X × Y × V.

Remark I.2 The mutual informations in equation (2) are continuous over the set of probability distributions

is compact since the supports of the auxiliary random variables (W

, W

) are finite and

has equality constraints. As mentioned in [8] pp. 7083 and in [2] pp. 9, we can consider the maximum instead of the supremum in equation (2).

Remark I.3 The achievability result of Theorem I.1 without state informations S and Z was already stated in [1], with a unique auxiliary random variable W = (W

, W

).

Remark I.4 As mentioned in Theorem I.1 1), probability distribution Q(u, s, z, x, y, v) should satisfy

the marginal distributions over the source P

(u, s, z), the channel T (y|x, s) and the Markov chains

representing the network topology. If a probability distribution Q(u, s, z, x, y, v) does not decomposes

with P

(u, s, z) ⊗ Q(x|u, s) ⊗ T (y|x, s) ⊗ Q(v|u, s, z, x, y), then the error probability can not converge

to zero and this probability distribution is not achievable. This remark is valid for all coding theorems

presented in this document.

A. Achievability Proof

Denote by Q(u, s, z, x, w

, w

, y, v) ∈

the joint probability distribution that achieves the maximum in equation (2). There exists δ > 0 and rate

≥

such that :

R

≥ I (W

, W

; U, S) + δ, (4)

R

≤ I (W

, W

; Y, Z) − δ. (5)

• Random codebook. We generate |M| = 2

pairs of sequences (W

(m), W

(m)) with index m ∈ M drawn from the i.i.d. marginal probability distribution Q

.

• Encoding function. The encoder observes the sequences of source symbols U

∈ U

and state symbols S

∈ S

. It finds the index m ∈ M such that the sequences (U

, S

, W

(m), W

(m)) ∈ A

(Q) are jointly typical. Encoder sends the sequence X

drawn from the conditional probability distribution Q

1w2

depending on sequences (U

, S

, W

(m), W

(m)).

• Decoding function. The decoder observes the pair of sequences (Y

, Z

) and finds the index m ∈ M such that the sequences (Y

, Z

, W

(m), W

(m)) ∈ A

(Q) are jointly typical. Decoder returns the sequence V

drawn from the conditional probability distribution Q

1w2

depending on sequences (Y

, Z

, W

(m), W

(m)).

From the properties of typical sequences, packing and covering Lemmas stated in [9] pp. 27, 46 and 208, equations (4), (5), imply there exists a n ¯ ∈

such that the expected probability of error events are bounded by ε for all n ≥ ¯ n:

E_c

P

(U

, S

) ∈ / A

(Q)

≤ ε, (6)

E_c

P

∀m ∈ M, (U

, S

, W

(m), W

(m)) ∈ / A

(Q)

≤ ε, (7)

E_c

P

∃m

6= m, s.t. (Y

, Z

, W

(m

), W

(m

)) ∈ A

(Q)

≤ ε. (8)

For all n ≥ ¯ n, there exists a code c

∈ C(n) such that sequences (U

, S

, Z

, W

(m), W

(m), X

, Y

, V

) ∈ A

(Q) are jointly typical for distribution P

(u, s, z) ⊗Q(x, w

, w

|u, s)⊗T (y|x, s) ⊗Q(v|y, z, w

, w

)

with probability more than 1 − 3ε.

The cardinality bound is stated in Lemma 6 in the Appendix. This concludes the achievability proof

B. Converse Proof

We introduce the random event of error E ∈ {0, 1} defined as follows:

E =

(

0 if Q

− Q

tv

≤ ε ⇐⇒ (U

, S

, Z

, X

, Y

, V

) ∈ A

(Q), 1 if Q

− Q

tv

> ε ⇐⇒ (U

, S

, Z

, X

, Y

, V

) ∈ / A

(Q).

(9) Consider a sequence of code c(n) ∈ C that achieves the probability distribution Q(u, s, z, x, y, v), i.e. for which the probability of error P

(c) = P (E = 1) is small. We have equations:

0 =

i=1

I (U

, S

, Y

, Z

; Y

, Z

|Y

, Z

)

−

i=1

I (Y

, Z

; U

, S

, Y

, Z

|U

, S

, Y

, Z

) (10)

≤

i=1

I (U

, S

, Y

, Z

; Y

, Z

|Y

, Z

)

−

i=1

I (Y

, Z

; U

, S

|U

, S

, Y

, Z

) (11)

=

i=1

I (W

; Y

, Z

|W

) −

i=1

I (W

; U

, S

|W

). (12) Equation (10) comes from Csiszár Sum Identity stated pp. 25 in [9].

Equation (11) comes from the properties of the mutual information.

Equation (12) comes from the introduction of the auxiliary random variables W

= (U

, S

, Y

, Z

) and W

= (Y

, Z

). The pair of random variables (W

, W

) satisfy the three Markov Chains that correspond to the set of probability distributions

Y

− − (X

, S

) − − (U

, Z

, W

, W

), (13) Z

− − (U

, S

) − − (X

, Y

, W

, W

), (14) V

− − (Y

, Z

, W

, W

) − − (U

, S

, X

). (15)

• The first Markov chain comes from the memoryless property of the channel and the fact that Y

does not belong to (W

, W

).

• The second Markov chain comes from the i.i.d. property of the source and the fact that Z

does not belong to (W

, W

).

• The third Markov chain comes from the non-causal decoding: V

is a function of the pair (Y

, Z

)

that is included in (Y

, Z

, W

, W

).

0 ≤

i=1

I (W

; Y

, Z

|W

) −

i=1

I (W

; U

, S

|W

)

≤ n ·

I (W

; Y

, Z

|W

, T ) − I(W

; U

, S

|W

, T )

(16)

≤ n ·

I (W

, T ; Y

, Z

|W

) − I(W

; U

, S

|W

, T )

(17)

≤ n · max

Q∈Q

I(W

; Y

, Z

|W

) − I (W

; U

, S

|W

)

(18)

≤ n · max

Q∈Q

I(W

; Y

, Z

|W

, E = 0) − I(W

; U

, S

|W

, E = 0) + ε

(19)

≤ n · max

Q∈Q

I(W

; Y, Z|W

) − I(W

; U, S |W

) + 2ε

. (20)

Equation (16) comes from the introduction of the uniform random variable T over {1, . . . , n} and the introduction of the corresponding mean random variables U

, S

, Z

, W

, W

, X

, Y

, Z

. Equation (17) comes from the properties of the mutual information.

Equation (18) comes from identifying W

with (W

, T ) and W

with W

and taking the maximum over the probability distributions Q that belong to

(W

, T ) and W

satisfy the Markov chains of the set of probability distributions

the Appendix.

Equation (19) comes from the empirical coordination requirement as stated in Lemma 8. Sequences are not jointly typical with small error probability P(E = 1).

Equation (20) comes from Lemma 9 that states that the probability distribution induced by the coding scheme P (U

, S

, Z

, X

, Y

, V

) = (u, s, z, x, y, v)

E = 0

is closed to the target probability

distribution Q(u, s, z, x, y, v). The continuity of the entropy function stated pp. 33 in [10] concludes

the converse proof of Theorem I.1.

II. P

C

U

S

Z

X

V

P

C D

Fig. 2. The perfect channel is defined byT_y|xs=

1

Theorem II.1 (Perfect Channel)

1) The joint probability distribution Q(u, s, z, x, v) is achievable if and only if it decomposes as follows:







Q(u, s, z) = P

(u, s, z), Z − − (U, S ) − − X,

(21) and P

(u, s, z) ⊗ Q(x|u, s) ⊗ Q(v|u, s, z, x) is achievable.

2) The probability distribution P

(u, s, z) ⊗ Q(x|u, s) ⊗ Q(v|u, s, z, x) is achievable if:

Q∈Q

max

I (W

; Z|X) + H(X) − I (X, W

; U, S )

> 0, (22)

3) The probability distribution P

(u, s, z) ⊗ Q(x|u, s) ⊗ Q(v|u, s, z, x) is not achievable if:

Q∈Q

max

I (W

; Z|X) + H(X) − I (X, W

; U, S )

< 0, (23)

where

is the set of distributions Q ∈ ∆(U × S × Z × W

× X × V) with auxiliary random variable W

that satisfies:



















 P

w2∈W2Q(u, s, z, x, w2, v)

=Pusz(u, s, z)⊗ Q(x|u, s)⊗ Q(v|u, s, z, x), Z−−(U, S)−−(X, W2),

V −−(X, Z, W2)−−(U, S).

The probability distribution Q ∈

decomposes as follows:

Q(u, s, z, x, w

, v) = P

(u, s, z) ⊗ Q(x|u, s) ⊗ Q(w

|u, s, x) ⊗ Q(v|x, z, w

).

The support of the auxiliary random variable W

is bounded by |W

| ≤ |B| + 1 with B = U × S × Z × X × Y × V .

Remark II.2 This result generalizes the coding theorem of Wyner Ziv stated in [2].

A. Achievability Proof

This proof can be obtained from achievability result of Sec. I-A, by replacing random variables W

and Y by X. Note that I (W

; Z|X) + H(X) − I(X, W

; U, S ) = I (X, W

; X, Z) − I(X, W

; U, S).

Denote by Q(u, s, z, x, w

, v) ∈

the joint probability distribution that achieves the maximum in equation (22). There exists δ > 0 and rate

≥

such that :

R

≥ I(X, W

; U, S) + δ, (24)

R

≤ I(X, W

; X, Z) − δ. (25)

• Random codebook. We generate |M| = 2

pairs of sequences (X

(m), W

(m)) with index m ∈ M drawn from the i.i.d. marginal probability distribution Q

.

• Encoding function. The encoder observes the sequences of source symbols U

∈ U

and state symbols S

∈ S

. It finds the index m ∈ M such that the sequences (U

, S

, X

(m), W

(m)) ∈ A

(Q) are jointly typical. Encoder sends the corresponding sequence X

(m) through the channel.

• Decoding function. The decoder observes the pair of sequences (X

, Z

) and finds the index m ∈ M such that the sequences (X

, Z

, X

(m), W

(m)) ∈ A

(Q) are jointly typical (for probability distribution Q

⊗ 1

). Decoder returns the sequence V

drawn from the conditional probability distribution Q

2

depending on sequences (X

, Z

, W

(m)).

From the properties of typical sequences, packing and covering Lemmas stated in [9] pp. 27, 46 and 208, equations (24), (25), imply there exists a n ¯ ∈

such that the expected probability of error events are bounded by ε for all n ≥ n: ¯

E_c

P

(U

, S

) ∈ / A

(Q)

≤ ε, (26)

E_c

P

∀m ∈ M, (U

, S

, X

(m), W

(m)) ∈ / A

(Q)

≤ ε, (27)

E_c

P

∃m

6= m, s.t. (X

, Z

, X

(m

), W

(m

)) ∈ A

(Q)

≤ ε. (28)

For all n ≥ n, there exists a code ¯ c

∈ C(n) such that sequences (U

, S

, Z

, X

(m), W

(m), V

) ∈

A

(Q) are jointly typical for distribution P

(u, s, z) ⊗ Q(x, w

|u, s) ⊗ Q(v|x, z, w

) with probability

more than 1 − 3ε.

The cardinality bound is stated in Lemma 6 in the Appendix. This concludes the achievability proof

of Theorem II.1.

B. Converse Proof

We introduce the random event of error E ∈ {0, 1} defined as follows:

E =

(

0 if Q

− Q

tv

≤ ε ⇐⇒ (U

, S

, Z

, X

, V

) ∈ A

(Q), 1 if Q

− Q

tv

> ε ⇐⇒ (U

, S

, Z

, X

, V

) ∈ / A

(Q).

(29) Consider a sequence of code c(n) ∈ C that achieves the probability distribution Q(u, s, z, x, v), i.e. for which the probability of error P

(c) = P (E = 1) is small. We have equations:

0 = H(X

, Z

|E = 0) − I (X

, Z

; U

, S

|E = 0) − H(X

, Z

|U

, S

, E = 0) (30)

≤

i=1

H(X

, Z

|E = 0) −

i=1

I(X

, Z

; U

, S

|U

, S

, E = 0)

− H(X

, Z

|U

, S

, E = 0) (31)

=

i=1

I(X

, Z

, U

, S

; X

, Z

|E = 0) −

i=1

I (X

, Z

, U

, S

; U

, S

|E = 0)

− H(X

, Z

|U

, S

, E = 0) + n · ε (32)

=

i=1

I(X

, Z

, U

, S

; X

, Z

|E = 0) −

i=1

I(X

, Z

, U

, S

; U

, S

|E = 0) +

Xn

i=1

I(Z

; X

, Z

|X

, Z

, U

, S

, E = 0) −

i=1

I(Z

; U

, S

|X

, Z

, U

, S

, E = 0)

− H(X

, Z

|U

, S

, E = 0) + n · ε (33)

=

i=1

I(X

, Z

, U

, S

; X

, Z

|E = 0) −

i=1

I(X

, Z

, U

, S

; U

, S

|E = 0) +

Xn

i=1

H(Z

|X

, Z

, U

, S

, U

, S

, E = 0) − H(X

, Z

|U

, S

, E = 0) + n · ε (34)

≤

i=1

I(X

, Z

, U

, S

; X

, Z

|E = 0) −

i=1

I(X

, Z

, U

, S

; U

, S

|E = 0) +

Xn

i=1

H(Z

|U

, S

, E = 0) − H(Z

|U

, S

, E = 0) + n · 2ε (35)

≤

i=1

I(X

, Z

, U

, S

; X

, Z

|E = 0) −

i=1

I(X

, Z

, U

, S

; U

, S

|E = 0) + n · 3ε (36)

=

i=1

I(X

, W

; X

, Z

|E = 0) − I(X

, W

; U

, S

|E = 0) + n · 3ε. (37)

Equations (30) and (31) come from the properties of the mutual information.

that implies

i=1

I(U

, S

; U

, S

|E = 0) ≤ n · ε.

Equations (33) and (34) come from the properties of the mutual information.

Equation (35) comes from the i.i.d. property of the information source (U, S, Z) and Lemma 10 in the Ap- pendix that implies

i=1

H(Z

|X

, Z

, U

, S

, U

, S

, E = 0)−

i=1

H(Z

|U

, S

, E = 0) ≤ n ·ε.

Equation (36) comes from the i.i.d. property of the information source (U, S, Z) and Lemma 10 in the Appendix that implies

i=1

H(Z

|U

, S

, E = 0) − H(Z

|U

, S

, E = 0) ≤ n · ε.

Equation (37) comes from the introduction of the auxiliary random variable W

= (X

, Z

, U

, S

). The random variable W

satisfy the Markov Chains that correspond to the set of probability distributions

corresponding to the perfect channel:

Z

− − (U

, S

) − − (X

, W

), (38) V

− − (X

, Z

, W

) − − (U

, S

). (39)

• The first Markov chain comes from i.i.d. property of the source and the fact that Z

does not belong to W

.

• The second Markov chain comes from the non-causal decoding: V

is a function of (X

, Z

) that is included in (X

, Z

, W

) = (X

, Z

, X

, Z

, U

, S

).

0 ≤

Xn

i=1

I(X

, W

; X

, Z

|E = 0) − I (X

, W

; U

, S

|E = 0) + 3nε

= n ·

I (X

, W

; X

, Z

|T, E = 0) − I (X

, W

; U

, S

|T, E = 0) + 3ε

(40)

≤ n ·

I (X

, W

, T ; X

, Z

|E = 0) − I (X

, W

, T ; U

, S

|E = 0) + 3ε

(41)

≤ n · max

Q∈Qp

I (X

, W

; X

, Z

|E = 0) − I(X

, W

; U

, S

|E = 0) + 3ε

(42)

≤ n · max

Q∈Qp

I (X, W

; X, Z ) − I (X, W

; U, S) + 4ε

. (43)

Equation (40) comes from the introduction of the uniform random variable T over {1, . . . , n} and the introduction of the corresponding mean random variables U

, S

, Z

, W

, X

, V

.

Equation (41) comes from the i.i.d. property of the information source (U, S) and Lemma 11 in the Appendix that implies I (T ; U

, S

|E = 0) = 0.

Equation (42) comes from identifying W

with (W

, T ) and taking the maximum over the probability

distributions that belong to

. This is possible since the random variables (W

, T ) satisfy the Markov

chains of the set of probability distributions

, as stated in Lemma 7 in the Appendix.

Equation (43) comes from Lemma 9 that states that the probability distribution induced by the coding scheme P (U

, S

, Z

, X

, V

) = (u, s, z, x, v)

E = 0

is closed to the target probability distribution

Q(u, s, z, x, y, v). The continuity of the entropy function stated pp. 33 in [10] concludes the converse

proof of Theorem II.1.

III. L

D

U

S

Z

X

Y

U ˆ

P

C T D

Fig. 3. Noisy channelT(y|x, s)and lossless decoding

1

Theorem III.1 (Lossless Decoding)

1) The joint probability distribution Q(u, s, z, x, y, u) ˆ is achievable if and only if it decomposes as follows:





























Q(u, s, z) = P

(u, s, z), Q(y|x, s) = T (y|x, s), Q(ˆ u|u) = 1 ^{(ˆ u|u),} Y − − (X, S) − − (U, Z), Z − − (U, S ) − − (X, Y ),

(44)

and P

(u, s, z) ⊗ Q(x|u, s) ⊗ T (y|x, s) ⊗ 1 ^{(ˆ u|u)} is achievable.

2) The probability distribution P

(u, s, z) ⊗ Q(x|u, s) ⊗ T (y|x, s) ⊗ 1 ^{(ˆ u|u)} is achievable if:

max

I(U, W

; Y, Z) − I (W

; S|U ) − H(U )

> 0, (45)

3) The probability distribution P

(u, s, z) ⊗ Q(x|u, s) ⊗ T (y|x, s) ⊗ 1 ^{(ˆ u|u)} is not achievable if:

max

I(U, W

; Y, Z) − I (W

; S|U ) − H(U )

< 0, (46)

where

is the set of distributions Q ∈ ∆(U × S × Z × W

× X × Y × U) with auxiliary random variable W

that satisfies:



















 P

w1∈W1Q(u, s, z, w1, x, y,ˆu)

=Pusz(u, s, z)⊗ Q(x|u, s)⊗ T(y|x, s)⊗

1

Y −−(X, S)−−(U, Z, W1), Z−−(U, S)−−(X, Y, W1).

The probability distribution Q ∈

decomposes as follows:

Q(u, s, z, w

, x, y, u) = ˆ P

(u, s, z) ⊗ Q(x|u, s) ⊗ Q(w

|u, s, x) ⊗ T (y|x, s) ⊗ Q(ˆ u|u).

The support of the auxiliary random variables W

is bounded by |W

| ≤ |B| + 1 with B = U × S × Z × X × Y × V .

Remark III.2 This result was already stated in [4] and [5] with a more restrictive lossless decoding

constraint: P ( ˆ U

6= U

) ≤ ε. It generalizes the coding theorem of Gel’fand Pinsker stated in [3].

A. Achievability Proof

This proof can be obtained from the achievability result of Sec. I-A, by replacing random variables W

and V by U . Note that I(U, W

; Y, Z) − I (W

; S|U ) − H(U ) = I (W

, U ; Y, Z) − I(W

, U ; U, S ).

Denote by Q(u, s, z, w

, x, y, u) ˆ ∈

the joint probability distribution that achieves the maximum in equation (45). There exists δ > 0 and rate

≥

such that :

R

≥ I (W

, U ; U, S) + δ, (47)

R

≤ I (W

, U ; Y, Z) − δ. (48)

• Random codebook. We generate |M| = 2

pairs of sequences (W

(m), U

(m)) with index m ∈ M drawn from the i.i.d. marginal probability distribution Q

.

• Encoding function. The encoder observes the sequences of source symbols U

∈ U

and state symbols S

∈ S

. It finds the index m ∈ M such that the sequences (U

, S

, W

(m), U

(m)) ∈ A

(Q) are jointly typical (for probability distribution Q

⊗ 1

). Encoder sends a se- quence X

drawn from the conditional probability distribution Q

1

depending on sequences (S

, W

(m), U

(m)).

• Decoding function. The decoder observes the pair of sequences (Y

, Z

) and finds the index m ∈ M such that the sequences (Y

, Z

, W

(m), U

(m)) ∈ A

(Q) are jointly typical. Decoder returns the sequence U ˆ

= U

(m).

From the properties of typical sequences, packing and covering Lemmas stated in [9] pp. 27, 46 and 208, equations (47), (48), imply there exists a n ¯ ∈

such that the expected probability of error events are bounded by ε for all n ≥ n: ¯

Ec

P

(U

, S

) ∈ / A

(Q)

≤ ε, (49)

Ec

P

∀m ∈ M, (U

, S

, W

(m), U

(m)) ∈ / A

(Q)

≤ ε, (50)

E_c

P

∃m

6= m, s.t. (Y

, Z

, W

(m

), U

(m

)) ∈ A

(Q)

≤ ε. (51)

For all n ≥ n, there exists a code ¯ c

∈ C(n) such that sequences (U

, S

, Z

, W

(m), X

, Y

, U ˆ

) ∈

A

(Q) are jointly typical for distribution P

(u, s, z) ⊗ Q(x, w

|u, s) ⊗ T (y|x, s) ⊗ 1 ^{(ˆ u|u) with}

probability more than 1 − 3ε.

The cardinality bound is stated in Lemma 6 in the Appendix. This concludes the achievability proof of Theorem III.1.

An alternative achievability proof based on superposition coding can be found in [4] and [5].

B. Converse Proof

We introduce the random event of error E ∈ {0, 1} defined as follows:

E =

(

0 if Q

− Q

tv

≤ ε ⇐⇒ (U

, S

, Z

, X

, Y

, U ˆ

) ∈ A

(Q), 1 if Q

− Q

tv

> ε ⇐⇒ (U

, S

, Z

, X

, Y

, U ˆ

) ∈ / A

(Q).

(52) Consider a sequence of code c(n) ∈ C that achieves the probability distribution Q(u, s, z, x, y, u), ˆ i.e.

for which the probability of error P

(c) = P (E = 1) is small. We have equations:

n · H(U ) = H(U

) (53)

= H(E) + H(U

|E) − H(E|U

) (54)

≤ 1 + P(E = 0) · H(U

|E = 0) + P(E = 1) · H(U

|E = 1) (55)

≤ 1 + H(U

|E = 0) + P (E = 1) · n · log

|U | (56)

≤ H(U

|E = 0) + n · ε. (57)

Equation (53) comes from the i.i.d. property of the source.

Equations (54), (55) and (56) comes from the properties of the mutual information.

Equation (57) comes from the hypothesis of small error probability P

(c) = P (E = 1) and large length

n ∈

of the codewords, hence

+ P(E = 1) · log

|U | ≤ ε.

H(U

|E = 0) = I(U

; Y

, Z

|E = 0) + H(U

|Y

, Z

, E = 0) (58)

= I(U

; Y

, Z

|E = 0) + n · ε (59)

=

i=1

I(U

; Y

, Z

|Y

, Z

, E = 0) + n · ε (60)

≤

i=1

I(U

, Y

, Z

; Y

, Z

|E = 0) + n · ε (61)

=

i=1

I(U

, Y

, Z

, S

; Y

, Z

|E = 0)

−

i=1

I(S

; Y

, Z

|U

, Y

, Z

, E = 0) + n · ε (62)

=

i=1

I(U

, Y

, Z

, S

; Y

, Z

|E = 0)

−

i=1

I(S

; Y

, Z

|U

, S

, E = 0) + n · ε (63)

=

i=1

I(U

, U

, Y

, Z

, S

; Y

, Z

|E = 0)

−

i=1

I(S

; U

, Y

, Z

, S

|U

, E = 0) + n · 2ε (64)

=

i=1

I(U

, W

; Y

, Z

|E = 0) −

i=1

I(W

; S

|U

, E = 0) + n · 2ε. (65) Equation (58) comes from the properties of the mutual information.

Equation (59) comes from Fano’s inequality that implies H(U

|Y

, Z

, E = 0) ≤ n · ε, as stated in Lemma 1.

Equation (60), (61) and (62) come from the properties of the mutual information.

Equation (63) comes from Csiszár Sum Identity stated pp. 25 in [9].

Equation (64) comes from the independence of random variables (U

, S

) with (U

, S

) that implies that

i=1

I(S

; U

, S

|U

, E = 0) ≤ n · ε, as stated in Lemma 10 in the Appendix.

Equation (65) comes from the introduction of the auxiliary random variable W

= (U

, Y

, S

).

The Markov chain property Y

− − (X

, S

) − − (U

, W

) is satisfied for all i ∈ {1, . . . , n} since the

channel is memoryless and Y

do not belong to W

. The random variable W

belongs to the set of

probability distributions

for lossless decoding.

H(Uⁿ|E = 0) ≤ Xn i=1

I(Ui, W1,i;Yi, Zi|E= 0)− Xn i=1

I(W1,i;Si|Ui, E= 0) +n·2ε

= n·

I(UT, W1,T;YT, ZT|T, E= 0)−I(W1,T;ST|UT, T, E = 0) + 2ε

(66)

= n·

I(UT, W1,T, T;YT, ZT|E= 0)−I(W1,T, T;ST|UT, E= 0)

− I(T;YT, ZT|E= 0) +I(T;ST|UT, E= 0) + 2ε

(67)

≤ n·

I(UT, W1,T, T;YT, ZT|E= 0)−I(W1,T, T;ST|UT, E= 0) + 2ε

(68)

≤ n·max

Q∈Q

I(UT, W1;YT, ZT|E= 0)−I(W1;ST|UT, E= 0) + 2ε

(69)

≤ n·max

Q∈Q

I(U, W1;Y, Z)−I(W1;S|U) + 3ε

. (70)

Equation (66) comes from the introduction of the uniform random variable T over {1, . . . , n} and the corresponding mean random variables U

, S

, Z

, W

, X

, Y

, U ˆ

.

Equation (67) comes from the properties of the mutual information.

Equation (68) comes from the independence of T with (U

, S

) that implies I(T ; S

|U

, E = 0) = 0, as stated in Lemma 11 in the Appendix. The memoryless property of the channel guarantees that the pair of random variable (W

, T ) satisfies the Markov chain Y

− − (X

, S

) − − (U

, W

, T ). Hence the pair (W

, T ) belongs to the set of probability distributions

, as stated in Lemma 7 in the Appendix.

Equation (69) comes from taking the maximum over the probability distributions Q that belong to

. Equation (70) comes from Lemma 9 that states that the probability distribution induced by the coding scheme P (U

, S

, Z

, X

, Y

, U ˆ

) = (u, s, z, x, y, u) ˆ

E = 0

is closed to the target probability distribution Q(u, s, z, x, y, u). The continuity of the entropy function stated pp. 33 in [10] concludes. ˆ

Combining equations (57) and (70) gives equation (71). It is satisfied for all c(n) ∈ C that achieves the probability distribution Q(u, s, z, x, y, u). ˆ

H(U ) ≤ max

Q∈Q

I(U, W

; Y, Z) − I(W

; S|U ) + 4ε

. (71)

This concludes the converse proof of Theorem III.1.

Lemma 1 By Fano’s inequality, we have the following equation:

H(U

|Y

, Z

, E = 0) ≤ n · ε. (72)

Proof III.3 (Proof of Lemma 1) The decoding g : Y

× Z

7→ U

is deterministic, hence H( ˆ U

|Y

, Z

, E = 0) = 0. We have the following equations:

H(U

|Y

, Z

, E = 0) ≤ H(U

, U ˆ

|Y

, Z

, E = 0) (73)

≤ H( ˆ U

|Y

, Z

, E = 0) + H(U

| U ˆ

, Y

, Z

, E = 0) (74)

≤ H(U

| U ˆ

, E = 0) (75)

≤ n ·

H(U | U ˆ ) + ε

(76)

= n · ε. (77)

Equations (73) and (74) come from the properties of the entropy.

Equation (75) comes from the deterministic decoding: U ˆ

is a deterministic function of (Y

, Z

).

Equation (76) comes from the cardinality bound log

u

s.t. u

∈ A

(ˆ u

) ≤ n ·

H(U | U ˆ ) + ε , on the set of sequences u

that are jointly typical with u ˆ

.

Equation (77) comes from the target joint distribution Q(u, s, z, x, y, u) ˆ that satisfy Q(ˆ u|u) = 1 ^{(ˆ u|u)}

hence H(U | U ˆ ) = 0.

IV. S

S

C

U

S

Z

X

Y

V

P

P

C T D

Fig. 4. Random variables of the source(U, Z, V)are independent of the random variables of the channel(S, X, Y).

Theorem IV.1 (Separation between Source and Channel)

1) The product of probability distribution Q(u, z, v)⊗Q(s, x, y) is achievable if and only if it decomposes as follows:















Q(u, z) = P

(u, z), Q(s) = P

(s),

Q(y|x, s) = T (y|x, s).

(78)

and P

(u, z) ⊗ Q(v|u, z) ⊗ P

(s) ⊗ Q(x|s) ⊗ T (y|x, s) is achievable.

2) Joint probability distribution P

(u, z) ⊗ Q(v|u, z) ⊗ P

(s) ⊗ Q(x|s) ⊗ T (y|x, s) is achievable if:

max

I (W

; Y ) + I (W

; Z) − I(W

; S) − I(W

; U )

> 0, (79)

3) Joint probability distribution P

(u, z) ⊗ Q(v|u, z) ⊗ P

(s) ⊗ Q(x|s) ⊗ T (y|x, s) is not achievable if:

max

I (W

; Y ) + I (W

; Z) − I(W

; S) − I(W

; U )

< 0, (80)

where

is the set Q ∈ ∆(U × Z × W

× V ) ×∆(S × W

× X × Y ) of product of probability distributions with auxiliary random variables (W

, W

) that satisfies:



























 P

(w1,w2)∈W1×W2Q(u, z, w2, v)⊗ Q(s, x, w1, y)

=Puz(u, z)⊗ Q(v|u, z)⊗ Ps(s)⊗ Q(x|s)⊗ T(y|x, s), Y −−(X, S)−−W1,

Z−−U−−W2, V −−(Z, W2)−−U.

The probability distribution Q ∈

decomposes as follows:

P

(u, z) ⊗ Q(w

|u) ⊗ Q(v|z, w

) ⊗ P

(s) ⊗ Q(x|s) ⊗ Q(w

|s, x) ⊗ T (y|x, s).

The supports of the auxiliary random variables (W

, W

) are bounded by max(|W

|, |W

|) ≤ (|B| + 1) · (|B| + 2) with B = U × S × Z × X × Y × V.

Remark IV.2 This separation result was already stated in [6] for a given distortion level and a given

channel cost.