Heterogeneous Coded Distributed Computing

Brief Summary: The last problem that we addressed in this thesis is the coded dis-tributed computing problem with heterogeneous computational powers. For a MapReduce-like framework with given reduce load vectorL and a total computation loadt, we have provided a file assignment and shuffle scheme which we proved to be order-optimal under the two reasonable assumptions of homogeneous file assignment and one-shot shuffling.

Similarly as it happened for the topology-aware heterogeneous shared-cache problem, the order-optimality is obtained with a file assignment strategy that assigns more files of the dataset, i.e. more computation load, to those nodes that will have to compute more output functions in the reduce phase. Indeed, the proposed scheme can be seen as a D2D version of the scheme presented in Section 4.4. Furthermore, we have also seen that in order to minimize the reduce time, as expected, one should assign to each computing node

a number of output functions that is proportional to its computational power. Finally, we also provided a lower bound on the communication load for an arbitrary output function assignment and an arbitrary allocation of the total computation load across the nodes.

Discussions for Possible Future Directions: Part of our future effort on this topic will be to extend the shuffle scheme proposed in this thesis to the case with arbitrary computation load and arbitrary reduce load at each node, for which we have already provided a converse bound. For this latter problem, some recent advances have been recently published (see the related works subsection in 8.1.1). Finally, we believe that — while we mostly focused on minimizing the shuffle time and, separately, the reduce time — one should consider the much harder problem of identifying the optimal file assignment, shuffle scheme and output function assignment that jointly minimize the sum of the map, shuffle and reduce time for a MapReduce-like distributed computing system with heterogenenous computational powers.

Appendix of Chapter 3

A.1 An Illustrative Example for the Converse

We here give an example of deriving the converse for Theorem 2, emphasizing on how to convert the caching problem to the index-coding problem, and how to choose acyclic subgraphs. We consider the case of havingK = 9 receiving users, and a transmitter with N₀ = 2 transmit antennas having access to a library of N = 9 files of unit size. We also assume that there are Λ = 3 caching nodes, of total normalized cache size t= Λγ. We will focus on deriving the bound for the cache occupancy vectorL= (4,3,2), meaning that we are interested in the setting where one cache is associated to 4 users, one cache to 3 users and one cache associated to 2 users.

Each file W⁽ⁿ⁾ is split into 2^Λ = 8 disjoint subfiles W_T⁽ⁱ⁾,T ∈ 2^[3] where each T describes the set of helper nodes in which W_T⁽ⁱ⁾ is cached. For instance,W₁₃⁽¹⁾ refers to the part of file W⁽¹⁾ that is stored in the first and third caching nodes.

As a first step, we present the construction of the setDL. To this end, let us start by considering the demand d= (1,2,3,4,5,6,7,8,9) and one of the 6 permutationsπ∈S3; for example, let us start by consideringπ(1) = 2, π(2) = 3, π(3) = 1. Toward reordering d to reflectL, we construct

d⁰₁ = (1,2,3,4), d⁰₂ = (5,6,7), d⁰₃ = (8,9) to obtain the reordered demand vector

d(U) = (d⁰_π−1(1),d⁰_π−1(2),d⁰_π−1(3))

= (d⁰₃,d⁰₁,d⁰₂)

which in turn yieldsd₁= (8,9),d₂= (1,2,3,4),d₃= (5,6,7). Similarly, we can construct the remaining 5 demands d(U) associated to the other 5 permutations π∈S₃. Finally, the procedure is repeated for all other worst-case demand vectors. These vectors are part of setDL.

With the users demandsd(U) known to the server, the delivery problem is translated into an index coding problem with a side information graph ofK2^Λ−1 = 9·2² nodes. For

each requested fileW^(dλ(j)), we write down the 4 subfiles that the requesting user does not have in its assigned cache. Hence, a given user of the caching problem requiring 4 subfiles from the main server, is replaced by 4 different new users in the index coding problem. Each of these users request a different subfile and are connected to the same cacheλas the original user. The nodes of the 6 side-information graphs corresponding to the aforementioned vectors d(U) (one for each permutation π ∈ S3) for demand d= (1,2,3,4,5,6,7,8,9), are depicted in Figure A.1.

For each side-information graph, we develop a lower bound as in Lemma 1. We recall that the lemma applies to acyclic subgraphs, which we create as follows; for each permutation¹ σ ∈S₃, a set of nodes forming an acyclic subgraph is

{W_T^(dσ(1)(j))

1 }^|Uj=1^σ(1)|for allT1⊆{1,2,3}\{σ(1)}, {W_T^(dσ(2)(j))

2 }^|U_j=1^σ(2)|for allT2⊆{1,2,3}\{σ(1), σ(2)}, {W_T^(dσ(3)(j))

3 }^|U_j=1^σ(3)|for allT3⊆{1,2,3}\{σ(1), σ(2), σ(3)}.

Based on this construction of acyclic graphs, our task now is to choose a permutation σ_s ∈ S₃ that forms the maximum-sized acyclic subgraph. For the case where d₁ = (8,9),d₂ = (1,2,3,4) andd₃ = (5,6,7), it can be easily verified that such a permutation σs is the one with σs(1) = 2,σs(2) = 3 and σs(3) = 1. In Figure A.1, for each of the six graphs, we underline the nodes corresponding to the acyclic subgraph that is formed by such permutation σ_s. The outer bound now involves adding the sizes of these chosen (underlined) nodes. For example, for the demandd(U) = ((8,9),(1,2,3,4),(5,6,7)) (this

corresponds to the lower center graph), the lower bound in (3.42) becomes T^∗(d(U),Z)≥1

2(|W_∅⁽¹⁾|+|W₁⁽¹⁾|+|W₃⁽¹⁾|+|W₁₃⁽¹⁾|+|W_∅⁽²⁾| +|W₁⁽²⁾|+|W₃⁽²⁾|+|W₁₃⁽²⁾|+|W_∅⁽³⁾|+|W₁⁽³⁾| +|W₃⁽³⁾|+|W₁₃⁽³⁾|+|W_∅⁽⁴⁾|+|W₁⁽⁴⁾|+|W₃⁽⁴⁾| +|W₁₃⁽⁴⁾|+|W_∅⁽⁵⁾|+|W₁⁽⁵⁾|+|W_∅⁽⁶⁾|+|W₁⁽⁶⁾|

+|W_∅⁽⁷⁾|+|W₁⁽⁷⁾|+|W_∅⁽⁸⁾|+|W₀⁽⁹⁾|). (A.1) The lower bounds for the remaining 5 vectorsd(U) for the samed= (1,2,3,4,5,6,7,8,9), are given in a similar way, again by adding the (underlined) nodes of the corresponding acyclic subgraphs (again see Figure A.1).

Subsequently, the procedure is repeated for allP(N, K) =K! = 9! worst-case demand vectors d∈ Dwc. Finally, all theP(N, K)·Λ! = 9!·3! bounds are averaged to get

T(L,Z)≥ 1

2 1 9!·3!

X

d(U)∈DL

X

λ∈[3]

L_σs(λ)

X

j=1

X

Tλ⊆[3]\{σs(1),...,σs(λ)}

|W_T^dσs(λ)(j)

λ | (A.2)

1We caution the reader not to confuse the current permutations (σ) that are used to construct large-sized acyclic graphs, with the aforementioned permutationsπwhich are used to constructDL.

d₁ = (1,2,3,4),d₂ = (5,6,7), d₁ = (1,2,3,4),d₂ = (8,9), d₁= (5,6,7),d₂ = (1,2,3,4),

Figure A.1 – Nodes of the side information graphs corresponding to demand vector d = (1,2,3,4,5,6,7,8,9) and cache occupancy vectorL= (4,3,2).

which is rewritten as where the proof of the transition from (A.3) to (A.4) and from (A.4) to (A.5) can be found in the general proof (Section 3.4).