CIRCUIT MODEL AND PHYSICAL REALIZATIONS

In a sense, the only sure way to predict the performance of a parallel architecture on a given set of problems is to actually build the machine and run the programs on it. Because this is often impossible or very costly, the next best thing is to model the machine at the circuit level, so that all computational and signal propagation delays can be taken into account. Unfortunately, this is also impossible for a complex supercomputer, both because generating and debugging detailed circuit specifications are not much easier than a full-blown implementation and because a circuit simulator would take eons to run the simulation.

Despite the above observations, we can produce and evaluate circuit-level designs for specific applications. The example of sorting networks will be covered in Chapter 7, where we take the number of two-input compare–exchange circuits as a measure of cost and the depth of the circuit as being indicative of delay. Additional examples, covering the fast Fourier transform, parallel prefix computations, and dictionary operations, will be provided in Chapter 8 where similar cost and delay criteria are used.

Figure 4.10. lntrachip wire delay as a function of wire length.

MODELS OF PARALLEL PROCESSING 81 A more precise model, particularly if the circuit is to be implemented on a dense VLSI chip, would include the effect of wires, in terms of both the chip area they consume (cost) and the signal propagation delay between and within the interconnected blocks (time). In fact, in modern VLSI design, wire delays and area are beginning to overshadow switching or gate delays and the area occupied by devices, respectively.

The rightmost column in Table 4.2 indicates which network topologies have local or short links, thus being less likely to suffer from long interprocessor signal propagation delays.

Figure 4.10 depicts the expected wire delays on a 1B-transistor chip of the future as a function of wire length, assuming the use of copper wires that are less resistive, and thus faster, than today’s aluminum wires [Parh98]. The circles designate the estimated wire lengths, and thus interprocessor propagation delays, for a 256-processor chip with three different architec-tures. It is seen that for the hypercube architecture, which has nonlocal links, the interproc-essor wire delays can dominate the intraprocinterproc-essor delays, thus making the communication step time much larger than that of the mesh- and torus-based architectures.

Determining the area requirements and maximum wire lengths for various interconnec-tion topologies has been a very active research area in the past two decades. At times, we can determine bounds on area and wire-length parameters based on network properties, without having to resort to detailed specification and layout with VLSI design tools. For example, in 2D VLSI implementation, the bisection width of a network yields a lower bound on its layout area in an asymptotic sense. If the bisection width is B, the smallest dimension of the chip should be at least Bw, where w is the minimum wire width (including the mandatory interwire spacing). The area of the chip is thus Ω(B²). If the bisection width is Θ( ), as in 2D meshes, then the area lower bound will be linear in the number p of processors. Such an architecture is said to be scalable in the VLSI layout sense. On the other hand, if the bisection width is Θ(p), as in the hypercube, then the area required is a quadratic function of p and the architecture is not scalable.

The following analogy is intended to reinforce the importance of the above discussion of physical realizations and scalability [Hart86]. You have all read science-fiction stories, or seen sci-fi movies, in which scaled up ants or other beasts destroy entire towns (top panel of Fig. 4.11). Let us say that a 0.5-cm ant has been scaled up by a factor of 10⁴ so that its new length is 50 m. Assuming linear scaling, so that the enlarged ant looks exactly like an ordinary ant, the ants leg has thickened by the same factor (say from 0.01 cm to 1 m). The weight of the ant, meanwhile, has increased by a factor of 10¹², say from 1 g to 1 M tons! Assuming that the original ant legs were just strong enough to bear its weight, the leg thickness must in fact increase by a factor of = 10⁶, from 0.01 cm to 100 m, if the ant is not to collapse under its own weight. Now, a 50-m-long ant with legs that are 100 m thick looks nothing like the original ant!

Power consumption of digital circuits is another limiting factor. Power dissipation in modern microprocessors grows almost linearly with the product of die area and clock frequency (both steadily rising) and today stands at a few tens of watts in high-performance designs. Even if modern low-power design methods succeed in reducing this power by an order of magnitude, disposing of the heat generated by 1 M such processors is indeed a great challenge.

82 INTRODUCTION TO PARALLEL PROCESSING

Figure 4.11. Pitfalls of scaling up.

PROBLEMS

4.1. Associative processing

A bit-serial associative memory is capable of searching, in one memory access cycle, a single bit slice of all active memory words for 0 or 1 and provides the number of responding words in the form of an unsigned integer. For example, the instruction Search(0, i) will yield the number of active memory words that store the value 0 in bit position i. It also has instructions for activating or deactivating memory words based on the results of the latest search (i.e., keep only the responders active or deactivate the responders). Suppose that initially, all words are active and the memory stores m unsigned integers.

a.

b.

c . Extend the algorithm of part (b) to deal with signed integers in signed-magnitude format.

Extend the algorithm of part (b) to deal with signed integers in 2’s-complement format.

Hint: In a 2’s-complement number, the sign bit carries a negative weight so that 1010 represents –8 + 2 = –6.

Devise an AM algorithm to find the largest number stored in the memory.

Devise an AM algorithm for finding the kth largest unsigned integer among the m stored values.

d.

MODELS OF PARALLEL PROCESSING 83 4.2. Associative processing

With the same assumptions as in Problem 4.1:

a . Devise an AM algorithm to find the exclusive-OR (checksum) of the values in all memory words.

b . Devise an AM algorithm for finding the arithmetic sum of the values stored in all memory words. Assume unsigned or 2’s-complement numbers, with all results representable in a single word.

c . Repeat part (b), this time assuming that some results may not be representable in a single word.

4.3. Cellular automata synchronization

Cellular automata form abstract models of homogeneous massively parallel computers. Con-sider the special case of a linear array of finite automata viewed as modeling a firing squad.

All cells are identical and the length of the linear array is unknown and unbounded. We would like to synchronize this firing squad so that all cells enter the special “fire” state at exactly the same time. The synchronization process begins with a special input to the leftmost cell of the array (the general). Cells (soldiers) can only communicate with their left and right neighbor in a synchronous fashion and the only global signal is the clock signal, which is distributed to all cells. Hint : Try to identify the middle cell, or the middle two cells, of the array, thus dividing the problem into two smaller problems of exactly the same size.

4.4. Image-processing computer

Two types of operations are commonly used in image processing. Assume that an image is represented by a 2D matrix of 0s and 1s corresponding to dark and light pixels, respectively.

Noise removal has the goal of removing isolated 0s and 1s (say those that have at most one pixel of the same type among their eight horizontally, vertically, or diagonally adjacent neighbors). Smoothing is done by replacing each pixel value with the median of nine values consisting of the pixel itself and its eight neighbors.

a . Discuss the design of a 2D array of simple cells capable of noise removal or smoothing.

b . Propose suitable generalizations for noise removal and smoothing if each pixel value is a binary integer (say 4 bits wide) representing a gray level.

c . Sketch the modifications required to the cells of part (a) if part (b) is to be implemented.

4.5. MISD architecture

In the MISD architecture depicted in Fig. 4.2, the rate of data flow or throughput in a synchronous mode of operation is dictated by the execution time of the slowest of the five blocks. Suppose that the execution times for the five blocks are not fixed but rather have uniform distributions in the interval from a lower bound li to an upper bound u_i, 1 ≤ i ≤ 5, where l_i and u_i are known constants. Discuss methods for improving the throughput beyond the throughput 1/max (u_i) of a synchronous design.

4.6. Cache coherence protocols

Study the snooping and directory-based cache coherence protocols (see, e.g., [Patt96], pp.

655–666 and 679–685). Then pick one example of each and do the following exercises:

a . Present one advantage for each protocol over the other.

b . Discuss an application, or a class of applications, for which the chosen snooping cache coherence protocol would perform very poorly.

c . Repeat part (b) for the chosen directory-based protocol.

84 INTRODUCTION TO PARALLEL PROCESSING 4.7. Topological parameters of interconnection networks

Add entries corresponding to the following topologies to Table 4.2.

a . An X-tree; a complete binary tree with nodes on the same level connected as a linear array.

b . A hierarchical bus architecture with a maximum branching factor b (b = 4 in Fig. 4.9).

c . A degree-4 chordal ring with skip distance s; i.e., a p-node ring in which Processor i is also connected to Processors i ± s mod p, in addition to Processors i ± 1 mod p.

4.8. Hierarchical-bus architectures

Consider the hierarchical multilevel bus architecture shown in Fig. 4.9, except that each of the low-level clusters consists of four processors. Suppose that we want to emulate a 4 × 6 or 6 × 4 mesh architecture on this bus-based system. Consider the shearsort algorithm described in Section 2.5 and assume that each transfer over a shared bus to another processor or to a switch node takes unit time. Ignore computation time as well as any control overhead, focusing only on communication steps.

a . How long does this system take to emulate shearsort on a 4 × 6 mesh if each processor holds a single data item and each cluster emulates a column of the mesh?

b . How long does this system take to emulate shearsort on a 6 × 4 mesh (cluster = row)?

c . Devise a method for performing a parallel prefix computation on this architecture.

4.9. The LogP model

Using the LogP model, write an equation in terms of the four parameters of the model for the total time needed to do a “complete exchange,” defined as each processor sending p– 1 distinct messages, one to each of the other p – 1 processors.

4.10. The BSP and LogP models

A array holds the initial data for an iterative computation. One iteration involves computing a new value for each array element based on its current value and the current values of its eight horizontally, vertically, and diagonally adjacent array elements. Each of the p processors stores a k × k block of the array at the center of a (k + 2) × (k + 2) local matrix, where the top/bottom rows and leftmost/rightmost columns represent data acquired from neighboring blocks and available at the beginning of an iteration. Processors dealing with blocks at the edge of the large array simply copy values from edge rows/columns in lieu of receiving them from a neighboring block.

a . Formulate the above as a BSP computation.

b . If you could choose the BSP parameter L at will, how would you go about the selection process for the above computation?

c . Estimate the running time for m iterations, assuming the BSP model.

d . Repeat Part (c) for the LogP model.

4.11. Physical realizations

Consider the column labeled “Local Links?” in Table 4.2. It is obvious how a 2D mesh can be laid out so that all links are local or short.

a . Describe a 2D layout for a 2D torus that has short local links. Hint: Use folding.

b . The 3D mesh and torus networks have also been characterized as having short local links.

However, this is only true if a 3D realization is possible. Show that a 3D mesh (and thus torus) cannot be laid out with short local links in two dimensions.

c . Show that any network whose diameter is a logarithmic function of the number of nodes cannot be laid out in two or three dimensions using only short local links.

MODELS OF PARALLEL PROCESSING 85 4.12. Physical realizations

a. Describe the network diameter, bisection width, and node degree of each of the networks listed in Table 4.2 as an asymptotic function of the number p of nodes (e.g., the 1D mesh has diameter, bisection, and degree of Θ(p), Θ(1), and Θ(1), respectively).

b. Using the results of part (a), compare the architectures listed in terms of the composite figure of merit “degree × diameter,” in an asymptotic way.

c. Obtain an asymptotic lower bound for the VLSI layout area of each architecture based on its bisection width.

d. The area–time product is sometimes regarded as a good composite figure of merit because it incorporates both cost and performance factors. Assuming that delay is proportional to network diameter, compute the asymptotic area–time figure of merit for each of the architectures, assuming that the area lower bounds of part (c) are in fact achievable.

4.13. Physical realizations

The average internode distance of an interconnection network is perhaps a more appropriate indicator of its communication latency than its diameter. For example, if the network nodes are message routers, some of which are randomly connected to processors or memory modules in a distributed shared-memory architecture, then the average internode distance is a good indicator of memory access latency.

a. Compute the average internode distance d _avgfor as many of the interconnection networks listed in Table 4.2 as you can.

b. If a processor issues a memory access request in every clock cycle, there is no routing or memory access conflict, and the memory access time is t_macycles, then, on the average, each processor will have 2 d_avg+ t_ma outstanding memory access requests at any given time.

Discuss the implications of the above on the scalability of the architectures in part (a).

REFERENCES AND SUGGESTED READING

Culler, D. E., et al., “A Practical Model of Parallel Computation,” Communications of the ACM, Vol.

39, No. 11, pp. 78–85, November 1996.

Fromm, R., et al., “The Energy Efficiency of IRAM Architectures,” Proc. Int. Conf. Computer Architecture, 1997, pp. 327–337.

Garzon, M., Models of Massive Parallelism: Analysis of Cellular Automata and Neural Networks, Springer, 1995.

Hartmann, A. C., and J. D. Ullman, “Model Categories for Theories of Parallel Systems,” Microelec-tronics and Computer Corporation, Technical Report PP-341-86; reprinted in Parallel Computing:

Theory and Comparisons, by G. J. Lipovski and M. Malek, Wiley, 1987, pp. 367–381.

Jain, A. K., J. Mao, and K. M. Mohiuddin, “Artificial Neural Networks: A Tutorial,” IEEE Computer, Vol. 29, No. 3, pp. 31–44, March 1996.

Parhami, B., “Panel Assesses SIMD’s Future,” IEEE Computer, Vol. 28, No. 6, pp. 89–91 June 1995.

For an unabridged version of this report, entitled “SIMD Machines: Do They Have a Significant Future?,” see IEEE Computer Society Technical Committee on Computer Architecture Newsletter, pp. 23–26, August 1995, or ACM Computer Architecture News, Vol. 23, No. 4, pp. 19–22, September 1995.

Parhami, B., “Search and Data Selection Algorithms for Associative Processors,” in Associative Processing and Processors, edited by A. Krikelis and C. Weems, IEEE Computer Society Press, 1997, pp. 10–25.

Parhami, B., and D.-M. Kwai, “Issues in Designing Parallel Architectures Using Multiprocessor and Massively Parallel Microchips,” in preparation.

Patterson, D. A., and J. L. Hennessy, Computer Architecture: A Quantitative Approach, 2nd ed., Morgan Kaufmann, 1996.

86 INTRODUCTION TO PARALLEL PROCESSING

[PPCA]

[Vali90]

Proc. Int. Workshop on Parallel Processing by Cellular Automata and Arrays, 3rd, Berlin, 1986; 6th, Potsdam, Germany (Parcella’94).

Valiant, L. G., “A Bridging Model for Parallel Computation,” Communications of the ACM, Vol. 33, No. 8, pp. 103–111, August 1990.