Parallel Algorithm Complexity
3.6. SOLVING RECURRENCES
The simplest method for solving recurrences is through unrolling. The method is best illustrated through a sequence of examples. In all examples below, ƒ(1) = 0 is assumed.
method exists for solving recurrences, we will review several methods along with examples.
(parallel) algorithms, it is instructive to discuss methods for their solution. As no general algorithm complexity. Because such recurrences arise quite frequently in the analysis of In our discussion of the divide-and-conquer method in Section 3.5, we presented the recurrence T (n) = T( n/2) + 2 log2n as an example of what might transpire in analyzing
PARALLEL ALGORITHM COMPLEXITY 59
Alternate solution for the recurrence ƒ( n) = 2 ƒ ( n /2) + n:
Rewrite the recurrence as
and denote ƒ(n)/n by h(n) to get
This is the same as Example 2 above and leads to
Another method that we will find useful, particularly for recurrences that cannot be easily unrolled, is guessing the answer to the recurrence and then verifying the guess by substitution. In fact, the method of substitution can be used to determine the constant multiplicative factors and lower-order terms once the asymptotic complexity has been established by other methods.
As an example, let us say that we know that the solution to Example 1 above is ƒ (n ) = Θ( n ²). We write ƒ (n) = a n² + g (n), where g(n) = o( n ²) represents the lower-order terms.
Substituting in the recurrence equation, we get
This equation simplifies to
Choose a = 1/2 in order to make g(n) = o(n ²) possible. Then, the solution to the recurrence g(n) = g(n – 1) + 1/2 is g(n) = n /2 – 1, assuming g (1) = 0. The solution to the original recurrence then becomes ƒ (n ) = n ²/2 + n /2 – 1, which matches our earlier result based on unrolling.
60 INTRODUCTION TO PARALLEL PROCESSING Unrolling recurrences can be tedious and error-prone in some cases. The following general theorem helps us establish the order of the solution to some recurrences without unrolling [Bent80].
THEOREM 3.1 (basic theorem for recurrences). Given the recurrence ƒ(n) = aƒ (n/b) + h(n), where a and b are constants and h is an arbitrary function, the asymptotic solution to the recurrence is
The recurrence given in the statement of Theorem 3.1 arises, for example, if we decompose a given problem of size n into b problems of size n/b, with these smaller problems solved in a batches (e.g., because the available resources, typically processors, are inadequate for solving all subproblems concurrently).
The function h ( n ) represents the time needed to decompose the problem and for obtaining the overall solution from the solutions to the subproblems. Therefore, a = 1 typically means that we have enough processors to solve all b subproblems concurrently, a = b means we have just one processor, so the subproblems are solved sequentially, and 1 < a < b means we can solve some of the problems concurrently but that the number of processors is inadequate to deal with all of them concurrently. An example of this last situation might be when the number of processors required is sublinear in problem size (problems with 1/b = 1/4 the size require half as many processors, say, when thus dictating a = 2 passes for solving the four subproblems).
each can say that the decomposition/merging overhead is more or less constant across the recursive iterations. In the third case, the overhead decreases geometrically, so the first term in the unrolling, i.e., h( n), dictates the complexity. Finally, in the first case, the unrolled terms form an increasing geometric progression, making the last term in the unrolling dominant. This last term can be obtained as follows:
Note that the first and third cases in the statement of Theorem 3.1 are separated from the middle case by ±ε in the exponent of n. Let us consider the middle case first. Unrolling the recurrence in this case will be done logb n times before getting to ƒ (1). As
of the logb n terms resulting from the unrolling is on the order of so, one
PARALLEL ALGORITHM COMPLEXITY 61
Theorem 3.1 only provides an asymptotic solution in cases where the theorem’s conditions are met. However, as noted earlier, once the order of the solution is known, substitution in the recurrence can be used to determine the constant factors and lower-order terms if desired.
PROBLEMS
3.1. Asymptotic complexity
For each of the following pairs of functions ƒ (n ) and g(n), determine which of the relationships if any, holds.
Explain your reasoning.
3.2. Asymptotic complexity
Order the following functions with respect to their growth rates, from the slowest growing to the fastest growing. Explain your reasoning.
3.3.
3.4.
3.5.
Computational complexity
Assume that the pairs of functions ƒ (n) and g(n) of Problem 3.1 correspond to the running times of two different algorithms A and B, respectively, for solving a given problem when the input size is n (do not worry about the fact that some of the instances do not represent real problems).
Determine, for each pair of functions, the problem sizes for which Algorithm A is better than Algorithm B.
Computational complexity
With reference to the data in Table 3.2, suppose that a running time of 3 hours or less is tolerable in a given application context. Assuming that Moore’s law holds (see Section 1.1), for each of the five functions shown in Table 3.2, determine the factor by which the problem size can increase in 10 years.
Comparing algorithms
Suppose two different parallel algorithms for solving a given problem lead to the following recurrences for the computation time. Which algorithm is asymptotically faster and why?
62 INTRODUCTION TO PARALLEL PROCESSING 3.6. Solving recurrences
a . Derive an exact solution to the recurrence T( n) = T( n /2) + cn assuming that n is a power of 2, c is a constant, and T (1) = 0.
b . More generally, the problem size (initially or in intermediate steps) is not necessarily even.
Consider the recurrence and use it to derive an upper bound for T( n) . Hint: The worst case occurs when n is 1 more than a power of 2.
c . Repeat part (b) for the more general recurrence 3.7. System of two recurrences
a . Find exact solutions for A( n) and B(n) if A(n) = B(n/2) + gn and B(n) = A(n/2) + hn, assuming n to be a power of 2, g and h known constants, and A (1) = B (1) = 0.
b . Repeat part (a) with A(n) = 2B(n/2 ) + gn and B (n) = A (n/2) + hn.
c . Repeat part (a) with A(n) = 2B(n/2) + gn and B (n) = 2A(n/2) + hn.
d . Formulate a general theorem for the asymptotic solution of system of two recurrences of the form A(n) = cB (n/a) + gn and B (n) = dA(n/b) + hn, where a, b , c, and d are constants.
3.8. Solving recurrences
Apply the method of Theorem 3.1 to the solution of Examples 1 through 6 that precede it, where applicable.
3.9. Solving recurrences
Apply the method of Theorem 3.1 to the solution of the recurrence ƒ ( n) = 2 ƒ (n /4) + cn, where c is a known constant. Then, find the constant factor and the order of residual terms through substitution.
3.10. Basic theorem for recurrences
In the proof of the first case of Theorem 3.1, we implicitly assumed that ƒ(1) is a nonzero constant. Would the proof fall apart if ƒ (1) = 0?
3.11. Solving recurrences
Solve the recurrence where b and c are constants and 0 < b < 1.
3.12. Asymptotic complexity
A rough set S is characterized by a lower bound set Slb consisting of elements that are certain to be members of S and an upper bound set Sub which also contains a boundary region consisting of possible elements of S (the possibility is not quantified) [Paw197]. Define the basic set operations on rough sets and determine if they are asymptotically more complex than ordinary set operations in sequential and parallel implementations.
3.13. Computational complexity
Consider the following results from number theory [Stew97]: (a) x is not prime iff some number in divides it. (b) x is prime iff it divides ( x – 1) ! + 1. (c) If x is prime, then it divides 2x – 2 (the converse is not true). A large number x, with hundreds of digits, is given and we would like to prove that it is not a prime. We do not need the factors; just a confirmation that it is not a prime. Which of the above three methods would lead to a more efficient parallel implementation and why? Note that in computing a large number whose divisibility by x is to be tested, we can do all of the calculations modulo x.
3.14. NP-completeness
In our discussion of NP-completeness, we stated that the satisfiability problem for OR–AND
PARALLEL ALGORITHM COMPLEXITY 63 Boolean expressions is NP-complete and that even the special case where each of the ORed terms consists of exactly 3 literals (known as the 3-satisfiability problem) is not any easier.
a. Show that the AND–OR version of the satisfiability problem, i.e., when the Boolean expression in question consists of the OR of AND terms, is in P.
b. Show that 2-satisfiability is in P. Hint: Use the equivalence of x OR y with (NOT x) IMPLIES y to reduce the 2-satisfiability problem to a problem on a directed graph.
3.15. NP-completeness
Consider a set S of integers. The set partition problem is that of determining if the set S can be partitioned into two disjoint subsets such that the sum of the integers in each subset is the same.
Show that the set partition problem is NP-complete.
REFERENCES AND SUGGESTED READING
Cook, S., “The Complexity of Theorem Proving Procedures,” Proc. 3rd ACM Symp. Theory of Computing, 1971, pp. 151–158.
Cormen, T. H., C. E. Leiserson, and R. L. Rivest, Introduction to Algorithms, McGraw-Hill, 1990, Chapter 36, pp. 916–963.
Gupta, R., S. A. Smolka, and S. Bhaskar, “On Randomization in Sequential and Distributed Algorithms,” ACM Computing Surveys, Vol. 26, pp. 7–86, March 1994.
Krishnamurthy, E. V., “Complexity Issues in Parallel and Distributed Computing,” Chapter 4 in Parallel and Distributed Computing Handbook, edited by A. Y. Zomaya, McGraw-Hill, 1996, pp.
89–126.
Parberry, I., Parallel Complexity Theory, Pitman, 1987.
Pawlak, Z., J. Grzymala-Busse, R. Slowinski, and W. Ziarko, “Rough Sets,” Communications of the ACM, Vol. 38, No. 11, pp. 89–95, November 1995.
Pippenger, N., “On Simultaneous Resource Bounds,” Proc. 20th Symp. Foundations of Computer Science, 1979, pp. 307–311.
Stewart, I., “Mathematical Recreations: Big Game Hunting in Primeland,” Scientific American, Vol.
276, No. 5, pp. 108–111, May 1997,
This page intentionally left blank.