Part 1 The Algorithm Side: Regularity,
3 Examples of Complexity Analysis
3.1 General Techniques for Determining Complexity
Examples of Complexity Analysis
About This Chapter
We begin with a review of techniques for determining complexity functions.
Then we apply these techniques to a number of standard algorithms, among others representatives of the techniques of divide-and-conquer and dynamic programming, as well as algorithms for sorting, searching, and graph oper-ations. We also illustrate on-line and off-line algorithms.
This chapter concentrates on techniques for determining complexity mea-sures and how to apply them to a number of standard algorithms. Readers who have substantial knowledge of algorithm complexity may skip this chapter without major consequences. We first review approaches to finding the operation or statement count of a given algorithm. These range from simple inspection of the statements to much more sophisticated recursion-based arguments. Then we examine a number of standard algorithms that should be known to all computer scientists and determine their complexity measures, mostly time complexity and usually worst-case.
3.1 General Techniques for Determining Complexity
Suppose we are given an algorithm and want to determine its complexity.
How should we do this? If the algorithm were given as a linear sequence of simple statements (so-called straight-line code where every statement is exe-cuted once), the answer would be trivial: Count the number of statements — this is its time complexity. Of course, such an algorithm would be utterly trivial. Virtually all algorithms of any interest contain more complex state-ments; in particular, there are statements that define iteration (for loops, while loops, repeat loops), statements that connote alternatives (if statements, case statements), and function calls, including those involving recursion.
C6730_C003.fm Page 49 Friday, August 11, 2006 8:12 AM
50 A Programmer’s Companion to Algorithm Analysis Iteration: The most important aspect is to determine the number of
times the body of the iterative statement is executed. In the case of for loops, we may be able to do this exactly. In other cases, we may only be able to determine an upper bound on, or some other type of estimate of, the number of iterations that will be performed. The quality of this upper bound or estimate will affect the quality of our analysis; a tight upper bound will usually give us a better complexity measure than a loose one. Once we have determined or estimated the number iterations, the statement count of the loop is then that number of iterations times the statement count of the body of the iteration statement. It is usually necessary to factor into this process the type of complexity we are interested in. If we want worst-case, then we must determine an upper bound; for average complexity, a good estimate may be preferable.
Alternatives: Again, this depends on the desired type of complexity.
For worst-case complexity, we must determine the complexities of all the alternatives and take the maximum of them; for average complexity, we use the average of these complexities. (Recall our discussion of average in Chapter 1; it is important to be aware of what exactly average means within the context of a given algorithm.1)
Function calls: If no recursion is present, we simply determine the complexity of executing the function in dependence of its arguments, that is, using (one or more of) the function’s arguments as para-meters for our measures. Then we must integrate this into the overall complexity analysis of the algorithm. If recursion is involved in the function call, more powerful techniques are needed.
Recursion: A function F is called recursive if its body contains a call to itself. An important aspect of recursion is that it has to terminate;
this implies that the argument2 must change for this to be achieved.
Intuitively, that argument must evolve toward one of possibly sev-eral basis cases, and for each of these basis cases, the recursion stops, that is, no more calls to F occur. In most practical situations, the argument of F decreases until it hits a threshold when F returns without further recursive calls. The way in which this decrease hap-pens is crucial for our task of determining the complexity of the call to the recursive function.
1 Thus, for the statement if cond then s1else s2, the probability of cond to be true determines how likely s1 is executed in preference over s2. One should not automatically assume that cond is true 50% of the time (merely because true is one of two possible values cond may take on; but see the discussion in Section 6.6 about initialization).
2 We assume here for the sake of simplicity that our function F has just one argument or para-meter; if more arguments are present, an analogous statement must hold. Typically, this single argument is a function of the size of the input set.
C6730_C003.fm Page 50 Friday, August 11, 2006 8:12 AM
Examples of Complexity Analysis 51 Let us make this a bit more concrete. Let F(n) be the function, with the argument n being the size of the input set. Thus, n is an integer and n≥ 0. Let us assume that for all values of n, at most n0, no recursion, occurs; in other words, for all n≤n0, we have basis cases. Here, n0 is some fixed nonnegative integer. Conversely, for all n > n0, one or more recursive calls to F occur in F’s body. Let us assume that there are exactly r recursive calls, for r≥ 1, a fixed value. It should be clear that in this formulation, termination of the call F(n) is assured precisely if the argument of each of these r recursive calls to F is strictly less than n.3n. In determining the complexity of F with argument n, it is crucial to know the decrease of the argument in each of these recursive calls. Let us write down schematically what we need to know of F:
F(n)
If n ≤ n0 then do basis case Else
{…; F(n1) (first recursive call to F, with argument n1 < n)
…; F(n2) (second recursive call to F, with argument n2 < n)
…; F(nr) (rth recursive call to F, with argument nr < n)
… }
We must now distinguish several cases according to the number r of calls and according to the nature of the decrease of the argument n.
Case 1
Each of the nj is equal to n−c, where c is a fixed integer constant, for all j
= 1, …, r: Then for r > 1, the (worst-case time) complexity of computing F(n) is at least rO(n); that is, it is at least the number r raised to a linear function of n.4 If r = 1, then the complexity of computing F(n) is at least O(n).5 In each of these situations, what complicates things is the elided stuff (the “…” in the schematic above). If these statements together have a complexity that is
3 We use here that n is an integer; otherwise this statement about termination is no longer true, as the following counter example indicates: Consider the function
F(x): if x > 0, then do basis case else F(x/2).
When called with an argument x > 0, the function call F(x) does not terminate as an algorithm, since the argument x will never be equal to 0. Successive division by 2 of a positive value will always result in another positive value. However, it would terminate when implemented as a program, since eventually the argument will be so small that it will be set equal to 0 (by rounding error), but the number of recursion calls is unpredictable and clearly dependent on the word length of the parameter x.
4 Let T(n) be the time required to execute the algorithm, assuming the elided statements take O(1) time. Then we can write: T(n) = r·T(n −c) + O(1) for all n > n0, and T(n) = C0 otherwise, for C0 some constant. From this, one shows by induction on n that T(n) = r(n-n0)/c + O(1), which proves the claim.
5 Let T(n) be the time required to execute the algorithm, assuming the elided statements take O(1) time. Then we can write T(n) = T(n−c) + O(1) for all n > n0, and T(n) = C0 otherwise, for C0 some constant. From this follows the claim.
C6730_C003.fm Page 51 Friday, August 11, 2006 8:12 AM
52 A Programmer’s Companion to Algorithm Analysis O(1), the complexities are exactly rO(n) (for r > 1) and exactly O(n) (for r = 1).
If the complexity of the elided statements is greater than O(1), then the resulting complexity may increase accordingly.
Typical examples of these situations are the Towers of Hanoï,6 where r = 2, and computing the factorial function7 n! recursively (r = 1).
A similar result is obtained if the constant c depends on j, that is if nj = n − cj, where cj is a constant, for all j = 1, ..., r. Specifically, the (worst-case time) complexity remains at least rO(n) if r > 1 and at least O(n) if r = 1. A typical example is provided by the Fibonacci numbers.8
Case 2
A more interesting case happens when for all j = 1, …, r, the nj are not reduced by some constant subtrahend, but each nj is a fraction of n. More specifically, assume that the elided statements have a time complexity of b·n, where b is a positive constant. Also assume that for all j = 1, …, r, nj = n/c, where c is another positive constant, with c > 1. Finally, we assume that the basis case (i.e., when there is no recursive call) is of complexity b (the same b as before)9 and that n0 = 1.10 In this case, we can formulate the time complexity of F as follows. Let T(n) be the time complexity of F(n). Then we can write:
T(1) = b
6 The Towers of Hanoï is a famous game. We are given n disks of different sizes and three pegs.
The n disks reside initially on the Start peg. The task consists of moving the n disks to the Des-tination peg, using the Auxiliary peg. Movements are constrained as follows: (1) We may move only one disk at a time. This disk must be the top disk on a peg; it may be moved to any other peg, subject to: (2) No larger disk may ever be placed on top of a smaller disk. Here is an algo-rithm that solves the problem of moving the n disks from St to De:
Basis case: If n=1, move disk 1 from St to De else
Recursion: { move recursively the smallest n-1 disks from St to Au;
move the largest disk n from St to De;
move recursively the n-1 smallest disks from Au to De }
The complexity of the Towers of Hanoï problem is O(2n) since precisely 2n − 1 moves must be car-ried out to get the n disks from St to De.
7 Here is the algorithm for nonnegative n: If n=0 then n!=1 else n!=n·(n-1)! .
8 The Fibonacci fn numbers are defined as follows: f1 = 1, f2 = 2, fn = fn−1 + fn−2 for n > 2. Here r = 2, c1 = 1, and c2 = 2. While it is not properly part of this discussion, we cannot resist pointing out that just because we can do something recursively does not mean it is always a good idea. In fact, it is a truly terrible idea to compute the Fibonacci numbers recursively. (A similar statement holds for the recursive computation of the factorial function.) The reason should be obvious: As the above theory indicates, the complexity of doing this is exponential in n. It is a no-brainer that the Fibonacci numbers can be computed in linear time in n. (As a first cut, use an array with an element for each of the n numbers. Initialize the first two elements with f1 and f2 and then com-pute each of the subsequent elements by adding up the two previous ones. This then can be modified to use only three simple variables instead of the array.) There is a fairly convoluted for-mula that permits the computation of fn directly, resulting in a time complexity of O(log2(n)).
9 Really, we require that the time complexity of the basis cases be some constant b’. If b’ is differ-ent from b, then we replace b’ as well as b by max(b,b’) and everything goes through nicely.
C6730_C003.fm Page 52 Friday, August 11, 2006 8:12 AM
Examples of Complexity Analysis 53 T(n) = r·T(n/c) + b·n,
reflecting that the problem T(n) is split into r problems of size n/c.11 Then we can state for all values of n that are a power of c:
T(n) = O(n) if r < c T(n) = O(n·log2(n)) if r = c T(n) = O(nlogc(r)) if r > c.
While the statement is technically valid only for powers of c, one usually assumes that for all other values of n, corresponding results are obtained by interpolation. Thus, assuming some degree of smoothness (which is ordi-narily quite appropriate), one tends to omit the qualifier “n a power of c”
and acts as if this statement is true for all values of n.12
3.2 Selected Examples: Determining the Complexity of