Approach Creative

(1)

(2)

INTRODUCTION

TO ALGORITHMS

A Creative Approach

(3)

(4)

INTRODUCTION TO ALGORITHMS

A Creative

Approach

UDIMANBER

University of Arizona

\342\226\274\342\226\274

ADDISON-WESLEY PUBLISHING COMPANY

Reading,

Massachusetts

\342\200\242Menlo Park, California \342\200\242New York Don Mills,

Ontario

\342\200\242

Wokingham, England \342\200\242Amsterdam Bonn \342\200\242

Sydney \342\200\242

Singapore \342\200\242

Tokyo \342\200\242Madrid \342\200\242San Juan

(5)

Library of

Congress Cataloging-in-PublicationData

Manber, Udi.

Introduction to algorithms.

Includes bibliographies

^and ^index.

1. Data structures

(Computer science)

2.

Algorithms. I. Title.

QA76.9.D35M36 1989

005.7 ' 3 88-2186

ISBN 0-201-12037-2

Reproduced by Addison-Wesley from

camera-ready copysupplied

^by ^the ^author.

The programs and

applications presented

ⁱⁿ ^this ^book ^have been included for their

instructional value. They have

been tested

^with care, but are not guaranteed for any

purpose. The publisher does not offer any warranties or

representation,

^nor ^does ^it accept

any liabilities with respect tothe programs or applications.

Reprinted ^with corrections October, 1989

Copyright ^\302\251¹⁹⁸⁹ by Addison-Wesley Publishing

Company Inc.

of

^this publication may be reproduced,

stored

ⁱⁿ ^a retrieval

system, or transmitted, in any form or _by _any means, electronic, mechanical, photocopying,

recording, or

otherwise, without prior written permission

of

^the publisher.

Printed in the United States of America. Published simultaneously in Canada.

EFGHIJ-DO-943210

(6)

To

my parents Eva and Meshulam

(7)

(8)

PREFACE

This book grew out of my frustrations with not being able to explain algorithms

clearly.

Like

many other teachers, I discovered that not _only is it hard for

some

^students ^to ^solve

(what seemed to

me) simple problems

^by themselves, but it is

also

^hard ^for ^them to understand the solutions that are given to

them. I believe

^that ^these ^two _parts \342\200\224the creation and the explanation \342\200\224are related and should not be separated. It is

essential to follow

^the steps leading to asolution in order to understand it fully. It is not sufficient to

look at the finished product.

This book emphasizes ^the ^creative ^side ^of algorithm

design. Its

^main ^purpose ^is ^to

show the reader how to design

a

^new algorithm. Algorithms are not

described

ⁱⁿ ^a

sequence of '

'problem

^X, âlgorithm Â, algorithm Â\\ ^program ^P, program ^P\\\" ând ^so on.

Instead, ^the sequence usually (although not

always) looks

^more like \"problem X, the

straightforward algorithm, its drawbacks, the difficulties overcoming these drawbacks,

first attempts ^at ^a ^better algorithm (including

possible

^wrong ^turns), improvements, analysis, relation to other

methods and

algorithms,\" and so on. The

goal is to present

^an

algorithm not in a _way that makes it easier for

a

programmer to translate into a

program,

but rather in a _way that makes it easier to understand the algorithm's principles. The

algorithms are thus explained through

a creative process,

^rather ^than ^as ^finished products.

Our goals

ⁱⁿ ^teaching âlgorithms âre ^{to show} ^not ônly ^how ^to ^solve ^particular ^problems,

but also how to

solve

^new problems when they arise in the future. Teaching the _thinking

involved in designing an algorithm is

as

important as teaching the details

of

^the ^solution.

To further help ^the thinking process involved in creating algorithms, ^an

\"old-new\" methodology for designing algorithms is used in this

book. This

methodology covers many known techniques for designing algorithms, and ^it ^also

provides an elegant ^intuitive ^framework ^for explaining the

design of

_algorithms ⁱⁿ ^more

depth. It does not, however, cover all possible ways of designing algorithms, and we

do

not use it exclusively. The heart of the methodology lies in an analogy between the

intellectual process of proving mathematical

theorems

by induction and that of designing

combinatorial

algorithms. Although these two processes

serve

different purposes and achieve different types

of results,

they are more similar than they may appear to

be. This

analogy has been observed by many people.

The

novelty of this book is the degree to

which this _analogy is exploited. We show that the analogy encompasses many known

algorithm-design techniques,

^and ^helps considerably in the

process of

_algorithm creation.

The methodology is

discussed

briefly in Chapter 1 and is introduced more formally in Chapter

5.

(9)

vi Preface

Consider the following analogy.

Suppose

^that ^you ^arrive ^{at an} ^unfamiliar ^city, ^rent

a car, and ^want ^directions ^to get to_your hotel. You would be _quite _impatient if you were

told about the history of the city, its general layout, ^the ^traffic ^patterns, ^and so

on.

^You

would rather have directions

of

^the ^form \"go straight for two blocks, turn right, go

straight for three miles,\" and

so on.

However, your outlook would change if _you

planned to live ⁱⁿ ^that city for a _long time. You could probably get ^around ^for ^a ^while

with directions of the second form

(if

you find someone who gives you those directions), but eventually you will need to know

more

^about ^the city. This book isnot a source of easy

directions.

^It ^does ^contain explanations of how to solve many particular

problems,

but the emphasis is on

general principles

ând ^methods. Âs â ^result, ^the ^book îs

challenging. It demands

involvement

^and thinking. I believe that the extra effort is well worthwhile.

The

design of

^efficient ^nonnumeric algorithms is becoming important in many diverse fields, including mathematics, statistics, molecular biology, ^and engineering.

This book can

serve as

^an introduction to algorithms and to nonnumeric computations in general. Many

professionals, and even

^scientists ^not deeply involved with computers, believe that programming is _nothing more than grungy nonintellectual work. It sometimes is. But such

a belief

may lead to straightforward, trivial, inefficient solutions, where elegant, more efficient solutions

exist. One

goal of this book is to

convince readers

^that algorithm design is an elegant

discipline, as

^well ^as ^an _important one.

The bookis

self-contained. The presentation is mostly ^intuitive, ^and technicalities are either kept to

a

^minimum ^or ^are _separated from the main discussion. In particular, implementation

details are

separated from the algorithm-design ideas

as

^much ^as

possible. There are _many _examples of algorithms that were

designed especially

^to

illustrate the principles emphasized ⁱⁿ ^the ^book. ^The material in this book is not

presented as

something to be mastered and

memorized.

It is presented as a

series of ideas, examples,

counterexamples, modifications, improvements, and so

on.

Pseudocodesfor most algorithms are given following the descriptions. Numerous exercises and

a discussion

^of further reading, with a relevant bibliography, follow each chapter. In most

chapters,

^the êxercises âre ^divided înto ^two ^classes, ^drill êxercises ând

creative exercises.

^Drill ^exercises are meant to test the reader's understanding of the

specific examples

^and algorithms presented in that chapter.

Creative exercises

^are ^meant ^to test the

reader's

_ability to use the techniques

developed

ⁱⁿ ^that ^chapter, ⁱⁿ addition to the

particular algorithms, to solve new problems. Sketches of solutions to

selected exercises

(those whose numbers are underlined)

are

given at the end of the book. The chapters

also include a summary

of

^the ^main ^ideas introduced.

The

book is organized as

^follows. Chapters ¹ through 4 present introductory

material. Chapter 2 is

^an introduction to mathematical induction.

Mathematical

induction is, as we will see, very important to _algorithm _design. _Experience with

induction

proofs is therefore

very helpful. Unfortunately, few computer-science ^students

get enough exposure to induction proofs. Chapter 2 may be _quite difficult for some students.

We suggest

^skipping ^the ^more ^difficult examples ^at ^first reading, and returning to them later.

Chapter 3 is

^an introduction to the analysis

of

algorithms. It describes the process

of

analyzing algorithms, and gives the

basic tools

one needs to be able to

perform

(10)

Preface

^vii

simple analysis of the algorithms

presented

ⁱⁿ ^the ^book. Chapter 4 is

a brief

introduction to data structures. Readers ^who ^are familiar with basic data structures and who have a

basic

mathematical background can start directly from

Chapter 5

(it is always agood

idea

to read

^the introduction though). Chapter 5

presents

^the ^basic ^ideas ^behind ^the

approach

of

designing algorithms through the analogy to induction proofs. It gives several

examples of

simple algorithms, and describes their

creation. If

you read only one chapter

in this book, read Chapter

5. There

^are ^two ^basic _ways to

organize a book

^on algorithms. One way is to

divide

the book according to the

subject of

^the algorithms, for example, graph algorithms,

geometric

algorithms. Another way isto

divide

^the ^book according to design

techniques.

Even though the emphasis of this book is on design

techniques, I have chosen

^the ^former

organization. Chapters 6 _through 9 present algorithms ⁱⁿ ^four ^areas: algorithms for

sequences

^and ^sets ^(e.g., ^sorting, ^sequence

comparisons, data compression),

^graph

algorithms (e.g., spanning trees, shortest paths, matching), geometric algorithms (e.g.,

convex

hull, intersection problems), and numerical and algebraic algorithms (e.g., matrix

multiplication, fast Fourier transform). I believe that this organization is clearer and

easier to follow.

Chapter 10 is devoted to

reductions.

Although examples of reductions appear ⁱⁿ

earlier chapters, the subject is unique and important enough to warrant a chapter of its

own.

This chapter also serves as

ân ôpening âct ^to Chapter

11,

^which ^deals ^with ^the subject

of NP-completeness.

^This ^aspect ^of complexity theory has

become

an essential part of algorithm theory. Anyone who designs algorithms should know about NP- completeness and the techniques for proving this property.

Chapter 12is

an introduction to parallel algorithms. ^It ^contains ^several interesting algorithms ^under ^different ^models of parallel

computation.

The

material in this book is

more

^than ^can ^be covered in a one-semester course,

which

leaves

many choices for the instructor. A first course in algorithm

design

^should

include parts of Chapters

3, 5, 6, 7,

^and ⁸ ⁱⁿ ^some depth, although not necessarily all of

them. The

^more ^advanced parts of these

chapters, along

^with ^Chapters ^9, ^10, ^11,^and ^12,

are optional for

a

^first course, and can be

used as a basis

^for a more advanced course.

Acknowledgments

First and foremost I thank my wife Rachel for _helping ^me ⁱⁿ more ways ^than ^I ^can ^list

here _throughout this adventure. She was instrumental in the development of the

methodology on which the

book is based. She

contributed suggestions, corrections, and

\342\200\224more important than anything else \342\200\224sound advice. I could not

have done

it without her.

Special thanks

are due to

Jan van Leeuwen for an excellent and thorough review of a large portion of this book. His

detailed comments,

^numerous suggestions, and many corrections

have

improved the book enormously. Ialso thank Eric Bach, Darrah Chavey,

Kirk Pruhs, and Sun Wu, ^who ^read parts of the _manuscript and made many helpful

comments,

^and ^the ^reviewers Guy T.

Almes (Rice

University), Agnes H. Chan (Northeastern University), Dan Gusfield (University of

California, Davis),

^David ^Harel

(Weizmann Institute, Israel),

Daniel Hirschberg

(University of California, Irvine),

(11)

viii Preface

Jefferey H. Kingston (University of Iowa), Victor Klee (University of Washington), Charles Martel (University of California, Davis), Michael

J.

Quinn (University of New Hampshire), and

Diane M. Spresser

^(James ^Madison University).

I thank the

people

^at Addison-Wesley who failed to_supply pie with any examples

of

^horror ^stories ^that ^authors ^are

so fond of

telling. They were very helpful and

incredibly patient and understanding. In particular, I thank my production supervisor

Bette Aaronson, my

editor

^Jim DeWolf, and my copy

editor

Lyn Dupr6, who not only guided

me

^but ^also ^let ^me ^do things my way even when _they knew better. I also thank

the National Science Foundation for financial support, through a Presidential _Young

Investigator Award, and AT&T, Digital Equipment Corporation, Hewlett Packard, ^and

Tektronix, for matching funds.

The book

^was designed and typeset by me. It was formatted in troff, and printed

on a Linotronic 300 ^at ^the Department of Computer

Science,

University of Arizona. I thank Ralph

Griswold for his advice,

^and ^John ^Luiten, ^Allen Peckham, and _Andrey

Yeatts for technical help ^with ^the typesetting. The figures

were prepared

^with ^gremlin \342\200\224 developed at the University

of California,Berkeley

\342\200\224

except for Fig. 12.22, ^which ^was

designed and drawn _by _Gregg Townsend. The index was

compiled

^with ^the help of a

system

^by ^Bentley ^and Kernighan [1988]. I^thank ^Brian Kernighan for supplying

me

the code within minutes after I

(indirectly) requestedit.

^The ^cover ^was ^done by Marshall

Henrichs, based

ôn ân îdea _by the author.

I must stress, however, ^that ^the ^final manuscript was

prepared

^by ^the typesetter.

He was the one who decided to

overlook

many comments and suggestions of the people listed here. And he isthe one who should bear the consequences.

Tucson, Arizona Udi

Manber

(Internet address: [email protected].)

(12)

Chapter 1 Chapter 2

Chapter 3

Introduction l

Mathematical

Induction 9

2.1 2.2 2.3 2.4 2.5 2.6

2.7 2.8 2.9 2.10 2.11

2.12 2.13 2.14

Introduction

Three Simple Examples

Counting

Regions

ⁱⁿ ^the ^Plane

ASimple

Coloring Problem

A More Complicated Summation Problem

A Simple Inequality Euler's Formula

A Problem in Graph Theory

Gray

Codes

Finding Edge-Disjoint Paths in a Graph

Arithmetic versus Geometric Mean Theorem

Loop

Invariants: Converting aDecimal Number to_Binary

Common Errors

Summary

Bibliographic

Notes and

Algorithms 37

3.1 3.2 3.3 3.4 3.5

3.6

3.7

Introduction The 0 Notation

Time and Space

Complexity Summations

Recurrence Relations 3.5.1 Intelligent Guesses

3.5.2 Divide and

Conquer Relations 3.5.3

^Recurrence ^Relations ^with Full History Useful

Facts

Summary

Bibliographic Notes and Further Reading

Exercises

9 11 13 14 15 16 17 18 20 23

24 26 28 29 30

31 37 39 42 43 46 47 50 51 53 55 55 56

IX

(13)

x Contents

Chapter 4

Data Structures 61

4.1 4.2

Introduction

Elementary Data Structures 4.2.1

Elements

Arrays Records Linked Lists 4.3

4.2.2 4.2.3 4.2.4

Trees 4.3.1 4.3.2 4.3.3 4.3.4

4.4 4.5 4.6 4.7

Representation of Trees

Heaps

Binary

Search Trees

AVL Trees

Hashing

The Union-Find

Problem Graphs

Summary

Exercises

Chapter 5 Designof

^Algorithms ^by ^Induction ⁹¹

5.1 Introduction

5.2 Evaluating Polynomials 5.3

Maximal Induced

Subgraph

5.4 Finding One-to-One Mappings

5.5 The

Celebrity Problem

5.6 A Divide-and-Conquer Algorithm: The Skyline Problem 5.7

Computing Balance

^Factors ⁱⁿ Binary Trees

5.8 _Finding the Maximum Consecutive Subsequence

5.9

Strengthening the Induction Hypothesis

5.10

Dynamic

Programming: The Knapsack Problem 5.11

Common Errors

5.12

_Summary

Exercises

Chapter 6

_Algorithms _Involving _Sequences and Sets

119 6.1

Introduction

6.2 Binary Search and Variations 6.3 Interpolation Search

6.4 _Sorting

6.4.1 Bucket Sort and Radix Sort

6.4.2 Insertion Sort and Selection Sort 6.4.3 Mergesort

61 62 62 63 63 64 66 67 68 71

75 78 80 83 84

85 86

91 92 95 96 98 102 104 106

107 108 111 112 113

114

119

120

125

127

127 130 130

(14)

Contents xi

6.4.4 Quicksort 131

6.4.5 Heapsort 137

6.4.6

^A ^Lower ^Bound for Sorting

141

6.5

^Order ^Statistics 143

6.5.1 Maximum and Minimum Elements 143

6.5.2 _Finding the fcth-Smallest Element 144

6.6 Data

Compression 145

6.7 String Matching

148 6.8 Sequence

Comparisons 155

6.9 Probabilistic Algorithms

158

6.9.1

Random Numbers 160

6.9.2 A

Coloring Problem 161

6.9.3

^A Technique for Transforming Probabilistic

Algorithms into Deterministic Algorithms 161

6.10

Finding a Majority 164

6.11

Three Problems

Exhibiting Interesting Proof Techniques 167

6.11.1 Longest

^Increasing Subsequence 167

6.11.2 Finding ^the ^Two Largest Elements in a Set 169

6.11.3

Computing

^the ^Mode ^of ^a^Multiset

171

6.12

Summary 173

Bibliographic Notes and Further Reading 173

Exercises 175

Chapter 7 Graph

^Algorithms ¹⁸⁵

7.1 Introduction 185

7.2

^Eulerian Graphs 187

7.3 Graph

Traversal 189

7.3.1

Depth-First Search 190

7.3.2 Breadth-First

Search 198

7.4

Topological Sorting 199

7.5 Single-Source

Shortest

^Paths ²⁰¹

7.6 Minimum-Cost Spanning

Trees 208

7.7

^All Shortest Paths 212

7.8 Transitive

Closure 214

7.9

Decompositions of Graphs 217

7.9.1

Biconnected Components 217

7.9.2

Strongly Connected Components 226 7.9.3

Examples of

^the ^Use ^of _Graph Decomposition

230

7.10

Matching 234

7.10.1 Perfect Matching ⁱⁿ Very Dense Graphs .234

7.10.2

Bipartite Matching 235

7.11 Network

Flows 238

7.12

Hamiltonian Tours 243

7.12.1 Reversed Induction 244

(15)

xii Contents

7.12.2 _Finding Hamiltonian Cycles in Very

Dense Graphs

7.13

Summary

Exercises

Chapter 8 Geometric

Algorithms 265

8.1 8.2 8.3

8.4 8.5 8.6 8.7

Introduction

Determining Whether a Point

Is Insidea

Polygon

Constructing Simple Polygons

Convex Hulls

8.4.1 A Straightforward Approach

8.4.2

^Gift Wrapping 8.4.3 Graham's Scan

Closest Pair

Intersections

^of ^Horizontal ^and ^Vertical ^Line

_Segments

Summary

Exercises

244 246 247 248

265 266 270

273 273 274 275 278

281 285 286 287 Chapter 9

Algebraic and Numeric Algorithms 293

9.1 9.2 9.3 9.4 9.5

9.6

9.7

Introduction Exponentiation Euclid's Algorithm Polynomial Multiplication Matrix Multiplication 9.5.1 Winograd's Algorithm 9.5.2 Strassen's Algorithm 9.5.3

Boolean Matrices

The Fast Fourier Transform Summary

Bibliographic Notes

^and ^Further Reading Exercises

Chapter

10 Reductions 321

10.1

Introduction

10.2 Examples of Reductions

10.2.1

^A _Simple String-Matching Problem 10.2.2

Systems of

Distinct Representatives

10.2.3 AReduction _Involving _Sequence Comparisons 10.2.4 Finding a

Triangle

ⁱⁿ ^Undirected Graphs 10.3 Reductions _Involving Linear Programming

10.3.1 Introduction and Definitions

10.3.2 Examples of Reductions to

Linear

Programming

293 294 297 298

301 301 301

304 309 316 316 317

321 323

323 323 324 325 327

327

329

(16)

Bounds 331

10.4.1

^A ^Lower ^Bound for Finding

Simple Polygons 331 10.4.2

Simple Reductions Involving Matrices 333

10.5 Common Errors 334

10.6

Summary 336

Exercises 337

Chapter 11 NP-Completeness 341

11.1

Introduction 341

11.2 Polynomial-Time Reductions

342

11.3

Nondeterminism and Cook's Theorem 344

11.4 Examples

^of NP-Completeness Proofs 347

11.4.1 Vertex

Cover 348

11.4.2

Dominating Set 348

11.4.3 3SAT

350 11.4.4 Clique 351

11.4.5

3-Coloring 352

11.4.6 General Observations

355

11.4.7 Techniques For Dealing

^with NP-Complete Problems 357

11.5.1

Backtracking and Branch-and-Bound 358

11.5.2

Approximation Algorithms with Guaranteed

Performance

363

11.6

Summary 368

Exercises 370

Chapter 12 Parallel

Algorithms 375

12.1 12.2 12.3

12.4

Introduction

Models of Parallel Computation

Algorithms for Shared-Memory Machines

12.3.1

Parallel

^Addition

12.3.2 Maximum-Finding Algorithms

12.3.3

The Parallel-Prefix

^Problem

12.3.4 Finding Ranks ⁱⁿ Linked Lists

12.3.5 The

Euler's Tour

Technique

Algorithms for Interconnection Networks

12.4.1

Sorting on an Array 12.4.2 Sorting Networks

12.4.3 Finding the

fcth-Smallest

Êlement ôn âTree 12.4.4 Matrix Multiplication on the Mesh

12.4.5

Routing in a Hypercube

375

376 378 379 380 382

385 387 389 390 393

396

398

401

(17)

xiv Contents

12.5

12.6

References

Index

Systolic

Computation

12.5.1 Matrix-Vector Multiplication

12.5.2

The

Convolution Problem 12.5.3 Sequence Comparisons Summary

Bibliographic Notes

^and ^Further _Reading

Exercises

tions

to Selected Exercises 417

445

465 404 404 405

407

409

411

(18)

CHAPTER 1 INTRODUCTION

Great importance has been _rightly attached to thisprocess

of

\"construction,\" and some claim to

see

ⁱⁿ ^it ^the

necessary and _sufficient condition of the progress

of

^the

exact sciences. Necessary, no doubt, but not sufficient!

For

a

construction to be useful and not mere waste of

mental effort,for ît ^to ^serve âs astepping-stone to higher things, ît ^must first of allpossess

a

^kind of unity enabling us

to see something more than the juxtaposition of its elements.

Henri Poincare, 1902

The Webster's Ninth New Collegiate dictionary defines an algorithm as \"aprocedure for solving

a

mathematical problem (as of finding the

greatest common divisor)

ⁱⁿ ^a ^finite

number of

steps

^that ^frequently ^involves ^a repetition

of

^an _operation; ^or _broadly, a

step- by-step

^procedure ^for ^solving ^a problem

or accomplishingsome end.\"

^We ^will ^stick to the

broad

definition. The design of algorithms is^thus ^an ^old ^field of _study. _People have always been

interested

in finding better methods to

achieve

^their goals, whether those be

starting fires, building pyramids, or_sorting the mail. The study

of computer

^algorithms ^is

of course new.

Some computer

^algorithms ^use ^methods developed before the invention of computers, but

most problems

^require ^new approaches. For one _thing, it is not enough to tell

a computer to \"look over

^the ^hill ^and sound the alarm ifan army is advancing.\"

A

computer

^must ^know ^the ^exact meaning

of

\"look,\" how to identify an army, ^and ^how

to sound the alarm

(for some

reason, sounding an alarm is _always _easy). A computer receives its instructions via well-defined, limited _primitive operations. It is a difficult process to translate regular instructions to a language ^that ^a ^computer understands. This

necessary process,called

programming, is now performed on

one level or

another by millions of people.

(19)

2 Introduction

Programming a computer, however, requires

more

^than _just translating well-

understood instructions to alanguage a

computer can

understand. In most cases, we

need to devise

^totally ^new ^methods ^for ^solving

^a problem.

^It ^is ^not just learning ^the ^weird

language in which we \"talk\" to a computer that makes it hard to program; ^it ^is knowing what to

say. Computers execute

^not only operations that were previously

performed

^by

humans; with their enormous

speed, computers can do

^much more than was ever

possible.

Âlgorithms ôf ^the ^past ^dealt ^with ^dozens, ^maybe ^hundreds ôf

items, and,

^at

most, with thousands of instructions. Computers can deal with billions, or even trillions,

of

^bits ^of information, and can

perform

^millions ^of ^(their primitive) instructions

per second.

Designing algorithms on this order

of

magnitude is something new. It is in many respects counterintuitive. We

are used

^to thinking in terms of _things we can see and

feel.

^As ^a result, there is

a tendency

^when ^designing ^an algorithm to

use

^the

straightforward approach that works _very well for small problems. Unfortunately,

algorithms that work well for

small problems

^may ^be ^terrible ^for large

problems.

^It ^is

easy to lose sight of the complexity and inefficiency of an algorithm when

applied to

large-scale

computations.

There is another aspect to ^this problem. The algorithms we

perform

ⁱⁿ ^our daily

life are not too complicated and are not performed too often. It is_usually not worthwhile to expend

a

^lot ^of ^effort ^to develop ^the perfect algorithm. The payoff is

too small.

^For

example, consider the problem

of

unpacking grocery bags. There are obviously

less

efficient

^and ^more ^efficient _ways _{of doing} it, depending on the contents

of

^the bags and the _way the kitchen is organized.

Few people spend

^time ^even ^thinking ^about ^this

problem,

^much ^less developing algorithms for it. On the other hand,

people

^who ^do

large-scale commercial packing ^and unpacking must develop good

methods.

^Another

example is mowing the lawn. We can improve the _mowing _by _minimizing the number

of

turns, the total time for mowing,

or

the length of the trips to ^the garbage cans. Again,

unless

one

_really hates mowing the lawn,

one

would not spend an hour _figuring ^out ^how

to save

a

minute of mowing. Computers, on the other hand, can deal with very

complicated tasks, and _they _may have to perform

those tasks

_many ^times. ^It is

worthwhile to

spend a

^lot ^of ^time _designing better

methods, even

^if ^the resulting algorithms are

more complicated

^and ^harder ^to understand. The _potential of a payoff is much

greater. (Of course,

^we ^should ^not overoptimize, spending ^hours ^of programming time to

save overall a

^few ^seconds ^of _computer time.)

These

^two ^issues \342\200\224the need for counterintuitive

approaches to large-scale

algorithms and the possible complexities

of these

algorithms \342\200\224

point to the difficulties in learning this subject. First, we^must ^realize ^that straightforward intuitive methods

are

^not

always the best. It is_important to continue the search for better

methods. To do

that, we

need of course, to learn ^new ^methods. ^This book surveys ^and illustrates numerous methods for _algorithm _design. But it is not enough to learn even

a large

^number ^of

methods, just as it is not enough to

memorize

many games of chess in order to be a

good player. One

^must ^understand ^the principles behind the methods. One must know how to apply them and,

more

important, when to apply them.

A design and implementation of an algorithm is analogous to a

design

^and

(20)

Introduction 3

construction of

a house.

^We ^start ^with the basic

concepts, based

on the requirements for the

house.

^It ^is ^the architect's job to

present a

plan that satisfies the requirements. ^It ^is

the engineer's job to

make sure

^that ^the plan isfeasible and correct (so that the

house

^will

not collapse after ashort while). It is then the builder's job to construct the house based on these

plans. Of course,

^all along the way, the

costs associated

^with êach step must be analyzed ând ^taken înto âccount. Each

job is

^different, ^but they are all related and

intertwined. A design of an algorithm also starts with the basic ideas and

methods.

Then,a

plan is made. We must

prove

^the correctness of the plan ^and ^make ^sure ^that its

cost is effective.

^The ^last step is to

implement

^the ^algorithm ^for ^a particular

computer.

Risking oversimplification, we can divide the process into four steps:

design, proof

^of

correctness, analysis, and implementation. Again, each of these steps is ^different, ^but

they are all related. None

of

^them ^can ^be made in a vacuum, without a regard to ^the

others. One rarely goes through these steps in linear

order.

Difficulties arise in all

phases

of

^the construction. They usually require

modifications to

the design, which in turn require ^another feasibility proof, adjustment of

costs, and change

^of implementation.

This book concentrates on the first step, the design

of

algorithms. Following our analogy, the

book could

^have ^been ^entitled The Architecture

of

Algorithms. However, computer architecture has

a

^different _meaning, ^so _using this term would be confusing.

The book

does

not, however, ignore all the other

aspects.

^A ^discussion ^of correctness,

analysis, ^and implementation follows the description

of most

algorithms \342\200\224in detail for some algorithms, briefly for others. The emphasis is on

methods of design.

It is not enough tolearn many algorithms to be

a good

^architect ^and ^to ^be ^{able to}

design

^new algorithms. One must understand the principles behind the design.

We employ a

^different way of explaining algorithms ⁱⁿ ^this ^book. ^First, we _try to lead the reader to find his or her own solution; we strongly believe that the

best

way to learn how to

create

something is to try to

create

it. Second, and more important, we follow

a

methodology for designing algorithms that helps ^this ^creative process. The methodology,

introduced in Manber [1988], provides ^an elegant intuitive framework for _explaining the

design of algorithms ⁱⁿ ^more depth. It also

provides a

^unified way to approach the

design. The

^different ^methods ^that are encompassed by this methodology, and their numerous variations, are instances of the

same technique.

^The ^process ^of choosing among

those

_many _possible methods and applying them

becomes more methodical.

^This

methodology does not cover all

possible

ways of designing algorithms. It is useful,

however, for a great majority of the algorithms ⁱⁿ ^this ^book.

The methodology is

based on

mathematical induction. The heart of it lies in an analogy

between

^the intellectual process of proving

mathematical theorems

^and ^that ^of

designing combinatorial

algorithms. The

^main ^idea ⁱⁿ the principle

of

mathematical induction is that astatement

need

not be proven from scratch: It is sufficient to show that the correctness of the statement

follows

^from ^the correctness of the

same

^statement ^for

smaller instances and the correctness of the statement for a

small base case.

Translating this principle to algorithm

design suggests

^an ^approach ^that concentrates on _extending

1The two wonderful books by Tracy Kidder, The Soul of a NewMachine (Little Brown, 1981), and House

(Houghton Mifflin, 1985), inspired thisanalogy.