Decomposition and the control of errors in decision analytic models

WD2o .H41 4

lAUG

181988

WORKING

PAPER

ALFRED

P.

SLOAN

SCHOOL

OF

MANAGEMENT

DECOMPOSITION

AND

THE

CONTROL

OF

ERRORS

IN

DECISION ANALYTIC MODELS*

Don

N.

Kleinmuntz

Sloan School

of Management

Massachusetts

Institute

of Technology

SSWP S2025-88

May

1988

MASSACHUSETTS

INSTITUTE

OF

TECHNOLOGY

50

MEMORIAL

DRIVE

DECOMPOSITION

AND THE

CONTROL OF ERRORS

IN

DECISION

ANALYTIC MODELS*

Don N.

Kleinmuntz

Sloan School

of Management

Massachusetts

Institute

of

Technology

SSWP #2025-88

May

1988

*For

presentation

at the

Conference

on

Insights

in

Decision

Making:

Theory and Applications,

University

of Chicago,

June

12-14,

1988. I

would

like to

thank Jim

Dyer

and H.V.

Ravinder

for

providing

many suggestions and stimulating ideas

about

the

topics

covered

in

this paper.

The

comments

and

suggestions

of

the

participants of

the

Tjn.T.

UBHAHlbb

Decomposition

and

the

Control

of

Errors

in

Decision

Analytic

Models*

Don

N.

Kleinmuntz

Sloan

School

of

Management

Massachusetts

Institute

of

Technology

May

1988

ABSTRACT:

Decision analytic models rely upon the general principle of

problem decomposition: Large and complex decision problems are reduced to a

set ofrelatively simplejudgments. The component judgments are then combined

using mathematical rules derivedfrom normative theory. This paper discusses

the value of decomposition as aprocedurefor improving the consistency of

deci-sion making. Various definitions of error

and

consistency are discussed. Linear

decomposition models are argued to be particularly useful for the control of

ran-dom

response errors in the component judgments. Implications for decision

analysis research andpractice are considered, and decision makers' evaluations

of the costs and benefits of decision analysis are discussed.

1

Introduction

One

ofthe distinctive characteristics of decision research is the continuinginteraction

between

descriptive

and normative

theories of

judgment

and

choice. Historically, this involved a one-sided exchange,

where

the

normative

theory

was

taken as a given

—

intuitive responses

were

compared

to

normative

standards of optimality or rationality and, often,

were

deficient

(Edwards,

1961;

Einhorn

&;

Hogarth,

1981;

Kahneman,

Slovic,

&

Tversky, 1982;

Rapoport

&

Wallsten, 1972; Slovic, FischhofF,

*For presentation at the Confrrourc oi\ LisiglitK in DrciRinn Mnkiiig: Tlirory nnd Applicntions, University of Chicftgo, June 12 14, 1988. I would like to tlmnk Jim Dyer «nd H.V. Pftvinder for

providing mnjiy suggeRtions and stimulating ideas ahout llie topies covered in this paper. The

commentsand suggestions of the participants of the M.I.T. Judgment and Decision MakingResearch

Seminar are appreciated.

2

Kleinmuntz

&

Lichtenstein, 1977).

More

recently, the

exchange

of ideas has

become

a dialogue.

For instance, psychologists

and economists have

raised

important

questions

about

the descriptive validity of rationality

assumptions

in

economic

theory

(Hogarth

&

Reder, 1987;

Simon,

1978; Thaler, 1985).

Another

exciting

development

has

been

the use of the results of descriptive studies of decision

making

to guide

attempts

to reformulate the axiomatic foundations of utility theory (Bell

&

Farquhar, 1986;

Fishburn, 1982;

Machina,

1987).

One

area that blends

normative

logic with descriptive insight is the set of

tech-niques

known

as decision analysis.

These

techniques represent

an

engineering

ap-proach

to decision

making, drawing on

both

normative and

descriptive theory to

pre-scribe

methods

intended to

improve

the effectiveness of decision

making

(von

Win-terfeldt

&

Edwards,

1986; Winkler, 1982). Decision analysis relies

upon

the general

principle of

problem

decomposition:

A

large

and

complex

decision

problem

is

re-duced

to a set of alternatives, beliefs,

and

preferences of the decision

maker.

These

component

judgments

can then

be

combined

using

mathematical

rules derived

from

normative

theory.

An

obvious

advantage

of this

approach

is the reduction of

infor-mation

processing

demands,

since the decision

maker

can focus

on

individual

com-ponents

ofthe

problem

in turn. Also, the cognitively

demanding

task ofinformation

combination

can

be

performed

by

model,

typically

implemented on

a

computer.

Fur-thermore, the

framework

is general

enough

to incorporate information

from

diverse sources, including

both

'hard'

data

and

'soft' subjective assessments.

One

of the

important

contributions of psychological research to decision

anal-ysis has

been

in

understanding

the methodological

problems

associated with the

assessment of the

component

judgments.

In particular, research has

emphasized

the potential for serious errors in the assessment of uncertainty (Hogarth, 1975;

Lichtenstein, Fischhoff,

&

Phillips, 1982; Wallsten

&

Budescu,

1983)

and

preference (Farquhar, 1984; FischhofT, Slovic, &; Lichtenstein, 1980; Hershey,

Kunreuther,

&

Schoemaker,

1982).

The

general conclusion is that these

judgments

are extraordi-narily sensitive to: ₍₁₎ features of assessment

procedures

like response

modes

and

question formats, (2) aspects of the

problem

context, (3) the decision

maker's

own

preconceptions,

and

(4) the interaction

between

the decision

maker

and

the

ana-lyst (FischhofT, 1980). For a variety of reasons,

no

single assessment

methodology

has

emerged

or is likely to

emerge

as error-free (FischhofT, 1982). Practical

recom-mendations

for coping with error include using multiple assessment procedures for

consistency checking

and

using sensitivity analysis to

determine

whether

a

model

is robust over a plausible

range

of

assessment

errors (von Winterfeldt

&

Edwards,

1986).

Another approach

that has

been

suggested is to build error theories that can help

Decomposition

and

the Control of Eitots 3

Fischhoff, 1980).

The

purpose

of this

paper

is to contribute to the

development

of

an

error theory of this type. Specifically, I will argue that

decompositions

that use

a linear aggregation rule

have

a built-in error control

mechanism

that can help

deci-sion

makers

achieve greater consistency despite the presence of errors in

component

assessments.

Understanding

the nature of this error control can provide useful

in-sights into the theory

and

practice ofdecision analysis

and

can help to identify

some

unresolved issues related to the use of

decomposition

as a decision-aiding approach.

The

paper

is organized as follows: First, I will discuss variousdefinitions of error

in

judgment.

Second, the ability of linear

decompositions

to control response errors

is discussed. Third,

some

practical issues related to the use of

decomposition

and

some

important

issues for further research are discussed.

The

discussion concludes with an analysisofthe conflicting objectivesassociated with the use of

decomposition

in decision analysis.

2

Definitions

of

Error

in

Judgment

There

are

numerous ways

to define

and

measure

the quality of

judgment and

choice.

Much

of the decision

making

literature is

concerned

in

one

way

or another with the issue ofconsistency or inconsistency ofbehavior.

Hogarth

(1982) raises a

number

of

key points

about

the

meaning

ofconsistency that are

worth

repeating here. First,

consistency is a relative concept

—

behavior is consistent or inconsistent

when

com-pared

to

some

criterion or standard.

One

possible

standard

is consistency with the

rules ofa

normative system

(e.g., utility theory, probability theory, deductive logic,

and

so on).

Hogarth

uses the

term

logical consistency to

convey

the notion that this

comparison

has

an

all-or-none quality, since responses are either consistent with the

logical premises of the

normative system

at a particular point in time or not.

A

difll^erent

comparison

can

be

obtained

by

examining

the consistency of

behav-ioral processes across time.

Hogarth

uses the

term

process consistency to refer to

the extent to

which

an individual applies the

same

psychological rule or strategy

when

making

judgments.

One

form

of process consistency is the extent to

which an

individual applies the

same

underlying process across different problems.

An

exam-ple is provided by subjects

who

demonstrate

differenl preferences

depending on

how

a decision

problem

is described

(Tversky

fe

Kahneman,

1981).

Note

that

Tversky

and

Kahneman

have

argued

that these 'framing' effects are also a violation of

logi-cal consistency. Specifically, they identify a

normative

rule called invariance:

"Two

characterizations that the decision

maker, on

reflection,

would view

as alternative descriptions of the

same

problem

should lead to the

same

choice

—

even without the benefit ofsuch reflection"

(Tversky

_&

Kahneman,

1987, p. 69).

However,

violations of process consistency

do

not necessarily lead to violations of logical consistency or

4

Kleinmuntz

vice versa. For instance, a decision

maker

might

reflect

on

two

problem

descriptions

and

decide that they are noi alternate descriptions of the

same

problem.

The

use

of different decision processes in those

two problems need

not

have

any normative

implications.

Another form

of process consistency involves the consistency of responses to

the

same

problem

on two independent

occasions.

This form

of response variability

might be

caused

by

unpredictable fluctuations in attention or shifts in processing

strategy.

One

approach

to analyzing the

two forms

of process consistency is to

use a statistical

model

of the response process:

Judgments

are jointly

determined

by

a sysiemaiic

component and

a

random component

(Eliashberg

&

Hauser, 1985;

Hershey

&

Schoemaker,

1985;

Laskey

fe Fischer, 1987; Wallsten fc

Budescu,

1983).

The

two

components

may

be

interpreted in a

number

ofdifferent ways. Forinstance,

one

could view the systematic

component

as the individual's 'true' internal opinion,

and

the

random component

as distortions introduced in the expression of those

judgments.

A

strong version of this

assumption

is to

assume

that the internal

opinions are logically consistent.

Under

this

assumption,

in principle it is possible

to estimate the normatively appropriate true opinion

from

the observed

judgments

(Lindley, 1986; Lindley, Tversky,

t

Brown,

1979).

Unfortunately, there are persuasive

arguments

against the existence of logically

consistent internal opinions.

Judgments

of

both

probabilities

and

preferences

have

been

shown

to suffer

from

systematic response errors as well as

random

variability.

One

of the

most

pervasive systematic efl^ects are response

mode

biases

—

the use of

different question

forms

or response scales will

produce

consistent, predictable

dif-ferences in assessed probabilities (Hogarth, 1975; Wallsten

&

Budescu,

1983)

and

in assessed preferences (Goldstein &; Einhorn, 1987; Hershey,

Kunreuther,

&

Schoe-maker,

1982;

Hershey

&;

Schoemaker,

1985; Slovic fe Lichtenstein, 1983).

Another

problem

with

assuming

well-formed internal opinion is that there

seem

to

be

many

instances

where both

uncertainties

and

preferences are

ambiguous

(Einhorn

&

Hog-arth, 1985; Fischhoff", Slovic, fc Lichtenstein, 1980;

March,

1978). It

appears

that

both

beliefs

and

preferences are not

known

in advance, but are constructed

when

needed.

The

lack of fixed, precise internal

judgments

has

important

implications for

mea-suring response error: If the

same

situation

were

repeated a

number

of times,

and

an

individual

was

able to

respond

independently each time, then the set ofresponses

would

constitute a hypothetical statistical distribution.

The

expected value of this

distribution is the systematic

component

of

judgment, which

should

be viewed

not

as a true internal opinion but rather, asjointly

determined

by the person, task,

and

situation.

The

variance of this distribution is the

random component

of

judgment,

stochas-Decomposition

and

the Control of Eriors 5

tic variability in underlying opinions (Eliashberg fc Hauser, 1985). This partition

of

judgment

into systematic

and

random

components

is

based on

the psychometric concepts of validity

and

reliability, so there are a variety of techniques available for

estimation (Wallsten

&

Budescu,

1983, pp. 152-154).

What

is the appropriate

standard

for consistency in decision

making?

This

is,

of course, a 'loaded' question, since there is

no

reason to

presuppose

that only

one standard

exists.

Because

decision analysis is firmly

grounded

in a particular

normative

theory,

von

Neumann-Morgenstern

utility theory, logical consistency has

usually

been

granted precedence.

However,

recent

developments

in utility theory

have

created

an

embarrassment

of riches:

Recent

reviews report a flurry ofresearch

on

alternativeutility theories(Bell

&

Farquhar, 1986;

Machina,

1987).

Many

of these

developments

involve

weakening

utility theory

axioms

related to either iransHivity of

preferences or independence of preferences

and

probabilities.

The

issue is

no

longer

whether

decisions

ought

to

be

consistent with a

normative

standard, but rather,

which normative standard

to be consistent with.

Unfortunately, to date

no one

has

proposed

a logical basis for choosing a logical

basis for choice.

However, an important

consideration is related to the fact that

many

of the

new

formulations of utility theory are nonlinear. It is as yet uncertain

whether

the analytical convenienceoflinear

models

in decision analysis

can

be

main-tained with nonlinear

models

(Bell

&

Farquhar, 1986). In addition, linearity

may

have

other

important

advantages. Specifically, in the next section it is

argued

that

the use oflinear

decompositions

in decision analysiscan

promote

process consistency

by

controlling

random

response errors.

3

How

Does

Decomposition

Control

Error?

Consider two

different strategies for

making

a

judgment:

One

is an

unaided

intuitive

holistic procedure,

where

the individual does the best heor shecan given

presumably

limited information processing capabilities.

The

other strategy is a formal

decompo-sition procedure,

where

a large

number

ofintuitive

component judgments

are

made

and

a

mathematical

rule is used to aggregate the

judgments.

What

impact

does

the use of

decomposition

have on

response error? Following the previous discussion,

systematic

and

random

error are considered in turn.

3.1

Systematic

Error

and Convergent

Validity

One

way

to address the issue of systematic error is to

compare

the

judgments

ob-tained

through decomposition

to holistic

judgments.

These comparisons

are tests of

6

Kleinmuntz

agree, then it should be possible to observe a high degree of correlation

between

judgments

obtained using each procedure.

A

number

of studies

have

examined

the correlation

between

multiattribute utility

models

and

intuitive ratings or rankings

of a set of alternatives. For a set ofeleven studies, the typical

range

ofcorrelations

was

between

.7

and

.9,

which

supports the notion that the systematic

components

converge (von Winterfeldt fc

Edwards,

1986, pp. 364-366).

Most

ofthese studies fo-cused

on

one type ofmultiattribute utility

model,

risklessadditive models.

However,

there is also evidence indicating a high degree of

convergence

between

riskless

and

risky utility models, as well as additive

and

multiplicative

decompositions

(Barron,

von

Winterfeldt,

&

Fischer, 1984; Fischer, 1977).

In those situations

where

decomposed

and

holistic

judgments

do

tend to agree, it

seems

plausible to conclude that

decomposition

neither increases nor decreases the

frequency or severity of systematic response errors.

The

limited evidence

accumu-lated to date indicates a high degree of

agreement between

the systematic portion of

decomposition models and

the

presumably

defective systematic portion of holistic

judgments.

This raises serious questions

about

the efficacy of the

decomposition

approach

for controlling systematic biases in

judgment.

Some

caution in interpret-ing these results is needed:

High

correlations simply indicate that the systematic

portions of the

two

judgments

are linearly related.

To

actually

determine

that

de-composition

and

holistic

judgments

are the

same

requires

examining

the intercept

and

slope of the regression of

one

on

the other

—

a

comparison

that is not typically

reported in convergent validity studies. Clearly, additional investigations of the influence of

decomposition

models on

systematic biases are needed.

3.2

Random

Error

and

Models

of

Judgment

There

have

been

numerous

studies that used statistical

methods

to develop

descrip-tive

models and

measure

random

response errors in holistic

judgment

(Anderson,

1981; Einhorn,

Kleinmuntz,

&

Kleinmuntz,

1979;

Hammond,

Stewart,

Brehmer,

&

Steinmann,

1975; Slovic

&

Lichtenstein, 1971). In a typical study, an individual is

presented with a series of multidimensional stimuli

and

responses are recorded.

A

statistical technique (e.g., multiple regression) is used to fit a

model

of the

func-tional relationship

between

the stimuli

and

the

judgments.

The

statistical

model

incorporates

an

explicit

random

errorterm,

whose

variability can

be

estimated along with other

model

parameters.

Using

either this

measure

of variability or a closely

related goodness-of-fit

measure

(e.g., the correlation

between

predicted

and

actual

judgments), it is possible to directly assess the consistency of the

judgment

process

(Hammond,

Hursch, fc

Todd,

1964;

Hammond

fc

Summers,

1972). In other words,

Decomposition

and

the Control of Ettois 7

and

random

components

of holistic

judgment.

The

ability to separate the systematic

and

random components

of

judgments

has

an important

practical implication: It is possible to

improve

the consistency

of

judgment

by

replacing the

human

decision

maker

with the systematic portion

of the

judgment

process that

was

captured in the

model

(Bowman,

1963;

Dawes,

1971; Goldberg, 1970).

There

is considerable evidence that replacing

an

expert's

judgments

with a linear

model

of the expert leads to superior

performance because

the

model

uses the judge's

own

knowledge,

but uses it

more

consistently

(Camerer,

1981;

Dawes,

1979).

On

the other

hand,

this

procedure

of bootsirapping the

ex-pert is "vulnerable to

any

misconceptions or biases the

judge

may

have. Implicit in

the use of bootstrapping is the

assumption

that these biases will

be

less detrimen-tal to

performance

than the inconsistency of

unaided

human

judgment"

(Slovic &; Lichtenstein, 1971, p. 722).

This

assumption

is plausible in large part

because

the

systematic

components

of linear

models

tend to

be

insensitive to differencesin

model

formulation or

parameter

estimation

(Dawes

&; Corrigan, 1974;

von

Winterfeldt

&

Edwards,

1986, pp. 420-447).

The

success of bootstrapping suggests that formal models, broadly defined,

con-stitute a useful strategy for controlling

one

specific source of

random

error, the

in-consistenciesintroduced

by

the judge's

attempts

to

combine

information intuitively.

In fact, the psychological literature clearly supports the use of

mathematical

aggre-gation rules in place of intuitive information

combination

processes (Meehl, 1954;

Sawyer,

1966; Einhorn, 1972).

However,

there is

an

additional source oferror

when

using decomposition:

The

components

themselves are

judgments

and

are therefore

subject to errors

and

inconsistencies.

The

critical issue is

whether

errors in the

component judgments

propagate through

the aggregation process

and

influence the resulting

judgment.

3.3

Error

Propagation

in

Linear

Decomposition

A

model

of this error propagation process

was

recently

proposed

by Ravinder,

Klein-muntz,

and Dyer

(1988, abbreviated as

RKD).

RKD

originally

developed

this

model

to analyze the

impact

of

decomposition on

probability assessment.

The

specific

ques-tion addressed

was

whether

it

would be

better to intuitively assess the probability of

some

uncertain future event

A

or use a

decomposition

that explicitly uses

infor-mation about

the relationship

between

that event

and

a set of

background

events,

denoted

Bi,. .

.

,Bn-

Each

of the events B, could

be

a single event or a scenario

composed

of the intersection of multiple events. For instance,

one

could assess dif-ferent distributions for the future price of oil conditional

on

different

developments

com-8

Kleinmuntz

plex scenarios. In addition, it is also necessary to assess the

marginal

probability of

each conditioning event.

The

major

technical

requirement

is that the

background

events

form

a

mutually

exclusive

and

exhaustive set of events.

This

implies that

Pr(B,

n

BJ

=

_{for j}

^

k

and

_Er=i Pr(-5.)

=

1> so the probability of the target

event is:

Pr{A)

=

J:PTiA\B.)PT{B,).

(1)

i=l

While

the

RKD

model was

originally intended to deal only with probability

assessment, the

assumptions

are general

enough

to

permit

the analysis ofalmost

any

linear

decomposition

that

can be

formulated as a weighted average.

This

includes

additive multiattribute utility or value functions,

where

an

outcome

or alternative

X

is described by a set of attributes ii, . .

.,x„.

The

component

assessments consist

of single-attribute utility assessments u,{x,)

and

scaling constants (or weights) k^

that

sum

to 1.

The

result is the aggregate utility assessment:

U{X)

_{= J2kM^.)-}

(2)

t=l

Another

common

example

is the evaluation of lotteries or other risky prospects using subjective

expected

utility.

A

lottery

L

consists ofa set ofuncertain

outcomes

denoted

x,, each having probability p, of occurring,

where

Xl"=iPi

=

1- If the

preferences for the

outcomes

are described

by

a utility function tii(x,), then the

expected

utility is a linear

combination

of the utilities of each

outcome

weighted

by

the probabilities:

EU{L)

=

_J2pM^.)-

(3)

i=l

The

RKD

model

examines

the

way

in

which

random

errors in

component

judg-ments

combine

when

the

decomposition

rule is a linear function.

The

model

uses the

psychometric

perspective outlined in the previous section to analyze linear

de-composition rules of the general

form

n

where

a denotes the result of aggregating

component

weighis b,

and

component

evaluaiions Cj. For analytical convenience, all of the

component judgments

are

assumed

to be scaled

on

the interval

from

to 1.

The

weights _(6,) are

assumed

to

sum

to 1.

The

RKD

model

examines

the

amount

of

random

error in the

decomposition

Decomposition

and

ihe Control of Errors 9

component

judgments.

Each one

has

an expected

value (systematic portion)

and

a

standard

deviation

(random

portion).

The random

errors

may

be correlated.

The

main

results ofthe analysis are derived

from an

expression for the variance ofa, the

measure

of error in the

decomposition

estimate.

Four

major

factors help to

determine

the

amount

of

decomposition

error: First,

as the

component judgments become

more

precise(i.e., less error), the

decomposition

estimate also

becomes

more

precise. Second, the value of the systematic portions

of the

components

influences

decomposition

error, although the relationship is not

simple.

However,

the

expected

values of the weights (6,)

appear

to

be

the

most

important

factor. In particular,

when

all the

componeni

c, evaluations are assessed

with equal

amounts

of error, then

an

equal-weighting

scheme

is desirable.

However,

if a particular

component

evaluation is

known

to

be

unreliable, then

decomposi-tion error is

improved

by shifting weight

away

from

this

component

and towards

components

that are estimated with greater precision.

A

third factor that strongly

influences

decomposition

error is

dependency

among

the errors in the

components.

In particular, if the c,

components

have

positively correlated errors, then the error

associated with

decomposition

will

be

seriously inflated. Finally, it is possible to an-alyze

decomposition

error as a function of the

number

of

components

used. Initially,

as

components

are

added

to the decomposition, error decreases rapidly.

However,

this

improvement

only occurs

up

to a point: Eventually,

decomposition

error begins

to increase,

although

the rate of increase is relatively slow.

An

intuitive interpretation of the benefits of

decomposition

can

be

obtained

by

viewing the

decomposition

model

(equation 4) as a weighted average of the c,

com-ponents.

Decomposition

tends to reduce

random

error

because

a linear

composite

has less variability than its

components.

As

the

number

of

components

is increased,

the

marginal

error reduction value of each additional

component

decreases.

At

the

same

time, recall that the weights being used in the

composite

are themselves

sub-ject to

random

error, so that eventually the

added

error reduction benefit ofanother

Cj

term

is offset

by

the error in the associated weight. Positive

dependencies

among

the

component

errors

undermine

the value of

decomposition because

the quantities

being

averaged

are to

some

extent

redundant,

reducing the potential benefit to

be

derived

from

averaging.'

Finally, the

RKD

model

can

be

used to identify conditions

where decomposition

successfully controls error versus conditions

where

holistic

judgments

are preferable.

The

critical factor is the precision of the

components

relative to the precision of

holistic

judgments.

The

case for

decomposition

is easiest to

make when

the

compo-nent

judgments

are

more

reliable than the holistic

judgment, perhaps because

the

'Trrhnicftl issnos rclafcd to Hio coiiKtraiiit tlint tlir wriplitsmust i\<\<[ u]> t(i 1 iiiny roiuplirfttr tliis

10

Kleinmuntz

components

are relatively simple stimuli (e.g., single

dimensions

rather than

multi-dimensional objects). If the

components

have

small

enough

errors,

decomposition

is virtually

guaranteed

to decrease

random

error.

However,

even if the

components

are estimated with

no

greater

amount

ofreliability than the holistic

judgment,

there

is still considerable potential for the averaging process to

improve judgment.

4

Implications

for

Research

and

Practice

What

are the implications of the preceding discussion for the use of

decomposition

as a decision aiding technique?

There

are still a

number

of

open

issues

surrounding

decomposition, particularly with respect to the control of systematic biases.

How-ever,

decomposition

does

appear

to

be

an

extremely effective

method

for controlling

random

errors.

On

the other

hand,

the

way

in

which decomposition

is

implemented

can

influence the

chance

that

decomposition

will succeed.

Four

specific

implemen-tation issues will be discussed in turn: (1)

methods

for reducing the

amount

of error

in the

components;

(2) the implications of selecting different weighting schemes; ₍₃₎

the

problem

of

dependent

errors;

and

(4) the appropriate level ofdetail for

decom-position.

1.

Methods

for

reducing

component

errors.

The

reliability of the

com-ponent

judgments

is largely

determined by

ability of the decision

maker and

the

characteristics of the

problem.

However,

the analyst can use certain

methods

to

reduce the

component

errors.

One

approach

is to use the

model

fitting

approach

described

above

(section 3.2) to bootstrap the decision

maker.

For

example,

in

a multiattribute evaluation

problem,

rather than intuitively assessing the weights

for each attribute, a representative

sample

of alternatives can be presented to the

decision

maker

and

a statistical

model

used to estimate the weights.

These

model-based

estimates are likely to

be

more

precise than intuitive assessments

(Hammond,

Stewart,

Brehmer,

fc

Steinmann,

1975;

Laskey

fc Fischer, 1987).

Another approach

that

may

be quite effective is to use nested

decomposition

—

the

components

of a

decomposition

can themselves

be

estimated with decomposition. In principle,

nest-ing can

be extended

to multiple levels.

Of

course, unless this

approach

is used with

discretion, the

number

of

judgments

required

from

the decision

maker

could easily

exceed

any

reasonable

bounds.

A

less

promising

approach

is to try to obtain multiple

independent

judgments

of the

same component

and

take the average.

This

approach

failsin practice

because an

individual can not easily provide repeated independenl opinions.

A

closely related

alternative that has

been proposed

in the context of probability assessment is to

ask multiple experts for their

judgments.

Of

course, since the experts

may

have

Decomposition

and

the Control of Errors 11

with

dependent

errors

(Clemen

&; Winkler, 1985; Winkler, 1981).

A

final technique also uses multiple judges: Selected

component

judgments

can

be

delegated to experts

who

have

specialized

knowledge about

that

component.

For

instance, estimates of probabilities of decision

outcomes might be

estimated by

sci-entists or engineers, while

judgments

of values or preferences are provided

by

the

decision

maker

(Hammond

&

Adelman,

1976).

Although

there is little direct

ev-idence

on

this issue, it

seems

plausible that experts will

be

able to provide

more

reliable

judgments

in their area ofexpertise than a novice could (Wallsten

&

Bude-scu, 1983).

2.

Picking

the best

weighting

scheme.

From

the perspective of controlling

decomposition

error, the ideal set of weights is

one

that

maximizes

the influence of

precisely estimated

components

and minimizes

the influence of unreliable

compo-nents.

However,

random

assessment

errors in the weights themselves are a concern.

One

method

for dealing with these

problem

is to use a unit weighting

scheme,

where

a

predetermined

set ofequal weights are applied to the

components.

The

advantage

is that the

random

assessment

errors associated with the weights are

removed.

Un-fortunately, this

advantage

can be offset

by

the introduction of a systematic bias,

since the weights that

would have been

selected

judgmentally

are unlikely to exactly

match

the unit weights.

On

the other hand,

under

certain conditions, the size ofthe

systematic bias is likely to

be

small

enough

to

make

the reduction of

random

error

appear

relatively attractive (Einhorn, 1976;

Einhorn

&

Hogarth,

1975).

3.

Dealing

with

dependent

errors.

One

of the

most

serious threats to the

error-reduction potential of

decomposition

is the presence of

dependent

errors.

Any

time

two

judgments

are

based on

shared information, there is considerable potential

for

dependent

errors.

This

might be

a particular

problem

for

judgments

that are ob-tained using

an

anchoring- and-

adjnsimeni

strategy

(Tversky

&

Kahneman,

1974).

Previous research

on

this strategy has focused on

problems

with an inappropriate

choice of

an anchor

or the

tendency

to adjust insufficiently

from

the anchor.

How-ever, if

two

different

judgments

share the

same

anchor, then

any

response errors in

the

anchor

will be present in

both

judgments.

In fact, in

many

preference

assess-ment

methods,

a

sequence

of

judgments

are obtained

and

there are linkages

between

successive

judgments.

For instance, the results of

one

assessment

may

be

used as

an

anchor

for the next assessment, inducing a positive serial correlation in the response

errors

(Laskey

fe Fischer, 1987).

Furthermore,

factors like question presentation

for-mats

and

response

modes

can

strongly interact with the anchoring process

(Johnson

&

Schkade, in press;

Schkade

h

Johnson,

1987).

Much

more

research is

needed on

the distribution of response errors associated with various assessment techniques.

4.

Choosing

the

appropriate

level

of

detail.

A

critical issue in

12

Kleinmuntz

in multiattribute evaluations, the analyst has considerable leeway in determining

how

many

attributes to include in the model.

The

analysis of

random

error

sug-gests that the relationship

between

the precision of

decomposition

and

the

number

of

components

is single-peaked:

Adding

detail to the

model

helps control error, but only

up

to a point.

Beyond

that point, the incremental error associated with a

new

attribute

may

outweigh

any

error cancellation effect.

On

the margin, ifa

component

is

known

to

be

assessed unreliably, it will

probably

not be

worth

incorporating in

the analysis.

On

the other

hand,

there

may

be

systematic errors associated with choosing

the

wrong

level of detail: Specifically, there are a

number

of studies that suggest

that there are

problems

when

models

fail to incorporate

enough

components.

For

instance, in multiattribute evaluation models, using

an incomplete

set of factors

may,

under

certain conditions, lead to biased evaluations

(Aschenbrenner,

1977; Barron,

1987;

Barron

fe

Kleinmuntz,

1986).

Another

example

involves using fault trees to

estimate the probability that a

complex

system

will fail.

When

trees are

pruned

by

incorporating specific failure events into a catch-all 'other problems' category, decision

makers

systematically

underestimate

the probability of the missing events

and

are

unaware

that the

problem

was

incompletely represented

—

what

is out of

sight is also out of

mind

(FischhofT, Slovic,

&

Lichtenstein, 1978).

An

intriguing discussion of the relative merits ofsimple versus

complex

analyses

is provided

by

Politser

and Fineberg

(1988).

They

reviewed 24 published medical

decision analyses that incorporated a decision tree

and expected

utility calculations

and found

a relationship

between

the complexity of the analysis

and

the clarity

of the results.

Complexity

was measured

by

the

number

of terminal

branches

in

the decision tree, while clarity

was

assessed in

terms

of the

expected

utility to

be

gained

by performing

the analysis (Lindley, 1986).

The

results indicated a strong

positive correlation

between

the

complexity

of the tree

and

the clarity ofthe results.

Although

this relation held even after controlling for a variety of possible sources of

spurious correlation,

some

additional research is needed. For instance, it is not clear if the simple trees

were

reflections ofsimple

problems

or ofinsufficiently

decomposed

complex

problems.

Common

sense dictates that

more

complexity and

detail in analysesis not

always

a

good

thing. Larger

models

require greater expenditures of effort

from both

the

analyst

and

the decision

maker.

As

a

model becomes more

and

more

sophisticated,

the decision

maker

may

have

to struggle to

cope

with the

added

detail

and

com-plexity. Since the benefits of

added

complexity are likely to diminish

on

the margin,

and

since the negative

consequences

are likely to escalate as

more

and

more

detailed is

heaped

onto

an

already

complex

model,

it

seems

plausible to

argue

that there is

Decomposition

and

the Control of Ettois 13

and

the

bad

(Coombs

fe

Avninin,

1977).

Of

course, the

tough

question for decision analysts is to

determine

whether

current

modeling

practices are

above

or

below

this

level of optimal complexity.

The

answer

does not

seem

obvious.

5

The

Costs

and

Benefits of

Decision Analysis

On

the other hand, decisions often

have

to

be

made

in the presence of severe

lim-itations of time

and money.

Since decision analysis can

be both

costly

and

time

consuming, comprehensive

analysis

may

be

ruled out

under

all but the

most

unusual

circumstances.

The

relevant question for

many

decision

makers

is not

whether

to use simple or

complex

models, but

whether

to use a simplified

model

or intuition

(Behn

&

Vaupel, 1982). In this paper, decision analysis has

been

treated as an explicit

decision strategy: Error-prone intuitive processes are

augmented

by

a set offormal procedures. In reality, there are a

continuum

of available strategies, each relying to varying degrees

on

intuition

and

formal analysis.

This

perspective

emphasizes

the fact that it is the decision

maker

who

ultimately

must

determine

the extent to

which decomposition and

formal analysis should be used.

Recent

research

on

the selection of decision strategies has

emphasized

the

cog-nitive trade-offs

between

various positive

and

negative

dimensions

of alternative

strategies. Different strategies are selected

under

different task conditions if the

val-ues of these

dimensions change

(Payne, 1982).

A

generalization of this cost-benefit

view is to formulate the strategy selection process as a meiadecision

problem,

in

which one

"decides

how

to choose"

(Einhorn

&

Hogarth,

1981, p. 69).

Thus,

each

decision strategy can

be viewed

as a multidimensional object, with the

dimensions

corresponding

to the associated costs

and

benefits.

This

paper

has focused on

two

specific dimensions, the control of

random

and

systematic response errors,

and

has

touched

on two

others, logical consistency with

normative

principles

and

the effort required to

implement

decomposition.

A

num-ber of other costs

and

benefits

have been

associated with decision analytic models. Negative

dimensions

include: resistance to the use of quantification

and

failure to

appreciate the limitations of formal

models (Howard,

1980); reluctance to reveal sensitive information in

an

analysis (Keeney, 1982);

and

reliance

on

the uncertain

interpersonal

and

technical skills ofthe decision analyst (Fischhoff, 1980). Positive

dimensions

include: forcing

assumptions

to

be

scrutinized

and

justified (Hogarth,

1987, chap. 9); the

development

of insight

and understanding

of

complex

problems

through "immersion"

in the details of the

problem (Howard,

1980);

and

the

pro-motion

of

consensus

formation, conflict resolution,

and

communication

(Hammond,

1965;

Hammond

&

Adelman,

1976; Raiffa, 1982).

14

Kleinmuntz

since the decision

maker must

assess

whether

different

approaches

(e.g.,

decompo-sition, intuition)

meet

each objective. Ultimately, the decision

maker must

balance

these conflicting objectives

and

decide

what

decision

making

strategy is best.

This

is

complicated

by

many

uncertainties

and

ambiguities associated with the cost-benefit

dimensions. For

example,

many

of the

purported

benefits ofdecision analysis

can be

notoriously difficult to evaluate objectively, if they are evaluated at all (Fischhoff, 1980).

On

the other

hand,

the costs of using decision analysis are readily apparent. Since decision

makers

often

pay

more

attention to factors they

know

more

about,

the costs of decision analysis

may

outweigh

the benefits in the

minds

of

many

deci-sion

makers (Kleinmuntz

&

Schkade, 1988).

Furthermore,

the difficulty ofobtaining accurate

measures

of the benefits create precisely the conditions of

incomplete

and

ambiguous

feedback that can lead to overconfidencein one's abilities (Einhorn, 1980;

Einhorn

&

Hogarth,

1978). Therefore, the unverified claims ofanalysts

and

decision

makers about

the efficacy of decision analysis should be treated with a

measure

of skepticism.

An

increased sensitivity to decision

makers'

perceptions ofthe costs

and

benefits of analysis

can be

helpful. For instance, recognizing that decision

makers

are

more

concerned

with conserving cost

and

cognitive effort than with issues ofstrict logical

consistency has led to the

development

of simplified

modeling

techniques

(Einhorn

fc

McCoach,

1977;

Gardiner

&

Edwards,

1975).

These

linear

decomposition

techniques

are valuable because: (1) the

components judgments

tend to

be

less

complex,

so they

are less effortful and, perhaps,

more

reliable; ₍₂₎ process consistency is

promoted

because

the underlying

decomposition

is linear; (3) these

models

often

do

a

good

job

approximating

models

constructed in a

normative

framework;

and

(4)

when

the

normative

model

is nonlinear, using a linear

approximation

increases the

chance

that less technically sophisticated decision

makers

can

understand

and

use the

model

(Dyer

t

Larsen, 1984).

A

recurrent

theme

in psychological research is the adaptive interaction ofthe

or-ganism

with the

environment (Brunswik,

1952;

Simon,

1981).

Presumably

focusing

on

this interaction will

make

it possible to

understand

and

predict the effectiveness of

human

decision

makers

across a variety of tasks

and

settings.

One

goal of this

research should

be

to identify those situations

where

decision aids like

decomposi-tion are needed.

However,

it is also

important

to extend our conceptual

framework

to include the

mutual

interaction of the decision

maker,

the

environment,

and

the

decision aid, since the aid

mediates

the interaction

between

the

environment

and

the individual.

This

can help to

improve

the design

and implementation

of decision aiding techniques,

and

ultimately, lead to better decision

making.

Decomposition

and

the Control of Etiois 15

References

Anderson,

N. H. (1981).

Foundations

of infoTmaiion iniegTaiion.

New

York:

Aca-demic

Press.

Aschenbrenner,

K.

M.

(1977). Influence of attribnte formulation

on

the evaluation

of

apartments by

multiattribute utility procedures. In H.

Jungermann

&

G.

de

Zeeuw

(Eds.), Decision

making and

change in

human

affairs. Dordrecht,

The

Netherlands: Reidel.

Barron, F. H. (1987). Influence of missing attributes

on

selecting a best multiat-tributed alternative. Decision Sciences, 18, 194-205.

Barron, F. H.,

&

Kleinmuntz,

D. N. (1986). Sensitivity in value loss of linear

models

to attribute completeness. In B.

Brehmer,

H.

Jungermann,

P. Lorens,

&

G.

Sevon

(Eds.),

New

directions in research

on

decision making.

Amsterdam:

Elsevier.

Barron, F. H.,

von

Winterfeldt, D.,

&

Fischer, G.

W.

(1984). Theoretical

and

empiri-cal relationships

between

risky

and

riskless utility functions.

Ada

Psychologica,

56, 233-244.

Behn,

R. D.,

&

Vaupel, J.

W.

(1982).

Quick

analysi.i for busy decision makers.

New

York: Basic

Books.

Bell, D. E.,

&

Farquhar, P. H. (1986). Perspectives on utility theory. Operations

Research, 34, 179-183.

Bowman,

E. H. (1963). Consistency

and

optimality in

managerial

decision

making.

Management

Science, 9, 310-321.

Brunswik,

E. (1952).

Conceptual

framework

of psychology. Chicago: University of

Chicago

Press.

Camerer,

C. (1981).

General

conditions for the success of bootstrapping models. Organizational

Behavior

and

Human

Performance,

27, 411-422.

Clemen,

R. T.,

&

Winkler, R. L. (1985). Limits for the precision

and

value of

information

from

dependent

sources. Operations Research, 22, 427-442.

Coombs,

C. H.,

&

Avrunin,

G. S. (1977). Single-peaked functions

and

the theory of preference. Psychological Review, 84, 216-230.

Dawes,

R.

M.

(1971).

A

case study of

graduate

admissions: Application of three

principles of

human

decision

making.

American

Psychologist, 26, 180-188.

Dawes,

R.

M.

(1979).

The

robust

beauty

of

improper

linear

models

in decision

making.

American

Psychologist, 34, 95-106.

Dawes,

R. M.,

&

Corrigan, B. (1974). Linear

models

in decision

making.

16

Kleinmuniz

Dyer, J. S., fc Larsen, J. B. (1984).

Using

multiple objectives to

approximate

normative

models.

Annals

of Operations Research, 2, 39-58.

Edwards,

W.

(1961). Behavioral decision theory.

Annual

Review

of Psychology, 12,

473-498.

Einhorn, H. J. (1972).

Expert

measurement

and mechanical

combination.

Organi-zational

Behavior

and

Human

Performance,

7, 86-106.

Einhorn, H. J. (1976).

Equal

weighting in multi-attribute models:

A

rationale,

an example,

plus

some

extensions. In

M.

Schiff

&

G.

Sorter (Eds.),

Proceed-ings of the conference

on

topical research in accounting.

New

York:

New

York

University Press.

Einhorn, H. J. (1980).

Learning from

experience

and suboptimal

rules in decision

making.

In T. S. Wallsten (Ed.), Cognitive processes in choice

and

decision behavior. Hillsdale, NJ:

Erlbaum.

Einhorn, H. J.,

&

Hogarth,

R.

M.

(1975). Unit weighting

schemes

for decision

making.

Organizational

Behavior

and

Human

Performance,

13, 171-192. Einhorn, H. J., fe

Hogarth,

R.

M.

(1978).

Confidence

in

judgment:

Persistence of

the illusion of validity. Psychological Review, 85, 395-416.

Einhorn, H. J.,

&

Hogarth,

R.

M.

(1981). Behavioral decision theory: Processes of

judgment and

choice.

Annual

Review

of Psychology, 32, 53-88.

Einhorn, H. J., fc

Hogarth,

R.

M.

(1985).

Ambiguity and

uncertainty in probabilistic inference. Psychological Review, 92, 433-461.

Einhorn, H. J.,

Kleinmuntz,

D. N., fc

Kleinmuntz,

B. (1979). Linear regression

and

process-tracing

models

of

judgment.

Psychological Review, 86, 465-485. Einhorn, H. J.,

&

McCoach,

W.

(1977).

A

simple multi-attribute

procedure

for

evaluation. Behavioral Science, 22, 270-282.

Eliashberg, J.,

&

Hauser, J. R. (1985).

A

measurement

error

approach

for

modeling

consumer

risk preferences.

Management

Science, 31, 1-25.

Farquhar, P. H. (1984). Utility assessment

methods.

Management

Science, 30, 1283-1300.

Fischer,

G.

W.

(1976).

Multidimensional

utility

models

for risky

and

riskless choice.

Organizational

Behavior

and

Human

Performance,

17, 127-146.

Fischer,

G.

W.

(1977).

Convergent

validation of

decomposed

multiattribute utility

assessment

procedures for risky

and

riskless decisions. Organizational

Behavior

and

Human

Performance,

18, 295-315.

DecomposiiioTi

and

ihe Control of Errors 17

Fischhoff, B. (1982). Debiasing. In D.

Kahneman,

P. Slovic, fe A.

Tversky

(Eds.),

Judgment

under

uncertainly: Heuristics

and

biases.

New

York:

Cambridge

University Press.

Fischhoff, B., Slovic, P., fe Lichtenstein, S. (1978). Fault trees: Sensitivity of

esti-mated

failure probabilities to

problem

representation.

Journal

of

Experimental

Psychology:

Human

Perception

and Performance,

4, 330-344.

Fischhoff, B., Slovic, P.,

&

Lichtenstein, S. (1980).

Knowing

what

you

want:

Mea-suring labile values. In T. S. Wallsten (Ed.), Cognitive processes in choice

and

decision behavior. Hillsdale, NJ:

Erlbaum.

Fishburn, P. C. (1982). Nontransitive

measurable

utility.

Journal

of

Mathematical

Psychology, 26, 31-67.

Gardiner, P.

C,

&;

Edwards,

W.

(1975). Public values: Multiattribute-utility

mea-surement

for social decision

making.

In

M.

F.

Kaplan

&; S.

Schwartz

(Eds.),

Human

judgment and

decision processes.

New

York:

Academic

Press.

Goldberg,

L. R. (1970).

Man

vs.

model

of

man:

A

rationale, plus

some

evidence,

for a

method

of

improving on

clinical inferences. Psychological Bulletin, 73,

422-432.

Goldstein,

W.

M., fe Einhorn, H. J. (1987). Expression theory

and

the preference

reversal

phenomena.

Psychological Review, 94, 236-254.

Hammond,

K. R. (1965).

New

directions in research on conflict resolution.

Journal

of Social Issues, 21, 44-66.

Hammond,

K. R.,

&

Adelman,

L. (1976). Science, values,

and

human

judgment.

Science, 194, 389-396.

Hammond,

K.

R., Hursch, C. J., &;

Todd,

F. J. (1964).

Analyzing

the

components

of clinical inference. Psychological Review, 71, 438-456.

Hammond,

K. R., Stewart, T. R.,

Brehmer,

B.,

&

Steinmann,

D. O. (1975). Social

judgment

theory. In

M.

F.

Kaplan

fc S.

Schwartz

(Eds.),

Human

judgment and

decision processes.

New

York:

Academic

Press.

Hammond,

K. R.,

&

Summers,

D. A. (1972). Cognitive control. Psychological Review, 79, 58-67.

Hershey, J.

C,

Kunreuther,

H.

C,

&

Schoemaker,

P. J. H. (1982). Sources of bias in assessment procedures for utility functions.

Management

Science, 28, 936-954. Hershey, J.

C,

&

Schoemaker,

P. J. H. (1985). Probability versus certainty

equiv-alence

methods

in utility

measurement:

Are

they equivalent?

Management

Science, 31, 1213-1231.

Hogarth,

R.

M.

(1975). Cognitive processes

and

the assessment of subjective

prob-ability distributions.

Journal

of ihe

American

Statistical Association, 70,

18

Kleinmuntz

Hogarth,

R.

M.

(1982).

On

the surprise

and

delight of inconsistent responses. In

R.

M.

Hogarth

(Ed.), Question

framing and

response consistency.

San

Fran-cisco: Josey-Bass.

Hogarth,

R.

M.

(1987).

Judgement

and

choice:

The

psychology of decision

(2nd

ed.). Chichester,

UK:

Wiley.

Hogarth,

R. M.,

&

Reder,

M.

W.

(Eds.). (1987). Rational choice:

The

contrast

between

economic

and

psychology. Chicago: University of

Chicago

Press.

Howard,

R. A. (1980).

An

assessment

of decision analysis. Operations Research, 28, 4-27.

Johnson,

E. J.,

&

Schkade, D. A. (in press). Bias in utility assessments: Further evidence

and

explanations.

Management

Science.

Kahneman,

D., Slovic, P., fc Tversky, A. (Eds.). (1982).

Judgment

under

uncer-tainty: Heuristics

and

biases.

New

York:

Cambridge

University Press.

Keeney,

R. L. (1982). Decision analysis:

An

overview. Operations Research, 30, 803-838.

Kleinmuntz,

D. N., fe Schkade, D. A. (1988).

The

cognitive _{implications of}

informa-tion displays in computer-supported decision making.

Working

paper

2010-88,

Sloan School of

Management,

Massachusetts

Institute of Technology.

Laskey, K. B., &; Fischer,

G.

W.

(1987).

Estimating

utility functions in the presence

of response error.

Management

Science, 33, 965-980.

Lichtenstein, S., FischhofT, B.,

&

Phillips, L. (1982). Calibration of probabilities:

The

state of the art to 1980. In D.

Kahneman,

P. Slovic,

&

A.

Tversky

(Eds.),

Judgment

under

uncertainty: Heuristics

and

biases.

New

York:

Cambridge

University Press.

Lindley, D. V. (1986).

The

reconciliation of decision analyses. Operations Research, 34, 289-295.

Lindley, D. V., Tversky, A.,

&

Brown,

R. V. (1979).

On

the reconciliation of

prob-ability assessments (with discussion).

Journal

of the

Royal

Statistical Society

Series A, 142, 146-180.

Machina,

M.

J. (1987).

Decision-making

in the presence of risk. Science,

236,

537-543.

March,

J. G. (1978).

Bounded

rationality, ambiguity,

and

the engineering of choice. Bell

Journal

of

Economics,

9, 587-608.

Meehl, P. E. (1954). Clinical vs. statistical prediction. Minneapolis: University of

Minnesota

Press.

Payne,

J.

W.

(1982).

Contingent

decision behavior. Psychological Bulletin, 92,

Decomposition

and

the Control of Errors 19

Politser, P. E.,

&

Fineberg, H. V. (1988).

Toward

predicting the value ofa medical

decision analysis.

Unpublished

paper,

Harvard

School of Public Health.

Raiffa, H. (1982).

The

ari

and

science of negoiiaiion.

Cambridge,

MA:

Belknap/

Harvard

University Press.

Rapoport,

A.,

&

Wallsten, T. S. (1972). Individual decision behavior.

Annual

Review

of Psychology, 23, 131-175.

Ravinder, H. V.,

Kleinmuntz,

D. N.,

&

Dyer, J. S. (1988).

The

reliability of

subjec-tive probabilities obtained

through

decomposition.

Management

Science, 34,

186-199.

Sawyer,

J. (1966).

Measurement

an</prediction: Clinical an(f statistical.

Psycholog-ical Bulletin, 66, 178-200.

Schkade, D. A., &;

Johnson,

E. J. (1987). Cognitive processes in preference reversals.

Working

paper,

Department

of

Management,

University of

Texas

at Austin.

Simon,

H. A. (1978). Rationality as process

and

as

product

of thought.

American

Economic

Review, 68, 1-16.

Simon,

H. A. (1981).

The

sciences of the artificial (2nd ed.).

Cambridge,

MA:

MIT

Press.

Slovic, P., Fischhoff, B.,

&

Lichtenstein, S. (1977). Behavioral decision theory.

Annual

Review

of Psychology, 28, 1-39.

Slovic, P., Si Lichtenstein, S. (1971).

Comparison

of Bayesian

and

regression

ap-proaches to the

study

of information processing in

judgment.

Organizational

Behavior

and

Human

Performance,

6, 649-744.

Slovic, P.,

&

Lichtenstein, S. (1983). Preference reversals:

A

broader

perspective.

American Economic

Review, 73, 596-605.

Thaler, R. (1985).

Mental

accounting

and

consumer

choice.

Marketing

Science, 4,

199-214.

Tversky, A., &:

Kahneman,

D. (1974).

Judgment

under

uncertainty: Heuristics

and

biases. Science, 185, 1124-1131.

Tversky, A., fe

Kahneman,

D. (1981).

The

framing

ofdecisions

and

the

psychology

of choice. Science,

211,

281-299.

Tversky, A.,

&

Kahneman,

D. (1987). Rational choice

and

the

framing

of decisions.

In R.

M.

Hogarth

&

M.

W.

Reder

(Eds.), Rational choice:

The

contrast between

economics

and

psychology. Chicago: University of

Chicago

Press.

von

Winterfeldt, D.,

&

Edwards,

W.

(1986). Decision analysis

and

behavioral

re-search.

New

York:

Cambridge

University Press.

Wallsten, T. S.,

&

Budescu,

D. V. (1983).

Encoding

subjective probabilities:

A

„^

Kleinmuniz

Winkler, R. L. (1981).

Combining

probability distributions

from dependent

infor-mation

sources.

Management

Science, 27, 479-488.

Winkler, R. L. (1982).

Research

directions in decision

making

under

uncertainty.

Decision Sciences, 13, 517-533.

Date

Due

MITLlBRflftlES