On a multivariate implementation of the Gibbs sampler

HAL Id: hal-00894125

https://hal.archives-ouvertes.fr/hal-00894125

Submitted on 1 Jan 1996

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

On a multivariate implementation of the Gibbs sampler

La García-Cortés, D Sorensen

To cite this version:

Note

On

a

multivariate

implementation

of

the Gibbs

_sampler

LA

_{García-Cortés,}

D

Sorensen*

National Institute

_of

Animal

Science,

Research Center

Foulum,

PB 39, DK-8830

Tjele,

Denmark

(Received

2

_{August 1995; accepted}

30 October

₁₉₉₅₎

Summary - It is well established that when the parameters in a model are

correlated,

the

rate of convergence of Gibbs chains to the

_{appropriate stationary}

distributions is faster and Monte-Carlo variances of features of these distributions are lower for a

_given

chain

length,

when the Gibbs

_sampler

is

_{implemented by blocking}

the correlated _parameters and

_sampling

from the

_respective

conditional

_posterior

distributions takes

_place

in a

multivariate rather than in a scalar fashion. This block

_sampling

_strategy often _requires

knowledge

of the inverse of

_large

matrices. In this note a block

_sampling

_strategy is

implemented

which circumvents the use of these inverses. The

_{algorithm applies}

in the

context of the Gaussian model and is illustrated with a small simulated data set.

Gibbs sampling

/

block _sampling

_/

_{Bayesian analysis}

Résumé - Une mise en oeuvre multivariate de l’échantillonnage de Gibbs. Il est

bien établi que,

lorsque

les

paramètres

d’un modèle sont

corrélés,

l’estimation de ces

paramètres

par

échantillonnage

de Gibbs converge lentement

lorsque

les composantes du modèle sont traitées

_{séparément.}

Mais, si

_{l’échantillonnage}

de Gibbs est conduit en

fixant

des valeurs pour les paramètres corrélés et en échantillonnant dans les distributions conditionnelles respectives, la convergence est

plus rapide

et les variances de Monte-Carlo des

caractéristiques

des distributions sont diminuées pour une chaîne de

longueur

donnée. Cet

échantillonnage

multidimensionnel et non

_plus

scalaire requiert souvent l’inversion de matrices de

grande

taille. Cette note

_présente

une méthode

_{d’échantillonnage}

en bloc

de ce _{type qui} évite le passage par ces inverses.

_{L’algorithme s’applique}

dans le contexte

d’un modèle

_gaussien

et est illustré

numériquement

sur un _petit échantillon de données

simulées.

échantillonnage de Gibbs

_/

_{échantillonnage} en bloc

/

_{analyse bayésienne}

*

INTRODUCTION

The Gibbs

_sampler

is a numerical

_technique

that has received considerable attention

in statistics and animal

_breeding.

It is

_particularly

useful in the solution of

_high

dimensional

_integrations

and has therefore been

_applied

in likelihood and

_Bayesian

inference

_problems

in a wide

_variety

of models.

The Gibbs

_sampler

_produces

realizations from a

_{joint posterior}

distribution

by sampling repeatedly

from the full conditional

posterior

distributions of the

parameters

of the model. In

_theory,

_drawing

from the

_{joint posterior}

_density

takes

place

only

in the

_limit,

as the number of

_drawings

becomes infinite. The

_study

of

convergence to the

appropriate

distributions is still an active area of

research,

but it is well established that _convergence can be slow when

_highly

correlated scalar

components

are treated

individually

(Smith

and

Roberts,

1993).

In such cases

it is

_preferable

to block the scalars and to

_{perform sampling}

from multivariate conditional distributions. Liu et al

₍₁₉₉₄₎

show that a block

_sampling

_strategy

can lead to

_considerably

smaller Monte-Carlo variances of estimates of features of

posterior

distributions.

In animal

_{breeding applications}

this block

_sampling

strategy

can be difficult to

implement

because

_repeated

inversions and factorizations of _very

_large

matrices are

required

in order to

_perform

the multivariate

_sampling.

The _purpose of this note is to describe a block

_sampling

_{computer strategy}

which does not

_require

knowledge

of the inverse of these matrices. The results of

_applying

the method are illustrated

using

a small simulated data set. THE MODEL

Let y, a and b

_represent

vectors of data

_{(order n),}

of additive

_genetic

values

(order q)

and of fixed effects

_{(order p),}

_{respectively,}

and let X and Z be

_design

matrices

_associating

the data with the additive

_genetic

values and fixed

effects,

respectively.

We will assume that the data are

_{conditionally normally distributed,}

that is:

where

_Q

_e

_is

the residual variance.

_Invoking

an infinitesimal additive

_{genetic model,}

the distribution of additive

_genetic

values is also normal:

where A is the known numerator

_relationship

matrix and

_o,2

_is

the additive

_genetic

variance. For illustration purposes we will assume that the vector of fixed effects

b,

and of the variance

components

Q

a

and

Q

e

are all a

priori

independent

and

that

_they

follow _proper uniform distributions. Under the

_model,

the

_{joint posterior}

distribution of the

parameters

is:

METHODS

and C =

W’W+,f2,

where k =

e a

Then the mixed-model

equations

associated

with

_[1]

and

_[2]

are:

The

_{implementation}

of the Gibbs

_{sampler requires}

_sampling

0 from:

and

_!2

_from:

where 9

=

E(6!, af, y)

satisfies the linear

_system

[4],

and

C-

l

(j;

=

Var(0 )a£ ,

!e!y)!

In

_[6],

S

a

= a’A-

1

a; v

a

= q - 2; S

e

=

(y-Xb-Za)’(y-Xb-Za); !

= n - 2;

and

_X

_;;,

₂

is an inverse chi _{square variate} with Vi

degrees

of freedom. In order to

sample the

whole vector 0

_{simultaneously}

from

_[5]

without

_involving

the inverse of

the coefficient matrix

_C,

we draw from ideas in Garcia-Cortes et al

_{(1992, 1995).}

Given

starting

values for

_Q

_a,

_Q

_e,

_apply

the method of

composition

(eg,

Tanner,

1993)

to solve the

_following

_equation:

To obtain

_samples

from

_p(y!,o!),

draw

0

*

from

_{p (e l(j!(j!)}

and

_y

_*

from

p(y!e*,!,er!).

Then

_(y

_*

_{, 0}

_*

₎

is a

_drawing

from

p (y, 0)a£, af)

and

(y

*

)

from

p

(YI!a! !e!’

Further,

the

pair

(y

*

,

0

*

)

can also be viewed as a realized value from

p ( 0 ) a£ ,

af

, y

*

)

and from

_p(

_YI

₀

_*

_,

_Q

_a,

_Q

_e

_J’

By analogy

with the

_identity:

we define the random variable:

where

the random variable 0 in

_[9]

has

_density

_{p(o 10,2, a U2, e y*).}

The

_expected

value

of 0 in

_[9]

with

_respect

to

_{p(o 10,}

₂

_,

_U2

_,

₎

_y

_*

is

_equal

to

_{E ( 0 ) a£ , 0}

_,

₂

_{, y)}

and the variance is

_C-

_l

_(j;,

_{independent_of y}

_*

_.

In

addition,

the random variable 0 in

_[9]

is

_normally

which is the conditional

_posterior

distribution of interest.

_Using

_[4]

and

_!5!,

and

replacing

the random variable 0 in

_[9]

_by

its realized values

₀

_*

_,

it follows that

!*

drawings

from this conditional

_posterior

distribution,

which we denote

_{A ,}

can be

constructed

_{by solving}

the linear

_system:

A wide

_variety

of efficient

_algorithms

are available which do not

_{require C-}

1

to solve the mixed-model

_equations

_(10!.

We note in

_passing

that under the

_assumption

of either proper or

_improper

uniform

_priors

for

_b,

a

_{simple manipulation}

of

[10]

shows that this

_expression

is not a function of

_b

_*

_,

where

0*! _

(a*!b*!).

Therefore

the

_drawing

of

0

*

from _p

_af)

_£

_{(0 )a}

involves a*

_only,

and b* can be set

_arbitrarily

to _zero.

!*

With

0

_available,

draw from:

The

_samples

_Q2

_are

now used in the new round of iteration as

_input

in

_(7!.

AN EXAMPLE

As an

illustration,

the block

(multivariate)

sampling

_strategy

is

compared

with

the traditional scalar

_{implementation}

of the Gibbs

_sampler.

A data file based on a

univariate animal model with 250 individuals

_(all

with one

record,

25 individuals

per

generation,

ten

_generations)

and one fixed-effect factor with ten levels was

simulated. For both

_strategies,

a

_single

chain of

_length

31 000 was _run, and the first

1000

_samples

were discarded.

Table I shows estimates of the Monte-Carlo variance of the mean of the

marginal posterior

distributions of the first level of the fixed-effect

factor,

of the additive

_genetic

value of the last individual and of both variance

_components.

The mean was estimated

by summing

the

samples

and

dividing

by

the number

of

_samples

_{(30 000)}

and the Monte-Carlo variance was

computed using

the Markov

chain estimator

_{(Geyer, 1992).}

Also shown in table I are estimates of the chains effective

_{length. Briefly,}

the effective chain

_length

is associated with the amount of information available in a

_given

chain. This

_parameter

becomes smaller as the

dependence

between the

samples

of the chain increases. When the

samples

are in

fact

_independent,

the effective and actual chain

_lengths

are

_equal.

Details can be

found in Sorensen et al

_(1995).

The

figures

in table I show that for this model and data

_structure,

there is a reduction in the Monte-Carlo variance

_by

a factor of

ten

_using

the block

_sampling

_strategy

in the case of the fixed effects and

_breeding

values,

and a twofold reduction in the case of the variance

_components.

In the above

_example,

the reduction of the Monte-Carlo variance

_using

either the scalar or block

sampling

_strategies

was

compared

on the basis of the

simple

estimator of the mean of

_marginal

_posterior

distributions

_(the

raw _average of the

elements of the Gibbs

_chain).

A more efficient estimator is based on the _average of

refer to this as the mixture estimator. For

_example,

let

_X,

Y and Z denote three

parameters

and assume that interest focuses on the estimate of the mean of the

marginal

posterior

distribution of X.

The mixture estimator is

_{given by}

where the summation over i is over the n elements of the Gibbs

chain,

and the

summation

_{over j}

is over the chosen values of X.

_{Alternatively,}

when the

_integral

has a closed-form

solution,

the mixture estimator can take the form

The Monte-Carlo variances of the mixture estimator of the mean of the

marginal

posterior

distributions of the first level of the fixed effect

_factor,

of the additive

genetic

value of the last individual and of both variance

components

were

respec-tively

5.11 x

10-

3

,

7.62 x

,

10-

3

2.59 and 1.03 for the scalar

_sampling

strategy

and 0.13 x

10-

3

,

0.56 x

10-

3

,

1.30 and 0.43 for the block

_sampling

_strategy.

There is a

small increase in

_efficiency

relative to the ’raw means estimator’ in the case of the location

_parameters

but not in the case of the variance

_components.

The mixture

estimator is

_especially

beneficial when the chain mixes

_quickly

_(Liu

et

_al,

₁₉₉₄₎

and this is not the case in animal models.

CONCLUSIONS

We have

_presented

an

_algorithm

which allows us to

_implement

the Gibbs

_sampler

in a multivariate fashion. The

_development

is in terms of the

_single

trait Gaussian

model,

but extension to a

_multiple

trait

_analysis

with

_arbitrary

_pattern

of

_missing

data is

_{straightforward,}

provided

the

procedure

is

_used

in

_conjunction

with data

and data structure. The

_important

feature of the

strategy

is that it

_only

involves the solution of a linear

_system.

This means that either

_computer

time or

_storage

requirements

can be

_optimized

_by

choice of the

_appropriate

method to solve the linear

_system.

This is in contrast with the scalar Gibbs

_sampler

which has

_computer

requirements analogous

to Gauss-Seidel iterative methods.

ACKNOWLEDGMENTS

Our

colleagues

D Gianola and S Andersen contributed with

_insightful

comments.

REFERENCES

Garcia-Cortes

_LA,

Moreno

_C,

Varona

_L,

Altarriba J

₍₁₉₉₂₎

Variance component

estima-tion

_{by resampling.}

J Anim Breed Genet

109,

358-363

Garcia-Cort6s

LA,

Moreno

_C,

Varona

_L,

Altarriba J

₍₁₉₉₅₎

Estimation of

_prediction

error

variances

by resampling.

J Ani7n, Breed Genet

_112,

176-182

Gelfand

AE,

Smith AFM

₍₁₉₉₀₎

_{Sampling-based approaches}

to

_calculating

_marginal

densities. J Am Stat Assoc

_85,

398-409

Gelfand

_AE,

Hills

_SE,

Racine-Poon

_A,

Smith AFM

₍₁₉₉₀₎

Illustration of

_Bayesian

inference in normal data models

_using

Gibbs

_sampling.

J Am Stat Assoc 85, 972-985

Geyer

CJ

₍₁₉₉₂₎

Practical Markov chain Monte-Carlo

_{(with discussion).}

Stat Sci 7, 46-511 1

Liu

_{JS, Wong WH, Kong}

A

₍₁₉₉₄₎

Covariance structure of the Gibbs

_sampler

with

applications

to the

_comparisons

of estimators and

_augmentation

schemes. Biometrika 81, 27-40

Smith

_AFM,

Roberts GO

₍₁₉₉₃₎

_Bayesian

computation via the Gibbs

_sampler

and related Markov chain Monte-Carlo methods. J R Stat Soc B _55, 3-23

Sorensen

DA,

Andersen

S,

Gianola

_{D, Korsgaard}

I

₍₁₉₉₅₎

_Bayesian

inference in threshold models

_using

Gibbs

sampling.

Genet Set

Evol 27,

229-249

Tanner M

₍₁₉₉₃₎

_{Tools for}

Statistical

_{Inference. Springer-Verlag,}

New

_York,

USA

Wang

CS, Rutledge JJ,

Gianola D

₍₁₉₉₄₎

_{Bayesian analysis}

of mixed linear models via Gibbs

_sampling

with an

_application

to litter size in Iberian

pigs.

Genet Sel Evol 26,