8.2 Marginalising over Covariance Functions
8.2.6 Model Comparison Examples
Here, we examine two toy examples to demonstrate the use of annealed im-portance sampling and sequential annealed imim-portance sampling to calculate
P(D1|θ)P(θ)
P(D1,D2|θ)P(θ)
Figure 8.4: Illustration of a potential problem with sequential evidence evaluation with annealed importance sampling. The dotted line showsp(D1|θ)p(θ)and the solid linep(D1,D2|θ)p(θ), which consists of two isolated modes.
the evidence and compare models.
Firstly, consider two GP models - one simple and isotropicHs, and the other more complex and anisotropic Hc. The models are defined by the following covariance functions:
ks(xi,xj) = exp −12 X2
d=1
(x(d)i −x(d)j )2 exp(2r0)
!
(8.27)
kc(xi,xj) = exp −12 X2
d=1
(x(d)i −x(d)j )2 exp(2rd)
!
(8.28) where r0 and r = {r1, r2} are hyperparameters controlling smoothness. Hs has a single hyperparameter controlling smoothness in all directions (spheri-cal covariance), whileHc has two hyperparameters controlling smoothness in each direction aligned with the axes (axis-aligned elliptical covariance).
16 input points from R2 were generated by sampling from a bivariate nor-mal distribution. Two data sets were generated by sampling the two Gaus-sian processes with the covariance functions above (with r0 = log(0.5), r1 =
log(0.1), r2 = log(1.0)) at the16input points to give a simple data setDs and a complex data set Dc. Gaussian prior distributions were set over r0 and r such that ri ∼ N(log(0.5),1), and their most probable values were found by maximising the posterior density. For each data set, the Laplace method was used to find a Gaussian approximation to the posterior distribution. These approximations were used as the initial sampler densities for generating sam-ples with annealed importance sampling with wmaxrat = 1.5. The importance weights were used to estimate the evidence for each model, over both data sets. The experiment was repeated 5times, and Bayes’ factor was calculated for each trial. The results for the simple data are shown in table 8.2. The evidence is mostly strongly in favour ofHs, with trial4giving evidence mod-erately in favour.
trial AIS Laplace True
1 14.0 14.4 14.1
2 15.0 15.9 15.4
3 13.5 14.4 13.9
4 3.9 4.0 3.9
5 13.4 13.7 13.5
Table 8.2: Bayes’ Factor in comparison ofHs andHc, for thesimpledataset. Values greater than1are in favour ofHs
The results for the complex data shown in table 8.3. The evidence is extremely strong in favour of Hc. The table shows Bayes’ factors calculated using an-nealed importance sampling, the Laplace approximation and brute force nu-merical integration (true value). We see that all the results are comparable, in-dicating that the Laplace approximation is sufficient to calculate the evidence in this example.
Our second model comparison example consists of a simple model,Hs, with an axis aligned covariance function defined by (7.1) (page 91), and complex model,Hc, with a rotated covariance function defined by (7.2) whereΣis pa-rameterised via Givens angles (section 7.3). For both models, equal prior dis-tributions were placed over length scales, vertical scales, and noise. The com-plex model had a uniform prior distribution over Givens angles p(ρij) = 2π1 for −π ≤ ρij ≤ π. Two sets of 2D data were generated by stretching and
trial AIS Laplace True 1 6.39×10−12 6.36×10−12 6.37×10−12 2 5.66×10−14 5.63×10−14 5.52×10−14 3 8.93×10−13 9.24×10−13 9.00×10−13 4 6.44×10−12 6.54×10−12 6.42×10−12 5 1.40×10−13 1.42×10−13 1.37×10−13
Table 8.3:Bayes’ Factor in comparison ofHsandHc, for thecomplexdataset. Values less than1are in favour ofHc
rotating a set Xs of 16samples from a spherical Gaussian distribution with varianceI. The first data set,Xa was generated by multiplying each element inXs by a diagonal matrixE = diag(0.12,1.02). The second data set, Xr was generated by multiplying the elements in Xs by a matrixVTEVwith V an orthonormal rotation matrix with Givens angles equal to π4. The result is that Xa is Gaussian distributed with covariance matrix Λ = E2, and Xr has co-variance matrixΣ = VTΛV. In other words, the first set looks like samples from an axis aligned elliptical Gaussian distribution, and the second set looks like samples from a rotated elliptical Gaussian distribution; the second set is simply a rotation of the first set.
Sequential annealed importance sampling was run on both data sets, for both the simple and complex models. 100 annealing runs were made in parallel, using a wmaxrat = 2 heuristic to design the annealing schedule. The evidence for each model was recorded for all densities p(θ), p(D1|θ)p(θ), . . . p(D1, . . .D16|θ)p(θ), and the posterior probability that the model wasHswas found from:
p(H=Hs) = p(Hs|D)
p(Hs|D) +p(Hc|D) (8.29)
= p(D|Hs)
p(D|Hs) +p(D|Hc) (8.30) given equal prior probability of each model. This process was repeated 5 times and the results are shown in figure 8.5. It is clear that once enough data has been observed that the evidence clearly favours the simple model for the axis aligned data set. Similarly, the evidence for the complex model is strong
2 4 6 8 10 12 14 16 0
0.2 0.4 0.6 0.8 1
Number of data points p(H=Hs)
Figure 8.5: Model comparison using sequential annealed importance sampling with a simple and complex GP model (see text). The posterior probability that the model is simple is shown as each point was sequentially added to the evaluation. The experi-ment was repeated5times and shown is the mean with1σerror bars for axis aligned (dotted) and rotated data (solid).
for the rotated data set. Furthermore, the extent of the1σ error bars indicate that, for this example at least, that sequential annealed importance sampling is a reasonably precise tool for model comparison.
8.3 Summary
In this chapter, we have discussed and demonstrated methods to calculate the evidence for Gaussian process models with differing covariance functions.
We have described a new method - sequential annealed importance sampling - which can be used to produce an evidence estimate that is updated with the addition of new data. In later chapters, we demonstrate how model
com-parisons done in this way can be applied to enhance the performance of the Gaussian process optimisation algorithm.
Chapter 9
Gaussian Processes for Optimisation
9.1 Introduction
Until now, we have discussed and demonstrated basic Gaussian processes for regression and introduced some enhancements allowing the construction of more sophisticated models. In this chapter, we shift focus and examine the application of some of these methods. In particular, we discuss and demon-strate the use of Gaussian processes for the efficient solution of optimisation problems.