Raghuram Shankar
← Bayesian Optimization
← Conditioning: the posterior in closed form⌂ Fifty Evaluations to Find the Best: Bayesian OptimizationQuiz: The Maths I: Gaussian Process Surrogates →

Part 3 · The Maths I: Gaussian Process Surrogates

Choosing hyperparameters: the marginal likelihood

Everything above assumed ℓ and σn were known. In practice they are fit by maximizing the log marginal likelihood — the probability the prior assigns to the data, with the latent function integrated out (closed form, because everything is Gaussian):

log p(y | X, θ) = −½ yT(Kθ + σn²I)−1y  −  ½ log det(Kθ + σn²I)  −  n2 log 2π

θ the kernel hyperparameters being fitted · Kθ the kernel matrix they produce · y the observations, X their inputs · n how many there are

The first term rewards fitting the data; the second — the log-determinant — punishes models flexible enough to fit anything (short lengthscales inflate it). Maximizing their sum is an automatic Occam’s razor. Feel it yourself:

Five fixed observations, live posterior, live log marginal likelihood. Slide ℓ to the extremes: very short fits the data but pays a huge complexity penalty; very long is simple but fits badly. The LML peak sits where the working posterior looks “right” — that agreement is the point.

One honest caveat: the LML surface can be multi-modal (a “fits-the-noise” mode and a “smooths-through” mode), which is why careful implementations use restarts or, as in Snoek et al. (2012), integrate over hyperparameters with MCMC rather than committing to one point estimate.

← Conditioning: the posterior in closed formQuiz: The Maths I: Gaussian Process Surrogates →