A prior over functions

A Gaussian process is a probability distribution over functions with one defining property: for any finite set of inputs, the vector of function values is jointly Gaussian. It is fully specified by a mean function (we take m(x) = 0 after normalizing the data) and a kernel k(x, x′) giving the covariance between the values at any two inputs:

f ~ GP(0, k), Cov(f(x), f(x′)) = k(x, x′)

f the unknown objective · GP(0, k) a Gaussian process prior with zero mean and covariance function k · x, x′ any two input points

The workhorse kernel is the squared exponential (RBF), with signal variance fixed at 1 and a lengthscale ℓ:

k(x, x′) = exp( −‖x − x′‖²2ℓ² )

‖x − x′‖ the distance between two inputs · ℓ the lengthscale: how far apart inputs must be before their values decorrelate

Nearby inputs get correlation ≈1 (the function barely moves between them); inputs more than a few lengthscales apart get correlation ≈0 (their values are essentially independent). The kernel is a hypothesis about how fast f varies — and you can see exactly what that hypothesis asserts by drawing random functions from the prior:

Slide the lengthscale. ℓ = 0.03 hypothesizes a jittery function that data can pin down only locally; ℓ = 1 hypothesizes near-linear behavior across the whole domain. Every posterior in this topic is this prior, conditioned on data.

← The Maths I: Gaussian Process Surrogates Conditioning: the posterior in closed form →