Conditioning: the posterior in closed form

Observations (with Gaussian noise variance σ_n², possibly ≈0) and the prediction target f(x) are jointly Gaussian by definition — so conditioning is not an approximation, it is an identity. Writing K for the n×n matrix K_ij = k(x_i, x_j) and k_* for the vector k(x, x_i):

μ_n(x) = k_*^T (K + σ_n²I)⁻¹ y

σ_n²(x) = k(x, x) − k_*^T (K + σ_n²I)⁻¹ k_*

μ_n, σ_n² posterior mean and variance after n observations · K the n×n kernel matrix between observed inputs · k_* the vector of kernel values between x and each observed input · y the observed values · σ_n² (inside the inverse) the observation-noise variance · I the identity matrix

Read them as prose. The mean is a data-weighted correction to the prior: y filtered through the kernel’s similarity geometry. The variance is prior variance minus explained variance — what the data could not pin down. Note that σ_n²(x) does not depend on y at all: with a fixed kernel, where you measured determines what you still don’t know, not what you measured.

Why the closed form is the whole game. One O(n³) factorization of (K + σ_n²I), then every candidate x costs O(n) for μ and O(n²) for σ — microseconds at BO’s n ≤ a few hundred. That is what lets the acquisition step hammer the posterior with thousands of queries per iteration. The same O(n³) is also the ceiling that motivates sparse GPs and the pretrained-transformer surrogates in the Papers section.

Faded derivation: the posterior from Gaussian conditioning

1Write the joint distribution of the noisy observations y and the target value f(x).

yf(x) ~ 𝒩( 0, K + σ_n²I k_*k_*^T k(x,x) )

(Block notation: the stacked vector is Gaussian with the blocked covariance.) This step is free — it is the definition of a GP plus independent Gaussian noise on the diagonal.

2State the Gaussian conditioning identity for a partitioned Gaussian (a | b).

a | b ~ 𝒩( μ_a + Σ_abΣ_bb⁻¹(b − μ_b), Σ_aa − Σ_abΣ_bb⁻¹Σ_ba )

a, b the two halves of a jointly Gaussian vector · μ_a, μ_b their prior means · Σ_ab the cross-covariance block, Σ_bb the block for the part being conditioned on

The single most useful identity in GP land; derived by completing the square in the joint density’s exponent.

3Apply it with a = f(x), b = y to get the posterior mean.

μ_n(x) = 0 + k_*^T(K + σ_n²I)⁻¹(y − 0) = k_*^T(K + σ_n²I)⁻¹ y

Σ_ab = k_*^T, Σ_bb = K + σ_n²I, priors zero.

4Same substitution for the posterior variance.

σ_n²(x) = k(x,x) − k_*^T(K + σ_n²I)⁻¹ k_*

Prior variance minus what the data explains. Independent of y — measurement locations alone determine remaining ignorance.

5Sanity check: one noiseless observation (n=1, σ_n=0, k(x₁,x₁)=1). What happens at x = x₁?

μ(x) = k(x, x₁)·y₁, σ²(x) = 1 − k(x, x₁)²

At x = x₁: k = 1, so μ = y₁ and σ² = 0 — exact interpolation. Far away: k → 0, so μ → 0 and σ² → 1 — the prior reasserts itself. Both limits are the picture from the Intuition widget.

Attempt each step on paper first. Step 2 is worth memorizing outright — it is also the innovation-free heart of Kalman filters and linear-Gaussian everything.

← A prior over functions Choosing hyperparameters: the marginal likelihood →