SAASBO: sparsity as a prior

Give each dimension its own inverse squared lengthscale ρ_i (this is the ARD kernel), then be Bayesian about relevance:

k(x, x′) = exp( −½ ∑_i ρ_i (x_i − x′_i)² ), τ ~ HalfCauchy(α), ρ_i ~ HalfCauchy(τ)

ρ_i the inverse squared lengthscale for dimension i · τ the global shrinkage controlling how many dimensions stay active · small ρ_i means dimension i is effectively ignored

The half-Cauchy is the load-bearing choice: its mode at zero means every dimension is switched off by default (ρ_i ≈ 0 ⇒ that coordinate leaves the kernel), while its heavy tail lets the handful of dimensions the data actually supports escape to large ρ_i. The global scale τ adapts how many escape. Inference is fully Bayesian (NUTS), with predictions averaged over posterior samples — expensive per iteration, but in BO’s regime the evaluations still dominate. Sparsity is assumed until the data earns density — the exact opposite of fitting 100 lengthscales by maximum likelihood to 30 points.

← The √d fix: why vanilla GPs drown, and the prior that saves them TuRBO: the trust-region algebra →