Part 5 · The Maths III: Inside the SOTA Methods
The √d fix: why vanilla GPs drown, and the prior that saves them
Start with the failure. For two independent uniform points x, x′ in [0,1]d, each coordinate contributes 𝔼[(xi−x′i)²] = 2·Var(u) = 1/6, so:
d the number of input dimensions · the expectation is over two points drawn uniformly in the unit cube — squared distance grows linearly with d
Feed that into the RBF kernel with a fixed lengthscale ℓ: the typical correlation between random points is exp(−d/(12ℓ²)), which crashes to zero as d grows. Every point is “far from the data,” the posterior reverts to the prior everywhere, and BO degenerates into uniform sampling. The 2024 “vanilla BO” result (Hvarfner et al.) is, at heart, one observation: keep ‖x−x′‖²/ℓ² stable by scaling the lengthscale like √d — implemented as a dimension-scaled prior:
With that one change, plain GP-EI matches the specialized high-dimensional machinery on many benchmarks — the claim the ICLR 2025 paper sharpened and the Nov 2025 rebuttal now contests. The controversy is real, but the √d arithmetic above is not in dispute; what is disputed is whether it is the whole story.