The catch: what can go wrong

Three failure modes matter enough to name before the math:

Myopia. EI, PI and UCB are one-step criteria: they value the next evaluation as if it were the last. A truly optimal policy would plan over the remaining budget — that is a dynamic program nobody can afford, and one-step lookahead is the working compromise (knowledge-gradient and entropy-search methods push one step further).
Model mismatch. The posterior is only a map of ignorance if the prior was roughly right. A lengthscale that is too long makes the GP confidently smooth over a spike it has never seen; the acquisition then never looks there. Hyperparameter fitting (Maths I) is the defense, not a cure.
Dimensionality. Distances concentrate in high dimensions and the “far from data” region is everywhere; vanilla GPs struggle past a few dozen dimensions. What to do about it is a live research fight (the Papers section) — 2024–25 papers claim plain GPs do fine after all; a late-2025 rebuttal says we still don’t understand why or when.

	Grid	Random	CMA-ES	BO (GP + EI)
Learns from observed values	No	No	Population statistics	Full posterior model
Uncertainty estimate	—	—	—	σ(x) everywhere
Evaluations to be competitive	Hundreds+	Hundreds	Thousands	Tens
Per-decision overhead	None	None	Tiny	O(n³) GP fit + acquisition search
Sweet spot	≤3 dims, cheap f	Cheap f, many cores	Cheap f, 10²–10³ dims	Expensive f, ≤~20 dims

Honesty box. BO is not magic sample efficiency; it is a bet that model-based decisions beat blind ones when the model is decent and the budget is small. If evaluations are cheap, random search plus more hardware is genuinely hard to beat (and is the right baseline to report). If dimensions are many and structure is absent, the GP’s map of ignorance goes blank. And the entire loop inherits every pathology of its surrogate — which is why the next section is about getting that surrogate right.

← Does the model really help? Run the experiment Quiz: Intuition →