Part 8 · The Papers & Sources

The foundations

Kushner (1964) proposed maximizing the probability of improvement over a stochastic-process model; Mockus (1974–78) introduced expected improvement and named the field. The modern template is Jones, Schonlau & Welch (1998) — EGO: kriging surrogate + closed-form EI, expensive engineering objectives falling in tens of evaluations (~8k citations); its worked examples and its honesty about kriging’s failure modes still read well. GP-UCB (arXiv:0912.3995, ICML 2010, ~4k) recast BO as a bandit problem and proved sublinear regret — the reason “BO provably converges” can be said with a straight face. And Snoek, Larochelle & Adams (2012) (arXiv:1206.2944, NeurIPS 2012, ~10k) took the machinery to ML hyperparameters, beat human experts, shipped Spearmint, and started AutoML — quietly also a great engineering paper (MCMC over GP hyperparameters, EI-per-second for cost-aware search, asynchronous parallelism).