Raghuram Shankar
← Bayesian Optimization
← Three live SOTA threads (2024 → mid-2026)⌂ Fifty Evaluations to Find the Best: Bayesian OptimizationQuiz: The Papers & Sources →

Part 8 · The Papers & Sources

The complete reference list

  1. H. J. Kushner (1964). A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise. J. Basic Engineering 86(1). Probability of improvement.
  2. J. Mockus (1974/1978). On Bayesian Methods for Seeking the Extremum. Optimization Techniques / Towards Global Optimization 2. Expected improvement; the name.
  3. Donald R. Jones, Matthias Schonlau, William J. Welch (1998). Efficient Global Optimization of Expensive Black-Box Functions. J. Global Optimization 13:455–492. The seminal EGO template.
  4. Niranjan Srinivas, Andreas Krause, Sham Kakade, Matthias Seeger (2010). Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design. ICML 2010. arXiv:0912.3995.
  5. James Bergstra, Rémi Bardenet, Yoshua Bengio, Balázs Kégl (2011). Algorithms for Hyper-Parameter Optimization (TPE). NeurIPS 2011.
  6. Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown (2011). Sequential Model-Based Optimization for General Algorithm Configuration (SMAC). LION 5.
  7. Jasper Snoek, Hugo Larochelle, Ryan P. Adams (2012). Practical Bayesian Optimization of Machine Learning Algorithms. NeurIPS 2012. arXiv:1206.2944.
  8. Philipp Hennig, Christian J. Schuler (2012). Entropy Search for Information-Efficient Global Optimization. arXiv:1112.1217.
  9. José Miguel Hernández-Lobato, Matthew W. Hoffman, Zoubin Ghahramani (2014). Predictive Entropy Search. NeurIPS 2014. arXiv:1406.2541.
  10. Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, Nando de Freitas (2016). Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proc. IEEE 104(1).
  11. Zi Wang, Stefanie Jegelka (2017). Max-value Entropy Search for Efficient Bayesian Optimization. ICML 2017. arXiv:1703.01968.
  12. Peter I. Frazier (2018). A Tutorial on Bayesian Optimization. arXiv:1807.02811. This page’s notation.
  13. David Eriksson, Michael Pearce, Jacob R. Gardner, Ryan Turner, Matthias Poloczek (2019). Scalable Global Optimization via Local Bayesian Optimization (TuRBO). NeurIPS 2019. arXiv:1910.01739.
  14. Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, Eytan Bakshy (2020). BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. NeurIPS 2020. arXiv:1910.06403.
  15. Samuel Daulton, Maximilian Balandat, Eytan Bakshy (2020). Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization (qEHVI). NeurIPS 2020. arXiv:2006.05078.
  16. David Eriksson, Martin Jankowiak (2021). High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces (SAASBO). UAI 2021. arXiv:2103.00349.
  17. Benjamin J. Shields, Jason Stevens, Jun Li, et al. (2021). Bayesian reaction optimization as a tool for chemical synthesis. Nature 590:89–96.
  18. Carl Hvarfner, Danny Stoll, Artur Souza, Marius Lindauer, Frank Hutter, Luigi Nardi (2022). πBO: Augmenting Acquisition Functions with User Beliefs. ICLR 2022. arXiv:2204.11051.
  19. Yutian Chen et al. (2022). Towards Learning Universal Hyperparameter Optimizers with Transformers (OptFormer). NeurIPS 2022. arXiv:2205.13320.
  20. Alexander I. Cowen-Rivers et al. (2022). HEBO: Pushing the Limits of Sample-Efficient Hyper-parameter Optimisation. JAIR 74; modernized in arXiv:2607.10669 (2026).
  21. Samuel Müller, Matthias Feurer, Noah Hollmann, Frank Hutter (2023). PFNs4BO: In-Context Learning for Bayesian Optimization. ICML 2023. arXiv:2305.17535.
  22. Leonard Papenmeier, Luigi Nardi, Matthias Poloczek (2023). Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spaces. NeurIPS 2023. arXiv:2307.00618.
  23. Sebastian Ament, Samuel Daulton, David Eriksson, Maximilian Balandat, Eytan Bakshy (2023). Unexpected Improvements to Expected Improvement for Bayesian Optimization (LogEI). NeurIPS 2023. arXiv:2310.20708.
  24. Carl Hvarfner, Erik Orm Hellsten, Luigi Nardi (2024). Vanilla Bayesian Optimization Performs Great in High Dimensions. ICML 2024. arXiv:2402.02229.
  25. Zhitong Xu, Shandian Zhe (2024). Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization. ICLR 2025. arXiv:2402.02746.
  26. Tennison Liu, Nicolás Astorga, Nabeel Seedat, Mihaela van der Schaar (2024). Large Language Models to Enhance Bayesian Optimization (LLAMBO). ICLR 2024. arXiv:2402.03921.
  27. Willie Neiswanger et al. (2025). Information-theoretic Bayesian Optimization: Survey and Tutorial. arXiv:2502.06789.
  28. (2025). GIT-BO: High-Dimensional Bayesian Optimization with Tabular Foundation Models. arXiv:2505.20685.
  29. (2025). LLMs for Bayesian Optimization in Scientific Domains: Are We There Yet?. arXiv:2509.21403.
  30. Leonard Papenmeier et al. (2025). We Still Don’t Understand High-Dimensional Bayesian Optimization. arXiv:2512.00170.
  31. (2026). Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization. arXiv:2605.20249.
  32. (2026). Evidence-Gated LLM Priors for Multi-Objective Bayesian Optimization. arXiv:2606.01730.
  33. (2026). When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study. arXiv:2606.21641.
  34. (2026). LLM-Driven Evolutionary Generation of Multi-Objective Bayesian Optimization Algorithms. arXiv:2607.08791.
  35. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, Masanori Koyama (2019). Optuna: A Next-generation Hyperparameter Optimization Framework. KDD 2019.
  36. Roman Garnett (2023). Bayesian Optimization. Cambridge University Press. Free at bayesoptbook.com (verified live).
  37. Carl E. Rasmussen, Christopher K. I. Williams (2006). Gaussian Processes for Machine Learning. MIT Press. Free at gaussianprocess.org/gpml.
  38. Robert B. Gramacy (2020). Surrogates. CRC. Free at bobby.gramacy.com/surrogates.
  39. (2023). Multi-fidelity Bayesian Optimization: A Review. arXiv:2311.13050. The fidelity axis this page leaves aside.
  40. James Bergstra, Yoshua Bengio (2012). Random Search for Hyper-Parameter Optimization. JMLR 13. The random-beats-grid argument in the Background.
  41. Nikolaus Hansen (2016). The CMA Evolution Strategy: A Tutorial. arXiv:1604.00772. The evolutionary baseline in tables and quiz distractors.
  42. Peter I. Frazier, Warren B. Powell, Savas Dayanik (2008). A Knowledge-Gradient Policy for Sequential Information Collection. SIAM J. Control Optim. 47(5). Introduces the knowledge gradient for discrete alternatives — the criterion derived in Part 4.
  43. Warren Scott, Peter I. Frazier, Warren B. Powell (2011). The Correlated Knowledge Gradient for Simulation Optimization of Continuous Parameters using Gaussian Process Regression. SIAM J. Optimization 21(3). KGCP: carries the knowledge gradient to continuous spaces under a GP prior by discretizing the inner maximum.
  44. Jian Wu, Peter I. Frazier (2016). The Parallel Knowledge Gradient Method for Batch Bayesian Optimization. arXiv:1606.04414. q-KG: the Bayes-optimal batch, and unlike heuristic batch-EI it handles noise directly.
  45. Jian Wu, Matthias Poloczek, Andrew Gordon Wilson, Peter I. Frazier (2017). Bayesian Optimization with Gradients. arXiv:1703.04389. d-KG: uses derivative observations when the objective is itself a training run.
  46. Peter I. Frazier (2018). A Tutorial on Bayesian Optimization. arXiv:1807.02811. The clearest single account of expected improvement, entropy search and the knowledge gradient side by side.
  47. Jacob R. Gardner et al. (2018). GPyTorch. NeurIPS 2018. The GP engine under BoTorch.

Suggested reading order: Frazier’s tutorial first (this page with more proofs), EGO for the origin, Snoek et al. for the practice, LogEI to update your defaults, then the three high-dimensional papers as a set — a live scientific argument in motion. Garnett’s book whenever a chapter-length treatment is wanted.

← Three live SOTA threads (2024 → mid-2026)Quiz: The Papers & Sources →