Part 5 · The Maths III: Inside the SOTA Methods
PFNs: surrogates without fitting
The pretrained-surrogate line replaces GP fitting with in-context learning. Draw synthetic tasks from a prior over functions; train a transformer qθ to predict a held-out value from the context set D by straight cross-entropy:
qθ the transformer's predictive distribution · D a dataset drawn from the prior · x*, y* a held-out query and its true value · minimizing this over many synthetic datasets makes one forward pass approximate Bayesian inference
The identity that makes this Bayesian rather than a heuristic: that expectation decomposes as 𝔼D,x*[ KL( p(·|x*, D) ‖ qθ(·|x*, D) ) ] plus an entropy constant — so the loss is minimized exactly when qθ equals the true posterior predictive under the synthetic prior. A PFN is amortized Bayesian inference: the prior is baked into the weights, and the O(n³) fit becomes one forward pass. PFNs4BO plugs this predictive (a binned “Riemann” distribution over y) straight into EI; GIT-BO rides the same idea on tabular foundation models into hundreds of dimensions. The open question is the one you’d expect: you no longer choose the prior — the pretraining distribution did, and auditing what a network believes is harder than reading a kernel.