TL;DR
The Linearized Laplace Approximation (LLA) turns a standard neural network into an expressive Gaussian process (GP) whose covariance is governed by the empirical neural tangent kernel. This paper investigates LLA as a surrogate model for Bayesian optimization, showing that it outperforms standard GPs in high-dimensional, structured problems like image optimization, but exposes a critical failure mode: standard ReLU activations cause acquisition functions to pathologically diverge toward infinity in unbounded search spaces.
What Problem Does This Paper Address?
Gaussian processes (GPs) with simple stationary kernels (like Radial Basis Functions, or RBF) are the default surrogate models in Bayesian optimization (BO). However, standard GPs struggle in high-dimensional search spaces or structured domains (e.g., images) where spatial inductive biases are crucial.
While Bayesian neural networks (BNNs) offer flexible representation and inductive bias, typical inference schemes (such as MCMC or variational inference) can be computationally prohibitive or difficult to tune in small-sample sequential regimes. The authors study whether the linearized Laplace approximation (LLA)—a method that bridges deep networks and GPs—can serve as a practical, scalable BO surrogate, while identifying its structural risks.
Methods
- Linearized Laplace Approximation (LLA): The network is first trained to find the maximum-a-posteriori (MAP) weight estimate $\theta_{\text{MAP}}$. The model is then linearized via a first-order Taylor expansion around $\theta_{\text{MAP}}$, yielding $f^{\text{lin}}(x; \theta) \approx f(x; \theta_{\text{MAP}}) + J(x)(\theta - \theta_{\text{MAP}})$, where $J(x)$ is the Jacobian. Under a Gaussian Laplace approximation to the posterior with a generalized Gauss-Newton (GGN) Hessian, this induces a GP over function space with:
- Mean function: The base network's MAP prediction $f(x; \theta_{\text{MAP}})$.
- Covariance function: The network's empirical Neural Tangent Kernel (NTK), $k(x_1, x_2) = J(x_1) J(x_2)^\top$.
- Dual Representation: Inference can be computed either in weight space (parametric) or function space (non-parametric), allowing flexible memory-runtime trade-offs.
- Hyperparameter Tuning: Compares online tuning (optimizing the marginal likelihood periodically during NN training) against post-hoc tuning (optimizing prior precision once after finding the MAP).
- Acquisition Functions: Evaluated using Expected Improvement (EI) and Upper Confidence Bound (UCB).
Datasets
- Bayesian Optimization Benchmarks:
- Branin function: 2D continuous optimization over $[-5, 10] \times [0, 15]$.
- Ackley function: Highly multimodal 2D benchmark over $[-32.768, 32.768]^2$.
- MNIST generation: A 784-dimensional image matching task optimizing $f(x) = \|x - x_0\|$ on $[0, 1]^{784}$ for a target image $x_0$.
- Small-Data Regression: Four UCI benchmark datasets (Concrete, Power, Naval, and Kin8nm) evaluated on 1 to 200 data points.
- Joint Uncertainty Evaluation: The
CLASSIFICATION_2D_TESTsubset from DeepMind’s Neural Testbed (seven 2D synthetic binary classification tasks).
Key Insights and Ideas
- Inherited Inductive Biases: Because the LLA posterior mean is identical to the MAP prediction, the surrogate inherits the architectural priors of the base network (e.g., translation invariance when using a CNN backbone).
- The ReLU Extrapolation Trap: Outside the support of the training data, ReLU networks extrapolate as linear functions ($f(x) \propto x$). Similarly, the standard deviation under the empirical NTK scales linearly far from the data ($\sigma(x) \propto x$). In an unbounded domain, acquisition functions like UCB ($\alpha(x) = f(x) + \beta \sigma(x)$) or EI grow unbounded away from the data, causing the optimizer to greedily query points infinitely far outside the data manifold.
- Post-hoc Efficiency: Post-hoc tuning of the prior precision performs comparably to continuous online marginal likelihood optimization during BO, significantly reducing overhead.
Key Results
- High-Dimensional Performance: On the 784-dimensional MNIST task, standard GP-RBF fails, performing no better than random search. In contrast, both MLP- and CNN-based LLA rapidly locate low-objective solutions. LLA with a LeNet-5 CNN backbone was substantially less noisy and more reliable than the MLP surrogate.
- Low-Dimensional Baselines: On 2D Branin and Ackley benchmarks, LLA is competitive with or slightly outperforms the tuned GP-RBF baseline.
- Memory & Scalability: Exact GP-RBF ran out of memory after approximately 800 iterations, whereas LLA easily supported mini-batching and scalable inference.
- Neural Testbed Surprise: In joint predictive distribution evaluations on the Neural Testbed, LLA achieved comparable accuracy (~91.5%) to standard MLPs and ensembles, but exhibited notably worse normalized KL divergence (0.567–0.643 vs. ~0.381 for ensembles), highlighting that out-of-the-box BNN approximations still struggle with calibrated joint posteriors.
Limitations and Future Work
- Domain Bounding Requirement: As a "hotfix" for the ReLU extrapolation issue, practitioners must enforce strict bounding boxes on $x$ and initialize training points uniformly across the interior.
- Architectural Fixes: Future work should explore activations that induce stationary covariance behavior in the NTK (such as periodic activation functions) to prevent linear divergence outside the data region.
- Constrained Acquisition Design: Designing trust-region or proximal acquisition functions that restrict evaluation to regions where the network's local linearization remains valid.
Summary updated