Abstract
The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. It is theoretically compelling since it can be seen as a Gaussian process posterior with the mean function given by the neural network's maximum-a-posteriori predictive function and the covariance function induced by the empirical neural tangent kernel. However, while its efficacy has been studied in large-scale tasks like image classification, it has not been studied in sequential decision-making problems like Bayesian optimization where Gaussian processes -- with simple mean functions and kernels such as the radial basis function -- are the de-facto surrogate models. In this work, we study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility. However, we also present some pitfalls that might arise and a potential problem with the LLA when the search space is unbounded.
Introduction: The Challenge of High-Dimensional Optimization
Bayesian Optimization (BO) is a powerful framework for optimizing "expensive" black-box functions, such as hyperparameter tuning or molecular design. Traditionally, Gaussian Processes (GPs) with simple kernels, like the Radial Basis Function (RBF), have been the surrogate models of choice. However, standard GPs face significant hurdles in high-dimensional spaces and lack the inductive biases necessary to exploit structured data, such as images. While Bayesian Neural Networks (BNNs) offer a potential solution by combining the flexibility of deep learning with uncertainty estimation, they are notoriously difficult to train and calibrate.
This research investigates the Linearized Laplace Approximation (LLA) as a bridge between these two worlds. By linearizing a neural network around its Maximum-a-Posteriori (MAP) weights, LLA creates a surrogate model that behaves like a GP with a kernel defined by the Neural Tangent Kernel (NTK). This approach allows BO to leverage the architectural strengths of neural networks while maintaining a principled Bayesian framework.
Methodology: Turning Networks into GPs
The LLA process involves three primary steps. First, a standard neural network is trained to find the MAP weights. Second, the network is linearized using a first-order Taylor expansion around these weights. Finally, this linearized model is interpreted as a GP where the mean is the network's prediction and the covariance is determined by the NTK.
One key practical consideration explored is hyperparameter tuning. The authors compare "Online" tuning (re-tuning hyperparameters throughout the BO process) with "Post-hoc" tuning (tuning only once after initial training). They find that post-hoc tuning is remarkably effective and significantly more computationally efficient, making LLA a practical "drop-in" replacement for standard GPs in many workflows.
Performance and the Power of Inductive Bias
The primary advantage of LLA is its ability to inherit the inductive biases of the underlying neural network architecture. This is most evident in high-dimensional tasks. In experiments on the 784-dimensional MNIST image generation task, LLA-based models—particularly those using Convolutional Neural Network (CNN) architectures—dramatically outperformed standard GP-RBF models.
As shown in Figure 1, while standard GPs struggle as dimensionality increases, the LLA-CNN "understands" the spatial structure of the image data, finding the objective minimum in just a few iterations. Furthermore, LLA models proved more memory-efficient than exact GPs, which often fail as the number of observations grows due to the cubic scaling of GP inference.
The Extrapolation Pitfall in Unbounded Spaces
Despite its successes, the research identifies a critical "extrapolation pathology" inherent to LLA models using ReLU activations. Because ReLU networks are piecewise linear, their predictions tend to grow or shrink linearly and infinitely when moving far away from the training data. In an unbounded search space, standard acquisition functions like Expected Improvement (EI) or Upper Confidence Bound (UCB) will perceive this infinite growth as a signal to explore further and further away from the data.
Figure 4 illustrates this phenomenon. Unlike the GP-RBF, which reverts to its prior mean and uncertainty far from the data, the LLA-ReLU model's mean and uncertainty continue to expand. This leads the acquisition function to suggest sampling points at "infinity," effectively breaking the optimization loop unless the search space is strictly bounded.
Conclusion and Future Directions
The Linearized Laplace Approximation represents a significant step forward for high-dimensional Bayesian Optimization, offering a scalable way to incorporate deep learning inductive biases into a probabilistic framework. However, the identified extrapolation issues with ReLU architectures suggest that practitioners must either strictly bound their search spaces or explore alternative activation functions (such as periodic functions) that exhibit more stable asymptotic behavior. Future work will likely focus on refining these architectures and developing acquisition functions specifically designed to handle the unique uncertainty profiles of linearized neural networks.
Summary updated