TL;DR
This paper reinterprets Bayesian Optimization (BO) through the lens of information geometry, showing that the difficulty of high-dimensional optimization often stems from "trace collapse"—where the surrogate model becomes locally insensitive to input changes. The authors introduce FITR (Fisher-Information Trust Region), a novel method that shapes search regions based on local predictive sensitivity rather than just fixed kernel parameters, enabling more robust optimization across diverse surrogate models.
What Problem Does This Paper Address?
Bayesian Optimization is the gold standard for optimizing expensive black-box functions, but it often hits a wall in high dimensions. As the number of input variables increases, the acquisition functions (the "utility" functions we maximize to find the next point to sample) frequently suffer from vanishing gradients. This means the optimizer gets "stuck" in flat regions where it can't figure out which way to move.
Existing solutions, like the popular TuRBO algorithm, use "trust regions" to focus the search. However, these methods typically rely on specific Gaussian Process (GP) kernels with explicit lengthscales (like Squared Exponential or Matérn). This creates a gap: how do we handle high-dimensional BO when using more complex surrogates like Bayesian Neural Networks (BNNs) that don't have simple lengthscale parameters?
Methods
The authors bridge this gap using Information Geometry. They treat the surrogate model as a map from the input space to a "statistical manifold" of probability distributions.
- Pullback Fisher Tensor: They "pull back" the Fisher Information Metric—a measure of how much a probability distribution changes—onto the input space. This creates a local sensitivity tensor ($g_t(x)$) that tells you exactly how much a small move in $x$ will change the model's prediction.
- Gradient Bounds: They prove a new theoretical bound showing that the gradient of the acquisition function is limited by the trace of this pullback tensor. If the trace collapses to zero (which happens easily in high dimensions far from data), the gradient vanishes, and BO fails.
- FITR Algorithm: Building on this, they propose the Fisher-Information Trust Region (FITR). Instead of using fixed kernel lengthscales to size the trust region, FITR uses the local Fisher weights. This makes the trust region "geometry-aware," stretching in directions where the model is less sensitive and shrinking where it is highly sensitive.
Datasets
The authors evaluated FITR against strong baselines (TuRBO, DSP, BOUNCE) on a suite of high-dimensional continuous benchmarks:
- Control/Robotics: Hopper (33D), Lunarlander (12D), RobotPushing (14D), Swimming (16D), Half Cheetah (102D).
- Engineering Design: HPA102-1 (32D), MOPTA08 (124D), Rover (60D).
- Machine Learning Tuning: LassoDNA (180D), SVM (388D).
Key Insights and Ideas
- Trace Collapse is the Enemy: The paper provides a unified geometric explanation for why HDBO fails. When the input dimension is high, the surrogate map often becomes nearly constant away from observed data, causing the Fisher tensor trace to vanish.
- Geometry over Euclidean Distance: Standard BO assumes a Euclidean search space. This work argues that the "true" distance between points should be measured by how much the predictive distribution changes between them.
- Kernel Agnostic: Because FITR relies on the Fisher Information of the predictive distribution, it can be applied to any differentiable surrogate, including BNNs and Infinite Width BNNs (IBNNs), which lack traditional lengthscales.
Key Results
- Competitive Performance: On standard GP benchmarks with Squared Exponential kernels, FITR-REI consistently matched or outperformed TuRBO-REI and other state-of-the-art methods like BOUNCE.
- High-Dimensional Success: In very high-dimensional tasks like SVM (388D) and LassoDNA (180D), FITR showed robust convergence, proving its ability to navigate "flat" acquisition landscapes.
- Transferability: FITR successfully improved performance when paired with non-lengthscale kernels like IBNNs (Figure 5), where traditional trust-region methods are difficult to apply.
Limitations and Future Work
- Local vs. Global Geometry: The current implementation uses a diagonal approximation of the Fisher tensor for efficiency. While a "FullFITR" (using the full tensor) was tested, it didn't always perform better because local geometry can change rapidly, making the rotated ellipses less accurate further from the center.
- Computational Overhead: Calculating the Fisher weights via Monte Carlo sampling adds a modest computational cost per iteration compared to simple lengthscale lookups.
- Boundary Handling: The authors noted that when trust regions intersect the search space boundaries, the resulting "clipped polytope" is harder to sample from than a simple box, suggesting a need for better boundary-aware sampling techniques.
Summary updated