Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods

Saksham Kiroriwal, Julius Pfrommer, Jürgen BeyererarXiv 2026 · arXiv

TL;DR

This paper reinterprets Bayesian Optimization (BO) through the lens of information geometry, showing that the difficulty of high-dimensional optimization often stems from "trace collapse"—where the surrogate model becomes locally insensitive to input changes. The authors introduce FITR (Fisher-Information Trust Region), a novel method that shapes search regions based on local predictive sensitivity rather than just fixed kernel parameters, enabling more robust optimization across diverse surrogate models.

What Problem Does This Paper Address?

Bayesian Optimization is the gold standard for optimizing expensive black-box functions, but it often hits a wall in high dimensions. As the number of input variables increases, the acquisition functions (the "utility" functions we maximize to find the next point to sample) frequently suffer from vanishing gradients. This means the optimizer gets "stuck" in flat regions where it can't figure out which way to move.

Existing solutions, like the popular TuRBO algorithm, use "trust regions" to focus the search. However, these methods typically rely on specific Gaussian Process (GP) kernels with explicit lengthscales (like Squared Exponential or Matérn). This creates a gap: how do we handle high-dimensional BO when using more complex surrogates like Bayesian Neural Networks (BNNs) that don't have simple lengthscale parameters?

Methods

The authors bridge this gap using Information Geometry. They treat the surrogate model as a map from the input space to a "statistical manifold" of probability distributions.

  1. Pullback Fisher Tensor: They "pull back" the Fisher Information Metric—a measure of how much a probability distribution changes—onto the input space. This creates a local sensitivity tensor ($g_t(x)$) that tells you exactly how much a small move in $x$ will change the model's prediction.
  2. Gradient Bounds: They prove a new theoretical bound showing that the gradient of the acquisition function is limited by the trace of this pullback tensor. If the trace collapses to zero (which happens easily in high dimensions far from data), the gradient vanishes, and BO fails.
  3. FITR Algorithm: Building on this, they propose the Fisher-Information Trust Region (FITR). Instead of using fixed kernel lengthscales to size the trust region, FITR uses the local Fisher weights. This makes the trust region "geometry-aware," stretching in directions where the model is less sensitive and shrinking where it is highly sensitive.
Figure 1: First: Conventional BO view with a GP surrogate (SE kernel) and LogEI [1] acquisition function (AF). Second: Visualization of Proposition 4.2, where the acquisition gradient is upper- bounded by the product of the trace of the pullback Fisher tensor and the sensitivity factor of the AF. Third: Induced geometry on the input space using the pullback Fisher tensor. Fourth: Sensitivity Cα (Section 4) of the AF with respect to the predictive-distribution parameters on the statistical manifold. + denotes x = 0 and × denotes x = 1 in input space (the same markers are used to denote the respective points in the manifold view). Gray boxes indicate regions with no observations where the trace of the pullback Fisher tensor collapses.

Datasets

The authors evaluated FITR against strong baselines (TuRBO, DSP, BOUNCE) on a suite of high-dimensional continuous benchmarks:

  • Control/Robotics: Hopper (33D), Lunarlander (12D), RobotPushing (14D), Swimming (16D), Half Cheetah (102D).
  • Engineering Design: HPA102-1 (32D), MOPTA08 (124D), Rover (60D).
  • Machine Learning Tuning: LassoDNA (180D), SVM (388D).

Key Insights and Ideas

  • Trace Collapse is the Enemy: The paper provides a unified geometric explanation for why HDBO fails. When the input dimension is high, the surrogate map often becomes nearly constant away from observed data, causing the Fisher tensor trace to vanish.
  • Geometry over Euclidean Distance: Standard BO assumes a Euclidean search space. This work argues that the "true" distance between points should be measured by how much the predictive distribution changes between them.
  • Kernel Agnostic: Because FITR relies on the Fisher Information of the predictive distribution, it can be applied to any differentiable surrogate, including BNNs and Infinite Width BNNs (IBNNs), which lack traditional lengthscales.
Gradient analysis shows how the natural gradient (purple) differs from the standard gradient (orange) and how the Fisher ellipses adapt to the local landscape.

Key Results

  • Competitive Performance: On standard GP benchmarks with Squared Exponential kernels, FITR-REI consistently matched or outperformed TuRBO-REI and other state-of-the-art methods like BOUNCE.
  • High-Dimensional Success: In very high-dimensional tasks like SVM (388D) and LassoDNA (180D), FITR showed robust convergence, proving its ability to navigate "flat" acquisition landscapes.
  • Transferability: FITR successfully improved performance when paired with non-lengthscale kernels like IBNNs (Figure 5), where traditional trust-region methods are difficult to apply.
Figure 4: BO performance (lower is better) comparison for different methods with a GP using an SE kernel. Solid curves show means over 11 trials, and shaded bands show standard error. FITR-REI and TuRBO-REI use the same GP modeling pipeline and regional trust-region center selection; they differ only in whether the trust-region shape is determined by ARD lengthscales or by the pullback Fisher tensor. FullFITR-REI was evaluated only on a subset of problems and is discussed separately in the text.

Limitations and Future Work

  • Local vs. Global Geometry: The current implementation uses a diagonal approximation of the Fisher tensor for efficiency. While a "FullFITR" (using the full tensor) was tested, it didn't always perform better because local geometry can change rapidly, making the rotated ellipses less accurate further from the center.
  • Computational Overhead: Calculating the Fisher weights via Monte Carlo sampling adds a modest computational cost per iteration compared to simple lengthscale lookups.
  • Boundary Handling: The authors noted that when trust regions intersect the search space boundaries, the resulting "clipped polytope" is harder to sample from than a simple box, suggesting a need for better boundary-aware sampling techniques.

Summary updated

Abstract

We study Bayesian optimization (BO) through the lens of information geometry. Pulling back the Fisher information metric through the surrogate posterior map yields a local sensitivity tensor on the input space, which leads to an upper bound on the gradient of reparameterizable acquisition functions. This view explains vanishing-gradient behavior in high-dimensional BO and provides a common interpretation of heuristics such as RAASP and dimension-scaled lengthscales. Building on this analysis, we propose FITR, a trust-region-based BO method that replaces lengthscale-based scaling by local pullback-Fisher weights. FITR is not restricted to GP kernels with explicit lengthscales. On GP benchmarks with an SE kernel, experiments show competitive performance using FITR. The proposed method also easily generalizes to non-isotropic surrogates, although the gains are more task-dependent in that setting.
BibTeX
@misc{kiroriwal2026bayesian,
  author = {Saksham Kiroriwal and Julius Pfrommer and Jürgen Beyerer},
  title = {{Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods}},
  year = {2026},
  eprint = {2609.31107},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.31107},
}
Figure