When to Trust the Surrogate's Subspace: Signal-Tested Subspaces for High-Dimensional Acquisition Search

Bayesian optimization tunes expensive black-box systems, such as accelerator beams, alloys, or control policies, in a loop. It fits a probabilistic model of the objective (the surrogate) to the points evaluated so far, then picks the next point by maximizing a cheap score called the acquisition function. That score rewards points that look good or that the model is unsure about. Once there are hundreds of input dimensions, maximizing the acquisition function becomes a hard non-convex problem in its own right. A popular fix is to search only inside a small subspace. Some methods choose that subspace at random. Others use the directions in which the surrogate's predictions change fastest. The second option sounds smarter, but a result from language-model training gives reason to doubt it: subspaces picked from the top singular directions of gradients lock onto noise once the true signal fades, and random subspaces then do better.

We propose to test whether this failure happens inside Bayesian optimization, and to fix it with a rule that reads the surrogate's own uncertainty. The surrogate tells us both its best guess of the gradient and how unsure it is about that gradient. We use that information to decide, at every iteration, how many search directions to take from the surrogate and how many to draw at random. In two sentences: data-driven search subspaces in Bayesian optimization may quietly align with directions the model is merely confused about, especially late in a run near the incumbent. A per-iteration test comparing gradient signal to gradient uncertainty can catch this and blend in random directions only when needed.

Core Claim

Three prior lines of work bear on this. GIT-BO builds its acquisition subspace from the eigenvectors of a matrix of outer products of the surrogate mean's input gradients. It uses a tabular transformer surrogate, a roughly 10-dimensional subspace centered at the data centroid, and random candidate evaluation inside it. It reports ranking first against SAASBO, TuRBO and BAxUS on synthetic functions up to 500 dimensions. MS-UCB instead maximizes UCB over many uniformly random low-dimensional affine slices. It proves that this removes a $\sqrt{D}$ factor from the GP-UCB regret bound without assuming the objective has low effective dimension. GoLore supplies the warning that motivates this work. It proves that projecting onto top singular directions of stochastic gradients, as GaLore does, stalls near stationary points because noise dominates the spectrum. Sampling projections uniformly from the Stiefel manifold (the set of matrices with orthonormal columns) restores convergence. GoLore reports BoolQ accuracy of 81.96% against GaLore's 79.79%.

Nobody has asked whether GIT-BO-style subspaces fall into the same trap. The analogue is plausible. Near the incumbent, late in a run, the mean gradient shrinks. Far from data, the mean gradient reflects only the model's prior. In both regimes the leading eigenvectors of the gradient matrix can follow fitting artifacts rather than the objective.

Our central observation is that Bayesian optimization has something language-model training lacks: the surrogate quantifies its own gradient noise. For a Gaussian process surrogate with posterior mean $\mu$, the expected gradient outer product under the posterior over functions $f$ at a point $x$ splits as

$$\mathbb{E}\left[\nabla f(x)\nabla f(x)^\top\right] = \nabla\mu(x)\nabla\mu(x)^\top + \Sigma_\nabla(x),$$

where $\Sigma_\nabla(x)$ is the posterior covariance of the gradient. It is available in closed form, because derivatives of a Gaussian process are jointly Gaussian with it. GIT-BO keeps only the first term. We average both terms over anchor points near the current search center to get a signal matrix $S$ and an uncertainty matrix $U$. We then keep the top eigen-directions $v$ of $S$ only while they pass a generalized-eigenvalue test,

$$\frac{v^\top S v}{v^\top U v} > \tau,$$

with threshold $\tau$. The remaining slots of a fixed $k$-dimensional subspace are filled with directions drawn uniformly from the Stiefel manifold, restricted to the orthogonal complement of the kept directions.

The claim is that this signal-tested subspace matches GIT-BO-style subspaces early in a run, when the signal is strong, and matches random subspaces late, when it is not. If so, it should do at least as well as both across a run, and clearly better on functions whose important directions are rotated rather than aligned with the coordinate axes.

Research Program

The method is built on a Gaussian process surrogate. This avoids the 500-dimension cap of the transformer surrogate in GIT-BO and makes $\Sigma_\nabla$ analytic. For dimensions beyond a few hundred we compute the gradient terms with structured kernels and a small set of anchors (around 32), so the cost scales linearly in $D$ per anchor. The subspace is centered at a trust-region-style center near the incumbent rather than at the data centroid. This change is itself tested, because the file flags incumbent-anchored slicing as prone to local trapping (LINEBO, Dropout).

Inside the $k$-dimensional subspace we maximize a Monte Carlo acquisition function with multi-start sample-path gradients. This follows Maximizing acquisition functions for Bayesian optimization, which showed that gradient-based inner search beats CMA-ES and random search. We supply the inner optimizer; we do not compete with it. We deliberately do not pursue cubic-regularized subspace Newton steps (Stochastic Subspace Cubic Newton) for the inner problem. Once the subspace is about 10-dimensional, the inner problem is cheap, and a reviewer would rightly say that quasi-Newton multi-start already captures most of the available curvature. The open question is which subspace to search, not how to search it.

The analysis part asks whether the GoLore argument transfers. On synthetic functions with known gradients we measure, at each iteration, the cosine between the true gradient at the search center and its projection onto the chosen subspace. A random $k$-dimensional subspace in $D$ dimensions has expected squared alignment $k/D$; that is the "do nothing" level. The trap hypothesis predicts that mean-gradient subspaces start well above $k/D$ and then fall to it, or below it, late in a run. A simple Gaussian-process analogue of GoLore's quadratic counterexample gives a way to check this. The function is a quadratic with a rotated low-rank Hessian, and observation noise is added. We will try to prove that when the norm of $\nabla\mu$ falls below the scale of $\Sigma_\nabla$, the top eigenvector of $S$ becomes uninformative about the descent direction. Under the same conditions, the test above rejects it with high probability.

Baselines

The naive reference is uniform random subspaces of the same dimension $k$, redrawn every iteration, with the same inner optimizer and the same center. This is the GoLore-style choice and the core of MS-UCB's slicing. Beating it means lower simple regret at matched evaluation budgets on functions with real low-dimensional structure, without losing to it on functions without such structure.

The strongest existing method on this question is GIT-BO. We run it with its own TabPFN surrogate and settings on tasks up to 500 dimensions. We also run a GIT-BO-style variant on our Gaussian process: mean-gradient eigenvectors only, centroid-centered. That variant separates the subspace rule from the surrogate. The file contains no reported regret values for GIT-BO, only a first-place ranking, so no reported result was found on Lacuna for GIT-BO regret on these functions. All numbers will come from our reruns. Beating it means statistically lower final regret, over at least 20 seeds, on rotated and late-stage regimes, with no worse regret on axis-aligned embedded functions.

MS-UCB is run with its published slicing scheme at the same $k$. The file gives no exact number for it on Ackley, Levy or Gisette, so no reported result was found on Lacuna for MS-UCB final regret. Full-dimensional multi-start gradient acquisition maximization (Maximizing acquisition functions) is given the same wall-clock budget per iteration as our method. LINEBO uses random and gradient-direction lines. Dimension Dropout uses its Mix filling strategy; its only reported scalar in the file is a 13-dimensional alloy utility of 5.1 against 4.8 for standard Bayesian optimization, which is too low-dimensional to be the main test.

Two ablations isolate the key ingredient. The first is a fixed half-and-half mix of mean-gradient and random directions with no test, which checks whether the uncertainty test matters or whether any mixing suffices. The second keeps the test but changes the center from incumbent to centroid. Every method shares the same evaluation budget, the same initial design, and the same inner-optimizer budget, with tuning done on a held-out function family.

Why not just…?

Why not just use random subspaces, as GoLore and MS-UCB do? This is the real threat. MS-UCB's theory needs no structure, and if the surrogate's gradients rarely carry usable signal in high-dimensional Bayesian optimization, random subspaces will tie with our method at lower cost. The alignment diagnostic settles it. If the mean-gradient alignment never beats $k/D$ even early in a run, we will report that random subspaces suffice and that GIT-BO's gains come from its surrogate, not its subspace. Our method only earns its place on functions with embedded or rotated structure, where the test should keep informative directions that random draws miss with probability near $1-k/D$.

Why not just use GIT-BO's mean-gradient subspace unchanged? GoLore's proof says this choice can stall when signal fades, and nothing in GIT-BO's evaluation measures late-run behavior. That evaluation reports rankings, not regret curves. The decisive comparison is late-stage regret and alignment on rotated quadratics and on Rosenbrock, where gradients near the valley floor are small.

Why not just add a fixed share of random directions to GIT-BO? This may recover most of the benefit, which is why it is an ablation and not a straw man. The argument for an adaptive test is that the right share changes over a run and differs across problems. If a well-tuned fixed share matches the test across all benchmark families, we will say so and recommend the simpler rule.

Why not just run multi-start Adam in the full space? MS-UCB argues that full-space inner search becomes budget-starved as $D$ grows. GIT-BO reports that GP baselines with full-dimensional inner search took hours on 100- to 500-dimensional tasks. We test this directly at matched wall-clock time per iteration rather than assuming it.

Proposed Experiments

The synthetic suite uses Ackley, Levy, Rosenbrock and Rastrigin at $D \in \{50, 100, 500, 1000\}$, in three versions. The first is axis-aligned, which favors Dropout. The second is randomly rotated, which defeats coordinate slicing. The third embeds a 10-dimensional function in the full space via a random rotation, which favors data-driven subspaces. Rotated quadratics with observation noise serve as the controlled trap test. The real tasks are the Gisette ramp-loss problem at $D=5000$ used by MS-UCB, and Walker-2D policy search up to 500 dimensions as in GIT-BO. Metrics are simple regret against evaluations, regret against wall-clock time, and the per-iteration alignment curve described above.

The first decisive check is whether mean-gradient alignment collapses to $k/D$ or below in late iterations on the noisy rotated quadratic, and whether our test's acceptance rate falls at the same moment. If neither happens, the trap does not transfer to Bayesian optimization, and the paper becomes a careful negative result. That result would still be useful, because it would vindicate GIT-BO's design.

The second is whether the signal-tested subspace matches the better of random and mean-gradient subspaces on every function family while beating both on at least the rotated family.

The third is whether the uncertainty term $U$ is what drives the gain, shown by the method beating the fixed-mix ablation on problems where the right mix varies over a run.