In the world of drug discovery and protein design, a "budget" isn't just a number on a spreadsheet—it’s the number of physical experiments a wet-lab can perform in a single round. While many machine learning benchmarks for high-dimensional Bayesian Optimization (BO) assume hundreds or thousands of evaluations, a realistic lab cycle might only allow for 30.
A new paper from researchers at the Czech Technical University and the Czech Academy of Sciences, titled "Match the Geometry Before the Surrogate," argues that in this extreme low-budget regime, we’ve been focusing on the wrong thing. It’s not the complexity of your surrogate model (like a Gaussian Process) that matters most; it’s the geometry of where you choose to look.
TL;DR
In extreme low-budget optimization ($N=30$), the shape of the search space matters more than the surrogate model. By restricting candidate generation to the "spherical shell" where VAE encoder means actually concentrate, simple search strategies can match or outperform sophisticated Bayesian Optimization methods.
What Problem Does This Paper Address?
Molecular optimization often happens in the "latent space" of a Variational Autoencoder (VAE). We map discrete molecules to continuous vectors, use BO to find "better" vectors, and decode them back into new molecules.
The authors identify a critical mismatch: most BO methods propose new candidates within axis-aligned boxes or trust regions. However, in high-dimensional VAEs with Gaussian priors, the encoded "means" of real molecules don't fill a box—they concentrate on a narrow spherical shell. If your optimizer is looking in the corners of a high-dimensional box, it’s searching in "empty space" where the decoder has never seen data, leading to poor molecular generation.
Methods
The authors introduce the on-sphere scaffold, a constraint that forces the optimizer to only propose candidates on the empirical shell occupied by known molecules. To test the power of this geometric constraint, they developed two "minimalist" optimizers:
- GEOWALK: A surrogate-free baseline. It performs uniform exploration on the shell, followed by a "geodesic local search" (a random walk that stays on the sphere) starting from the best point found.
- GEOCMA: Adapts the well-known CMA-ES algorithm to work within a random low-dimensional subspace, while still projecting candidates onto the spherical shell.
They also conducted a factorial experiment using a standard Gaussian Process (GP) to isolate two variables: the support (searching in a box vs. on the shell) and the acquisition maximizer (picking the best from a random set vs. using a gradient-based continuous optimizer like L-BFGS-B).
Datasets
- GuacaMol: A standard benchmark for goal-directed molecular generation (11 tasks used).
- ZINC-250k: A reference set of 250,000 molecules used to characterize the VAE's latent geometry.
- LassoBench: A high-dimensional sparse regression benchmark.
- NAS-Bench-101: A neural architecture search benchmark.
Key Insights and Ideas
- The "Shell" is Real: In a 256-dimensional VAE, the typical norm of a sample from the prior is $\approx 16$, but the actual encoded molecules have a mean norm of $\approx 8$. They live in a thin, mid-radius shell.
- Surrogates are Superfluous (at $N=30$): Surprisingly, after 30 evaluations, the surrogate-free GEOWALK performed just as well as a GP-based optimizer on the shell. The GP is helpful early on (around 10-20 evals), but the geometric constraint does most of the heavy lifting by the end of the budget.
- The Interaction Effect: The benefit of searching on the shell is most pronounced when using "candidate-set maximization" (scoring a bunch of random points and picking the best). If you use a continuous optimizer like L-BFGS-B, it can sometimes "climb" out of the box and find the shell anyway, making the explicit constraint less vital.
Key Results
The authors found that GEOWALK, despite its simplicity, outperformed several established high-dimensional BO methods that search in boxes, such as TuRBO and BAxUS.
In their factorial experiment (Figure 3), they demonstrated that the "Empirical shell" support consistently yielded higher scores than "Box" support when using candidate-set argmax.
Limitations and Future Work
- Scope of VAEs: The results are specific to Gaussian-prior VAEs. The authors note that other architectures (like Chembed) might not exhibit the same "shell" advantage.
- Budget Scaling: The "surrogate-free" advantage likely disappears as the budget increases toward 100 or 1,000 evaluations, where the GP has enough data to build a truly superior model.
- Practical Lesson: The authors conclude with a warning: before claiming a new surrogate model is "better," researchers must control for the candidate support and the acquisition maximizer. A gain attributed to a fancy new GP might actually just be the result of accidentally looking in a better part of the geometry.
For engineers and researchers, the message is clear: if you're working with a tiny experimental budget, stop tweaking your kernel hyperparameters and start looking at the "norm" of your data. You might find that your optimizer is simply looking in the wrong neighborhood.
Summary updated