How to Spend Your Oracle Budget: Practical Guidance for Protein Structure Prediction Models

Aleksandra Kalisz, Jack Simons, Krisztina Sinkovics, Noam Ghenassia, Shikha Surana, Henry Moss, Paul Duckworth

TL;DR

AI-driven protein structure prediction models can still produce unreliable or physically implausible geometries, requiring expensive external "oracles" (like molecular dynamics simulations) to validate and correct them. This paper provides the first systematic benchmark of oracle-budget-aware guidance methods, demonstrating that the newly adapted Optimisation Over Outputs (O3) framework excels at low budgets, while steering and fine-tuning methods perform better as the budget increases.

What Problem Does This Paper Address?

State-of-the-art protein structure foundation models like AlphaFold3 and Boltz-2 have revolutionized biology, but they aren't perfect. They can collapse onto a single conformation when multiple are relevant, output physically impossible geometries, or misassemble large complexes.

To fix these errors, researchers use external "oracles"—computational tools or wet-lab assays that score a model's prediction based on physical plausibility or binding energy. However, biological oracles are incredibly expensive; a single molecular dynamics simulation can take days of GPU compute. This creates a strict oracle budget ($N$). While several methods exist to guide generative models using oracle feedback, we haven't known which method yields the best protein structures for a given budget. This paper fills that gap.

Methods

The authors benchmark four distinct guidance strategies applied to the open-source Boltz-2 model:

  1. Best $K$-of-$N$ Sampling (Baseline): Generate $N$ random samples, score them all, and return the top $K$.
  2. Feynman-Kac Steering (FK-Steering): An inference-time approach that samples multiple "particles" through the diffusion denoising process, adaptively resampling them at intermediate steps based on oracle scores.
  3. Direct Preference Optimisation (DPO): A fine-tuning method that updates the actual weights of the foundation model using pairs of preferred (high-scoring) and rejected (low-scoring) structures. The authors test both offline (fixed dataset) and online (iterative on-policy sampling) variants.
  4. Optimisation Over Outputs (O3): Originally proposed for text/image generation, the authors adapt O3 to protein structure prediction for the first time. O3 takes a small set of initial high-scoring generations ("seeds") to map out a low-dimensional subspace within the model's latent space. It then runs Bayesian Optimisation (BO) over this compact subspace to efficiently find high-quality structures.
Figure 1. Bayesian Optimisation via O3 in a Boltz-2. d = 3 surrogate subspace created from three Calmodulin seed samples (top row), showing the 2D U space (bottom row), the GP training data (white dots) and mean predictive posterior over U, the max acquisition point (red cross) for two acquisition rounds, and a 25x25 exhaustive ground-truth grid directly evaluating the oracle (bottom right).

Datasets

The methods were evaluated on two distinct protein targets using different oracles:

  • Calmodulin (PDB: 1CLL): A well-characterized protein evaluated using TM-score, a metric that measures structural similarity against a known ground-truth crystal structure.
  • E. coli aspartate transcarbamoylase (PDB: 9EEH): A much larger, out-of-distribution 7,232-atom complex evaluated using MolProbity, a reference-free oracle that checks for physical violations like atomic clashes and bad bond angles.

Key Insights and Ideas

  • No Single Winner: The optimal guidance strategy depends entirely on your oracle budget.
  • O3 Dominates Low Budgets: When oracle queries are severely limited ($N \le 100$), O3 is the only method that meaningfully outperforms the simple Best $K$-of-$N$ baseline. By restricting optimization to a low-dimensional subspace defined by good initial examples, it avoids wasting budget exploring bad regions of the massive latent space.
  • Go Online for DPO: For fine-tuning with DPO, an online approach (iteratively generating data from the updating model) is absolutely critical. Offline DPO performance completely plateaus, whereas online DPO improves steadily with larger budgets.
Figure 7. Online vs. offline DPO across oracle budgets. Mean- of-K TM-score achieved by online and offline DPO on 1CLL, for oracle budgets N ∈{100, 500, 1000}. Bars are means over 5 random seeds; error bars show ±1 std. Online DPO improves consistently with N, while offline DPO remains flat across budgets.
  • Oracle Type Matters for Steering: FK-steering relies on scoring intermediate, noisy steps of the diffusion process. This works well for global metrics like TM-score, but fails for local geometric metrics like MolProbity, where residual noise dominates and makes intermediate scores uninformative.

Key Results

On the Calmodulin target, the quantitative trade-offs across budgets become highly apparent:

Figure 2. Mean-of-K TM-score across oracle budgets. Comparison of FK-steering, DPO (in the online setting), Best K-of-N, and O3 on 1CLL across the six (N, K) configurations of Section 4. Bars are the mean over 5 random seeds; error bars are ±1 std. Higher is better. O3 method dominates at budgets evaluated, while FK-steering and DPO improve steadily with N.
  • At a low budget of $N=20$, O3 achieves a mean TM-score of ~0.70, drastically outperforming FK-steering (~0.55) and DPO (~0.58).
  • As the budget scales to $N=1000$, O3 plateaus at a high mean TM-score of ~0.81. Meanwhile, FK-steering (~0.73) and online DPO (~0.71) show strong upward trajectories, closing the gap.

Limitations and Future Work

A key limitation of the highly effective O3 framework is that it requires a deterministic generative formulation (a probability-flow ODE) rather than stochastic sampling. This can inherently reduce the structural diversity of the generated batch, which might be a drawback if a practitioner needs to discover entirely different protein conformations.

Additionally, the authors note that while DPO requires a large budget to train, it amortizes that cost into the model weights. Once trained, generating subsequent batches is cheap, whereas O3 and FK-steering must burn through the oracle budget anew for every new target. Future work will focus on expanding these benchmarks to more diverse protein targets and even more complex, non-differentiable biological oracles.

Summary updated

Abstract

Foundation models for protein structure prediction remain unreliable on certain targets. External oracles can flag and correct these failures, but biological oracles are expensive, making oracle budget a critical constraint. Existing guidance methods, such as FK-steering, DPO, and Best K-of-N sampling, differ in how they spend this budget, yet no systematic comparison exists to guide method selection. To bridge this gap, we benchmark these methods alongside the recently proposed Optimisation Over Outputs (O3), which applies off-the-shelf optimisers within a generative model's latent subspace. We extend the usage of O3 to protein structure prediction models. Overall, our work provides the first practical reference for oracle budget-aware guidance. Our evaluation on two protein targets, calmodulin (1CLL) and E. coli aspartate transcarbamoylase (9EEH), reveals that no single method consistently dominates across all budgets and oracles. Specifically, O3 proves most effective at low oracle budgets, while FK-steering and DPO demonstrate improved performance as the budget increases. We distil these findings into actionable recommendations for practitioners operating under real-world oracle-budget constraints.
BibTeX
@inproceedings{kalisz2026how,
  author = {Aleksandra Kalisz and Jack Simons and Krisztina Sinkovics and Noam Ghenassia and Shikha Surana and Henry Moss and Paul Duckworth},
  title = {{How to Spend Your Oracle Budget: Practical Guidance for Protein Structure Prediction Models}},
  booktitle = {SPIGM @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=9De56syVZF},
}
Figure