TL;DR
AI-driven protein structure prediction models can still produce unreliable or physically implausible geometries, requiring expensive external "oracles" (like molecular dynamics simulations) to validate and correct them. This paper provides the first systematic benchmark of oracle-budget-aware guidance methods, demonstrating that the newly adapted Optimisation Over Outputs (O3) framework excels at low budgets, while steering and fine-tuning methods perform better as the budget increases.
What Problem Does This Paper Address?
State-of-the-art protein structure foundation models like AlphaFold3 and Boltz-2 have revolutionized biology, but they aren't perfect. They can collapse onto a single conformation when multiple are relevant, output physically impossible geometries, or misassemble large complexes.
To fix these errors, researchers use external "oracles"—computational tools or wet-lab assays that score a model's prediction based on physical plausibility or binding energy. However, biological oracles are incredibly expensive; a single molecular dynamics simulation can take days of GPU compute. This creates a strict oracle budget ($N$). While several methods exist to guide generative models using oracle feedback, we haven't known which method yields the best protein structures for a given budget. This paper fills that gap.
Methods
The authors benchmark four distinct guidance strategies applied to the open-source Boltz-2 model:
- Best $K$-of-$N$ Sampling (Baseline): Generate $N$ random samples, score them all, and return the top $K$.
- Feynman-Kac Steering (FK-Steering): An inference-time approach that samples multiple "particles" through the diffusion denoising process, adaptively resampling them at intermediate steps based on oracle scores.
- Direct Preference Optimisation (DPO): A fine-tuning method that updates the actual weights of the foundation model using pairs of preferred (high-scoring) and rejected (low-scoring) structures. The authors test both offline (fixed dataset) and online (iterative on-policy sampling) variants.
- Optimisation Over Outputs (O3): Originally proposed for text/image generation, the authors adapt O3 to protein structure prediction for the first time. O3 takes a small set of initial high-scoring generations ("seeds") to map out a low-dimensional subspace within the model's latent space. It then runs Bayesian Optimisation (BO) over this compact subspace to efficiently find high-quality structures.
Datasets
The methods were evaluated on two distinct protein targets using different oracles:
- Calmodulin (PDB: 1CLL): A well-characterized protein evaluated using TM-score, a metric that measures structural similarity against a known ground-truth crystal structure.
- E. coli aspartate transcarbamoylase (PDB: 9EEH): A much larger, out-of-distribution 7,232-atom complex evaluated using MolProbity, a reference-free oracle that checks for physical violations like atomic clashes and bad bond angles.
Key Insights and Ideas
- No Single Winner: The optimal guidance strategy depends entirely on your oracle budget.
- O3 Dominates Low Budgets: When oracle queries are severely limited ($N \le 100$), O3 is the only method that meaningfully outperforms the simple Best $K$-of-$N$ baseline. By restricting optimization to a low-dimensional subspace defined by good initial examples, it avoids wasting budget exploring bad regions of the massive latent space.
- Go Online for DPO: For fine-tuning with DPO, an online approach (iteratively generating data from the updating model) is absolutely critical. Offline DPO performance completely plateaus, whereas online DPO improves steadily with larger budgets.
- Oracle Type Matters for Steering: FK-steering relies on scoring intermediate, noisy steps of the diffusion process. This works well for global metrics like TM-score, but fails for local geometric metrics like MolProbity, where residual noise dominates and makes intermediate scores uninformative.
Key Results
On the Calmodulin target, the quantitative trade-offs across budgets become highly apparent:
- At a low budget of $N=20$, O3 achieves a mean TM-score of ~0.70, drastically outperforming FK-steering (~0.55) and DPO (~0.58).
- As the budget scales to $N=1000$, O3 plateaus at a high mean TM-score of ~0.81. Meanwhile, FK-steering (~0.73) and online DPO (~0.71) show strong upward trajectories, closing the gap.
Limitations and Future Work
A key limitation of the highly effective O3 framework is that it requires a deterministic generative formulation (a probability-flow ODE) rather than stochastic sampling. This can inherently reduce the structural diversity of the generated batch, which might be a drawback if a practitioner needs to discover entirely different protein conformations.
Additionally, the authors note that while DPO requires a large budget to train, it amortizes that cost into the model weights. Once trained, generating subsequent batches is cheap, whereas O3 and FK-steering must burn through the oracle budget anew for every new target. Future work will focus on expanding these benchmarks to more diverse protein targets and even more complex, non-differentiable biological oracles.
Summary updated