Structure-Based Experimental Datasets for Benchmarking Protein Simulation Force Fields

Chapin E. Cavender, David A. Case, Julian C. -h. Chen, Lillian T. Chong, D.a. Keedy, Kresten Lindorff‐Larsen, David L. Mobley, O. H. Samuli Ollila, Chris Oostenbrink, Paul Robustelli, Vincent A. Voelz, Michael E. Wall, David C. Wych, Michael K. GilsonarXiv 2023 · DOI · arXiv

TL;DR

This paper provides a curated "gold standard" roadmap of experimental datasets—primarily from NMR and room-temperature crystallography—designed to rigorously test the accuracy of protein molecular dynamics (MD) simulations. By categorizing observables based on their experimental uncertainty and simulation convergence speed, the authors offer a strategic framework for developers to refine the force fields that power modern drug discovery and structural biology.

What Problem Does This Paper Address?

Molecular dynamics (MD) simulations are the "microscopes" of the computational world, allowing researchers to watch proteins fold, bind ligands, and change shape in atomic detail. However, these simulations are only as good as their force fields—the mathematical models that describe how atoms interact.

Historically, force fields were tested against static, "frozen" protein structures. The problem? Proteins in a living cell are dynamic, jiggling ensembles, not statues. Furthermore, comparing a simulation to a single experimental structure is often an apples-to-oranges comparison. There has been a long-standing need for a consensus on which experimental datasets are reliable enough to serve as "ground truth" and how, exactly, we should compare simulation "snapshots" to messy, time-averaged experimental data.

Methods

This article is a comprehensive review and recommendation piece. Rather than introducing a new algorithm, it establishes a tiered benchmarking strategy.

The authors evaluate various experimental "observables" (things we can actually measure) and categorize them into two tiers:

  • Tier 1 (The Essentials): Observables like NMR chemical shifts, scalar ($J$) couplings, and residual dipolar couplings (RDCs). These are recommended because they have low experimental uncertainty, converge quickly in simulations, and have well-established "forward models" (mathematical functions that translate atomic coordinates into predicted experimental values).
  • Tier 2 (The Advanced): Observables like NOEs (distance constraints), B-factors (atomic mobility), and diffuse scattering. These provide deep insight into protein "breathing" and tertiary structure but are harder to use because they require massive amounts of simulation time to reach a stable average or have higher experimental noise.

Datasets

The paper identifies several high-quality datasets that cover the three pillars of protein states: short peptides, folded proteins, and intrinsically disordered proteins (IDPs).

  • NMR Benchmarks: Includes the Beauchamp short peptides, the Robustelli a99SB-disp set (which famously combines folded and disordered proteins), and the Stroet collection of 52 high-resolution protein models.
  • Crystallography Benchmarks: A major focus is placed on Room-Temperature (RT) Crystallography (e.g., Cyclophilin A, PTP1B, and Hen Egg-White Lysozyme). Unlike traditional cryogenic structures, RT data captures the natural conformational fluctuations that MD simulations aim to replicate.
  • Specialized Sets: Datasets for specific phenomena like salt-bridge stability and spin relaxation rates are also highlighted for fine-tuning kinetic properties.

Key Insights and Ideas

  • The "Direct Comparison" Fallacy: You cannot simply look at a simulation and an experimental structure and ask if they "match." You must use a forward model to calculate what the experiment would have seen if the simulation were reality. The paper emphasizes that the accuracy of the benchmark is limited by the accuracy of these mapping models (like SHIFTX2 for chemical shifts).
  • Ensemble Thinking: Experimental measurements are almost always averages over billions of molecules and long time scales. Therefore, force fields shouldn't be judged on a single "best" structure, but on whether their Boltzmann-weighted ensemble of shapes matches the experimental average.
  • The Disordered Advantage: Counter-intuitively, it is often easier to benchmark force fields using disordered proteins. Because they lack stable tertiary structures, their NMR signals are dominated by local backbone preferences, making them excellent for isolating errors in basic torsional parameters.
  • RT-Crystallography is the Future: The authors argue that RT-crystallography is a superior benchmark to cryogenic data because it avoids "freezing out" the very dynamics that MD researchers are trying to study.

Key Results

The paper culminates in a definitive "Cheat Sheet" (Table 2) that ranks observables by four criteria: experimental uncertainty, simulation convergence, structural information, and literature consensus.

Quantitatively, the authors highlight how modern force fields (like ff14SB or a99SB-disp) show markedly better agreement with RT-crystallography B-factors and NMR RDCs than older generations, proving that rigorous benchmarking leads to measurable improvements in simulation predictive power.

Limitations and Future Work

  • The Water Problem: A protein simulation is only as good as its water model. The authors acknowledge that many force field failures are actually "water failures" (e.g., incorrect viscosity or density), which can be hard to decouple from protein-specific errors.
  • Computational Cost: Some of the most valuable benchmarks—like simulating an entire crystal supercell to compare against Bragg diffraction—are still computationally expensive, limiting their use to well-funded labs.
  • Missing Consensus: For several promising observables, like diffuse scattering and Bragg intensities, there is still no community-wide agreement on the best statistical methods for comparison. The authors call for more "blind challenges" to standardize these metrics.

Summary updated

Abstract

This review article provides an overview of structurally oriented experimental datasets that can be used to benchmark protein force fields, focusing on data generated by nuclear magnetic resonance (NMR) spectroscopy and room temperature (RT) protein crystallography. We discuss what the observables are, what they tell us about structure and dynamics, what makes them useful for assessing force field accuracy, and how they can be connected to molecular dynamics simulations carried out using the force field one wishes to benchmark. We also touch on statistical issues that arise when comparing simulations with experiment. We hope this article will be particularly useful to computational researchers and trainees who develop, benchmark, or use protein force fields for molecular simulations.
BibTeX
@misc{cavender2023structure,
  author = {Chapin E. Cavender and David A. Case and Julian C. -H. Chen and Lillian T. Chong and D.A. Keedy and Kresten Lindorff‐Larsen and David L. Mobley and O. H. Samuli Ollila and Chris Oostenbrink and Paul Robustelli and Vincent A. Voelz and Michael E. Wall and David C. Wych and Michael K. Gilson},
  title = {{Structure-Based Experimental Datasets for Benchmarking Protein Simulation Force Fields}},
  year = {2023},
  eprint = {2303.11056},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.11056},
}
Figure