Energy-based models for atomic-resolution protein conformations

Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, Alexander Rives

Abstract: We propose an energy-based model (EBM) of protein conformations that operates at atomic scale. The model is trained solely on crystallized protein data. By contrast, existing approaches for scoring conformations use energy functions that incorporate knowledge of physical principles and features that are the complex product of several decades of research and tuning. To evaluate the model, we benchmark on the rotamer recovery task, the problem of predicting the conformation of a side chain from its context within a protein structure, which has been used to evaluate energy functions for protein design. The model achieves performance close to that of the Rosetta energy function, a state-of-the-art method widely used in protein structure prediction and design. An investigation of the model's outputs and hidden representations finds that it captures physicochemical properties relevant to protein energy.

Introduction

In protein design and folding, energy functions are essential for determining the stability of specific atomic arrangements. Historically, these functions have relied on manually engineered "knowledge-based" or "physics-based" potentials, such as the Rosetta energy function. These potentials represent decades of research into van der Waals forces, hydrogen bonding, and solvation effects. This work explores whether an atomic-resolution energy function can be learned directly from experimental data, bypassing manual engineering. We evaluate this approach using the "rotamer recovery" task—predicting the 3D orientation of an amino acid side chain given its local protein context.

Methods: The Atom Transformer

The proposed Atom Transformer is an Energy-Based Model (EBM) designed to operate on sets of atoms. For a target residue, the model processes the 64 nearest-neighbor atoms, representing each by its 3D coordinates and categorical features (element type, amino acid type, and side-chain role).

The Atom Transformer pipeline: from atomic coordinates and features to energy prediction.

The architecture utilizes a 6-layer Transformer, which is inherently permutation-invariant and thus suited for modeling atomic sets. The self-attention mechanism captures non-linear, higher-order interactions, such as hydrogen bond networks. The final representations are pooled and passed through a Multi-Layer Perceptron (MLP) to output a scalar energy value. Training is conducted using a maximum likelihood objective with importance sampling from a rotamer library to generate negative samples.

Results: Rotamer Recovery Benchmarks

The model was trained on high-resolution structures from the Protein Data Bank (PDB) and evaluated on a standard benchmark of 129 structures. The Atom Transformer achieved performance remarkably close to the gold-standard Rosetta energy function.

Rotamer recovery performance of the Atom Transformer compared to Rosetta baselines.

An ensemble of Atom Transformers reached an average recovery rate of 71.5%, compared to Rosetta's 72.2%. When using continuous minimization (rt-min), accuracy improved to 74.1% (vs. Rosetta’s 75.4%). Notably, the model outperformed Rosetta on polar amino acids like Asparagine and Glutamine, though it lagged slightly on bulkier residues like Phenylalanine.

Emergent Physicochemical Properties

Analysis of the model's representations reveals that it "re-discovers" fundamental physical principles without explicit instruction:

  • Packing Density: The model learns that residues in the protein's core have steeper energy wells than those on the flexible surface.
  • Molecular Symmetry: It correctly identifies the 180-degree rotational symmetry of residues like Tyrosine and Phenylalanine, assigning identical energy to symmetric states.
  • Hydrogen Bonding: Saliency maps indicate that the model focuses heavily on atoms involved in backbone-sidechain hydrogen bonds, effectively identifying stabilizing chemical interactions.

Conclusion

The Atom Transformer demonstrates that deep learning architectures can capture the complex, non-additive dependencies of molecular physics directly from experimental data. While currently limited by a fixed-backbone assumption and higher computational costs than pairwise potentials, this research provides a powerful proof-of-concept for data-driven molecular modeling in protein engineering.

Summary updated

Abstract

We propose an energy-based model (EBM) of protein conformations that operates at atomic scale. The model is trained solely on crystallized protein data. By contrast, existing approaches for scoring conformations use energy functions that incorporate knowledge of physical principles and features that are the complex product of several decades of research and tuning. To evaluate the model, we benchmark on the rotamer recovery task, the problem of predicting the conformation of a side chain from its context within a protein structure, which has been used to evaluate energy functions for protein design. The model achieves performance close to that of the Rosetta energy function, a state-of-the-art method widely used in protein structure prediction and design. An investigation of the model's outputs and hidden representations finds that it captures physicochemical properties relevant to protein energy.
BibTeX
@inproceedings{du2020energy,
  author = {Yilun Du and Joshua Meier and Jerry Ma and Rob Fergus and Alexander Rives},
  title = {{Energy-based models for atomic-resolution protein conformations}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=TTupsr222um},
}
Figure