Energy-based models for atomic-resolution protein conformations

Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, Alexander Rives

In the world of structural biology, "form follows function." To design new medicines or enzymes, scientists need to understand the energy landscapes that dictate how proteins fold. Traditionally, this required decades of painstaking manual "feature engineering"—hand-crafting mathematical formulas to represent physical forces.

A team of researchers from MIT, Facebook AI Research, and NYU recently challenged this status quo. They proposed a way to skip the manual labor and learn these complex energy functions directly from raw atomic data.

The Challenge of Protein Energy Functions

Designing a protein is essentially an optimization problem: finding a sequence of amino acids that settles into a specific, stable 3D shape. To do this, researchers rely on an energy function—a "fitness" score that evaluates how likely a specific arrangement of atoms is to exist in nature.

For decades, the gold standard has been tools like Rosetta, which use statistical potentials and physical approximations (like Van der Waals forces) meticulously tuned by experts. However, these models often struggle with non-additive interactions, such as complex networks of hydrogen bonds where the total energy isn't just the sum of its parts. The researchers asked: can a deep learning model discover these intricate physical rules automatically, simply by observing examples of stable proteins?

Methods: The Atom Transformer

The researchers framed the protein energy problem as a density estimation task, building an Energy-Based Model (EBM) powered by an Atom Transformer to act as their energy function.

Atom Transformer Architecture
  • The Architecture: The model utilizes a 6-layer Transformer. Unlike its typical application in natural language processing, the "tokens" here are individual atoms. Because Transformers use self-attention, the model can naturally capture complex, multi-body interactions between any group of atoms in a local neighborhood.
  • Input Representation: For a specific amino acid side-chain, the model examines the 64 nearest-neighbor atoms. It ingests their 3D Cartesian coordinates, atom types (C, N, O, S), and amino acid identity to form rich atom features.
  • Training via Contrastive Learning: Training an EBM requires making the energy of "real" configurations low and "fake" ones high. Since calculating the total probability space of all possible atomic positions is intractable, the team used importance sampling. They took real crystal structures as positive examples and sampled alternative "rotamers" (common side-chain orientations) from a known library as negative examples.
  • Rotational Invariance: To ensure the model evaluates the intrinsic structure rather than its arbitrary orientation in space, random rotations were applied to the coordinates during training.

Datasets and Rigorous Evaluation

The model was trained on a curated subset of the CullPDB database, containing roughly 12,500 high-resolution (<1.8 Å) protein crystal structures. It was evaluated on a standard benchmark of 129 structures. Crucially, the researchers applied strict filtering, removing any training proteins that shared more than 25% sequence similarity with the test set. This ensured the model learned generalizable physics rather than simply memorizing similar proteins.

Key Insights: Discovering Physics from Data

Remarkably, the model learned fundamental physicochemical properties without explicit programming:

  • Emergent Physics (Core vs. Surface): The model was never explicitly taught the difference between the hydrophobic core of a protein and its solvent-exposed surface. Yet, the energy landscapes it produced showed much steeper "wells" for buried residues, indicating it independently learned that the protein core is more tightly packed and less tolerant of structural deviation.
  • Learning Symmetry: Certain amino acids, like Tyrosine or Phenylalanine, are chemically symmetric—rotating the end of the side chain 180 degrees results in an identical structure. The model's learned energy function successfully captured this 180° periodicity entirely on its own.

Key Results: Approaching the Gold Standard

The model was benchmarked on the rotamer recovery task: given a fixed protein backbone, can the model correctly predict the natural 3D orientation of the side chains?

Rotamer Recovery Results
  • Competitive Performance: The Atom Transformer ensemble achieved a 71.5% recovery rate, coming remarkably close to the highly optimized Rosetta energy function, which scores between 72.2% and 73.6%.
  • Strength in the Core: The model excelled in the dense protein core, reaching an impressive 89.2% accuracy for buried residues.
  • Architectural Superiority: The Transformer significantly outperformed standard Graph Neural Networks (GNNs) and Fully Connected networks, demonstrating that self-attention is highly effective for modeling the complex spatial dependencies of atomic structures.

Limitations and Future Directions

While this data-driven approach is highly promising, the current model operates with certain constraints:

  1. Fixed Backbone: It predicts side-chain orientations for a rigid "skeleton." In biological reality, the protein backbone is flexible.
  2. Local Context: By restricting its view to the 64 nearest atoms, the model captures local chemistry but may miss critical long-range interactions.
  3. Combinatorial Complexity: The model currently scores single side-chains. The next frontier is scaling this to combinatorial optimization—predicting all side-chains simultaneously.

Ultimately, this research provides compelling evidence that the "expert-driven" era of hand-crafted protein energy functions may soon be augmented, or even replaced, by "data-driven" models capable of learning the intricate rules of atomic interactions directly from nature.

Summary updated

Abstract

We propose an energy-based model (EBM) of protein conformations that operates at atomic scale. The model is trained solely on crystallized protein data. By contrast, existing approaches for scoring conformations use energy functions that incorporate knowledge of physical principles and features that are the complex product of several decades of research and tuning. To evaluate the model, we benchmark on the rotamer recovery task, the problem of predicting the conformation of a side chain from its context within a protein structure, which has been used to evaluate energy functions for protein design. The model achieves performance close to that of the Rosetta energy function, a state-of-the-art method widely used in protein structure prediction and design. An investigation of the model’s outputs and hidden representations finds that it captures physicochemical properties relevant to protein energy.
BibTeX
@inproceedings{du2020energy,
  author = {Yilun Du and Joshua Meier and Jerry Ma and Rob Fergus and Alexander Rives},
  title = {{Energy-based models for atomic-resolution protein conformations}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=TTupsr222um},
}
Figure