Abstract: We propose an energy-based model (EBM) of protein conformations that operates at atomic scale. The model is trained solely on crystallized protein data. By contrast, existing approaches for scoring conformations use energy functions that incorporate knowledge of physical principles and features that are the complex product of several decades of research and tuning. To evaluate the model, we benchmark on the rotamer recovery task, the problem of predicting the conformation of a side chain from its context within a protein structure, which has been used to evaluate energy functions for protein design. The model achieves performance close to that of the Rosetta energy function, a state-of-the-art method widely used in protein structure prediction and design. An investigation of the model's outputs and hidden representations finds that it captures physicochemical properties relevant to protein energy.
Introduction
In protein design and folding, energy functions are essential for determining the stability of specific atomic arrangements. Historically, these functions have relied on manually engineered "knowledge-based" or "physics-based" potentials, such as the Rosetta energy function. These potentials represent decades of research into van der Waals forces, hydrogen bonding, and solvation effects. This work explores whether an atomic-resolution energy function can be learned directly from experimental data, bypassing manual engineering. We evaluate this approach using the "rotamer recovery" task—predicting the 3D orientation of an amino acid side chain given its local protein context.
Methods: The Atom Transformer
The proposed Atom Transformer is an Energy-Based Model (EBM) designed to operate on sets of atoms. For a target residue, the model processes the 64 nearest-neighbor atoms, representing each by its 3D coordinates and categorical features (element type, amino acid type, and side-chain role).
The architecture utilizes a 6-layer Transformer, which is inherently permutation-invariant and thus suited for modeling atomic sets. The self-attention mechanism captures non-linear, higher-order interactions, such as hydrogen bond networks. The final representations are pooled and passed through a Multi-Layer Perceptron (MLP) to output a scalar energy value. Training is conducted using a maximum likelihood objective with importance sampling from a rotamer library to generate negative samples.
Results: Rotamer Recovery Benchmarks
The model was trained on high-resolution structures from the Protein Data Bank (PDB) and evaluated on a standard benchmark of 129 structures. The Atom Transformer achieved performance remarkably close to the gold-standard Rosetta energy function.
An ensemble of Atom Transformers reached an average recovery rate of 71.5%, compared to Rosetta's 72.2%. When using continuous minimization (rt-min), accuracy improved to 74.1% (vs. Rosetta’s 75.4%). Notably, the model outperformed Rosetta on polar amino acids like Asparagine and Glutamine, though it lagged slightly on bulkier residues like Phenylalanine.
Emergent Physicochemical Properties
Analysis of the model's representations reveals that it "re-discovers" fundamental physical principles without explicit instruction:
- Packing Density: The model learns that residues in the protein's core have steeper energy wells than those on the flexible surface.
- Molecular Symmetry: It correctly identifies the 180-degree rotational symmetry of residues like Tyrosine and Phenylalanine, assigning identical energy to symmetric states.
- Hydrogen Bonding: Saliency maps indicate that the model focuses heavily on atoms involved in backbone-sidechain hydrogen bonds, effectively identifying stabilizing chemical interactions.
Conclusion
The Atom Transformer demonstrates that deep learning architectures can capture the complex, non-additive dependencies of molecular physics directly from experimental data. While currently limited by a fixed-backbone assumption and higher computational costs than pairwise potentials, this research provides a powerful proof-of-concept for data-driven molecular modeling in protein engineering.
Summary updated