Evolutionary-scale prediction of atomic level protein structure with a language model

Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, Alexander RivesbioRxiv 2022 · DOI

TL;DR

ESMFold leverages a massive 15-billion parameter protein language model (ESM-2) to predict atomic-level protein structures directly from a single primary sequence. By bypassing the computationally expensive step of searching for evolutionary relatives, ESMFold achieves up to a 60x speedup over AlphaFold2, enabling the structural characterization of over 617 million metagenomic proteins in just two weeks.

What Problem Does This Paper Address?

While AlphaFold2 revolutionized biology by providing high-accuracy protein structure predictions, it has a significant bottleneck: it relies on Multiple Sequence Alignments (MSAs). To predict a structure, these models must first search massive databases to find similar sequences, a process that can take minutes to hours per protein.

This "search-then-fold" paradigm cannot keep pace with the explosion of genomic data, particularly in metagenomics, where researchers sequence DNA directly from environmental samples (like soil or seawater). We are currently sitting on billions of "dark" protein sequences with no known structure or function. The authors aim to bridge this gap by creating a model that "internalizes" evolutionary information during training, allowing it to fold proteins instantly from a single sequence without an explicit MSA search.

Methods

The researchers developed two core components: ESM-2 and ESMFold.

  1. ESM-2 (The Language Model): This is a Transformer-based model trained on hundreds of millions of protein sequences using a "masked language modeling" objective—essentially, the model learns to guess missing amino acids in a sequence. The authors scaled this model from 8 million to 15 billion parameters, making it the largest protein language model to date.
  2. ESMFold (The Folding Head): To convert the language model's internal "understanding" into 3D coordinates, they added a folding trunk and a structure module (similar to AlphaFold2's architecture but simplified). Crucially, the language model parameters are frozen during this stage; the folding head simply learns to project the LM's representations into 3D space.
The ESMFold architecture simplifies the folding process by replacing the MSA-based Evoformer with a single-sequence Folding Block.

Datasets

  • Training (ESM-2): The UniRef database (UniRef50 and UniRef90 clusters), totaling approximately 65 million unique sequences.
  • Training (ESMFold): Experimentally determined structures from the Protein Data Bank (PDB), augmented with 12 million high-confidence structures predicted by AlphaFold2 (distillation).
  • Evaluation: Standard benchmarks including CAMEO (recent structures) and CASP14 (difficult targets).
  • Metagenomic Application: The MGnify90 database, containing over 617 million sequences from diverse environmental sources.

Key Insights and Ideas

  • Structure Emerges with Scale: The most striking finding is that as the language model grows, structural information "materializes" in its attention maps. A 15B parameter model "knows" significantly more about 3D folding than an 8M parameter one, even though neither was trained on structures.
  • Perplexity as a Proxy: There is a near-perfect correlation between a model’s perplexity (how well it predicts the next amino acid) and its structural accuracy. If the model "understands" the sequence better, it predicts the structure better.
  • The "Jump" Phenomenon: For specific proteins, the model doesn't just get slightly better with scale; it often undergoes a sudden "phase transition" where the predicted structure jumps from a random coil to a high-accuracy fold at a specific parameter threshold.
Visualizing the "jump" in accuracy: as the model scales from 8M to 15B parameters, the predicted structures (teal) align much more closely with the ground truth (gray).

Key Results

  • Speed: ESMFold is up to 60x faster than AlphaFold2 on the inference forward pass. When accounting for the time AlphaFold2 spends searching for MSAs, the end-to-end speedup can be one to two orders of magnitude.
  • Accuracy: On the CAMEO test set, ESMFold achieved a TM-score of 0.83, which is competitive with RosettaFold (0.82) and only slightly behind AlphaFold2 with full MSAs (0.88).
  • The ESM Metagenomic Atlas: The authors folded the entire MGnify90 database (617M sequences). They found that 225 million of these were high-confidence predictions.
  • Novelty: Tens of millions of these metagenomic structures have no close match in the PDB, representing a massive expansion of our known "structural universe."

Limitations and Future Work

  • The MSA Gap: While ESMFold is revolutionary for speed, AlphaFold2 (with a full MSA) still holds an edge in accuracy, especially for proteins with many known evolutionary relatives.
  • Sequence Length: The current Metagenomic Atlas is limited to proteins between 20 and 1024 amino acids due to computational constraints.
  • Complexes: While the paper shows some success in predicting protein-protein complexes (multimers), the model was primarily trained on single chains.
  • Future Direction: The authors suggest that further scaling (beyond 15B parameters) and more diverse training data could continue to close the accuracy gap with MSA-based methods.

Summary updated

Abstract

Abstract Artificial intelligence has the potential to open insight into the structure of proteins at the scale of evolution. It has only recently been possible to extend protein structure prediction to two hundred million cataloged proteins. Characterizing the structures of the exponentially growing billions of protein sequences revealed by large scale gene sequencing experiments would necessitate a break-through in the speed of folding. Here we show that direct inference of structure from primary sequence using a large language model enables an order of magnitude speed-up in high resolution structure prediction. Leveraging the insight that language models learn evolutionary patterns across millions of sequences, we train models up to 15B parameters, the largest language model of proteins to date. As the language models are scaled they learn information that enables prediction of the three-dimensional structure of a protein at the resolution of individual atoms. This results in prediction that is up to 60x faster than state-of-the-art while maintaining resolution and accuracy. Building on this, we present the ESM Metage-nomic Atlas. This is the first large-scale structural characterization of metagenomic proteins, with more than 617 million structures. The atlas reveals more than 225 million high confidence predictions, including millions whose structures are novel in comparison with experimentally determined structures, giving an unprecedented view into the vast breadth and diversity of the structures of some of the least understood proteins on earth.
BibTeX
@misc{lin2022evolutionary,
  author = {Zeming Lin and Halil Akin and Roshan Rao and Brian Hie and Zhongkai Zhu and Wenting Lu and Nikita Smetanin and Robert Verkuil and Ori Kabeli and Yaniv Shmueli and Allan dos Santos Costa and Maryam Fazel-Zarandi and Tom Sercu and Salvatore Candido and Alexander Rives},
  title = {{Evolutionary-scale prediction of atomic level protein structure with a language model}},
  year = {2022},
  doi = {10.1101/2022.07.20.500902},
  howpublished = {bioRxiv},
}
Figure