Bilingual language model for protein sequence and structure

Michael Heinzinger, Konstantin Weißenow, Joaquín Gómez Sánchez, Adrian Henkel, Milot Mirdita, Martin Steinegger, Burkhard Rost

TL;DR

This paper introduces ProstT5, a novel "bilingual" language model that can translate between protein sequences (1D) and their 3D structures. By leveraging a new 3Di tokenization scheme and fine-tuning an existing protein language model, ProstT5 enables significantly faster and more sensitive structural searches, opening new avenues for protein analysis and design.

What Problem Does This Paper Address?

Proteins are fundamental to life, acting as molecular machines with complex 3D structures while evolving as linear sequences of amino acids. For years, researchers have developed powerful tools to analyze these sequences (protein language models or pLMs) and, more recently, to predict their 3D structures with remarkable accuracy (e.g., AlphaFold2). However, these two modalities (sequence and structure) have largely been studied in isolation. The challenge is to bridge this gap and create models that can understand and translate between both, unlocking a more comprehensive view of protein function and evolution. Specifically, the authors aim to develop a model that can leverage the vast amount of predicted 3D structures to improve protein analysis tasks and enable faster, more sensitive structure-based searches.

Methods

The core innovation is ProstT5, a "bilingual" protein language model built upon the T5 architecture. The key steps include:

  • 3Di Tokenization: Protein structures are converted into 1D token sequences using the "3Di-alphabet" from the Foldseek method. Each residue is represented by a token indicating its structural context, effectively translating 3D information into a 1D string.
  • Model Architecture: ProstT5 is based on the ProtT5 model, a pre-trained protein language model. It uses an encoder-decoder architecture, which is well-suited for translation tasks.
  • Pre-training: The model is first pre-trained on a large, non-redundant dataset of AlphaFold2-predicted structures. This pre-training involves a "span denoising" objective adapted for both amino acid sequences and 3Di tokens, aiming to teach the model the new 3Di tokens while preventing "catastrophic forgetting" of its existing knowledge about amino acid sequences.
  • Bi-directional Translation Training: The model is then fine-tuned to explicitly translate between amino acid sequences and 3Di sequences. Special tokens (<fold2AA> and <AA2fold>) are used to indicate the translation direction.
  • Optimized Inference: For specific tasks like structural homology search, a more efficient CNN-based inference method is employed, bypassing the slower decoder for faster 3Di prediction.

Datasets

  • Training Data: A non-redundant dataset derived from the AlphaFold Protein Structure Database (AFDB) was used. This dataset was curated through multi-stage clustering (MMseqs2 and Foldseek) and quality filtering, resulting in approximately 17 million unique protein structures.
  • Validation and Testing Data: A small subset of the AFDB data was split for validation and testing.
  • Benchmarking Datasets: Various established datasets were used for evaluating downstream tasks, including CASP12, CASP14, NEW364 (for secondary structure prediction), DevSet1014/TestSet300 (for binding residue prediction), ConSurf10k (for conservation prediction), DeepLoc (for subcellular localization), and CATH (for superfamily classification).

Key Insights and Ideas

  • Bridging the Sequence-Structure Divide: The most significant insight is that a single language model can effectively learn to translate between the 1D sequence and 3D structural representations of proteins. This "bilingual" capability unlocks new possibilities for analysis.
  • 3Di as a Powerful Representation: The 3Di alphabet, originally developed for structural alignment, proves to be a rich enough representation of 3D structure to be directly usable by language models, capturing essential structural information.
  • Faster, Sensitive Structural Search: By predicting 3Di sequences directly from amino acid sequences, ProstT5 enables structural homology searches (using tools like Foldseek) at speeds comparable to sequence searches, achieving a three-orders-of-magnitude speedup over traditional structure prediction methods.
  • Orthogonal Information: ProstT5's ability to process both sequence and structure provides complementary information. Embeddings derived from structure-aware training improve performance on structure-centric tasks, while sequence-based embeddings still perform well on function-related tasks. Concatenating embeddings from both modalities further boosts performance.
  • Inverse Folding Potential: The model demonstrates promising capabilities in inverse folding, generating novel amino acid sequences that are predicted to adopt specific 3D structures, albeit with room for improvement compared to specialized methods.

Key Results

  • Structural Homology Detection: ProstT5-predicted 3Di sequences achieved performance close to experimental structures in Foldseek-based remote homology detection, significantly outperforming traditional sequence-based methods.
  • Speedup: Predicting 3Di sequences using ProstT5's CNN-based inference is approximately three orders of magnitude faster than deriving 3Di from full structure predictions. For example, predicting 3Di for a human proteome took minutes on a GPU, compared to days for structure prediction.
  • CATH Classification: ProstT5 embeddings improved performance on classifying proteins into CATH structural hierarchy levels, with structure-informed embeddings (ProstT5(3Di)) showing particular strength in fine-grained homologous superfamily classification.
  • Secondary Structure Prediction: ProstT5 embeddings, even when using only AA input, improved secondary structure prediction accuracy over its base model ProtT5. Using 3Di-derived embeddings approached near 90% accuracy on some benchmarks.
  • Inverse Folding: Generated sequences achieved an average IDDT of 72, a strong result for inverse folding, and in some cases outperformed specialized methods.

Limitations and Future Work

  • Data Bias: The authors acknowledge potential biases in the training data, particularly the enrichment of well-structured, helical proteins due to filtering for high pLDDT scores. This also leads to a significant class imbalance in 3Di tokens.
  • Circularity: There's a degree of circularity in evaluating secondary structure prediction using embeddings derived from 3Di tokens, which themselves capture secondary structure information. However, the authors argue this did not impair practical usefulness and ProstT5 embeddings still outperformed simpler representations.
  • Function-centric Tasks: ProstT5's performance on function-related tasks (like binding residue prediction or subcellular localization) showed a slight decrease compared to its base model, suggesting a potential trade-off in capacity. Concatenating embeddings from original ProtT5 and ProstT5(AA) could partially mitigate this.
  • Future Work: The authors suggest improving 3D-to-1D mappings to address class imbalance, exploring more balanced training data, and expanding ProstT5 to incorporate other conditioning tags (like Gene Ontology terms) to create truly "polyglot" models. Further work on optimizing sampling strategies for inverse folding and increasing context length for processing full protein sequences is also envisioned.

Summary updated

Abstract

Adapting language models to protein sequences spawned the development of powerful protein language models (pLMs). Concurrently, AlphaFold2 broke through in protein structure prediction. Now we can systematically and comprehensively explore the dual nature of proteins that act and exist as three-dimensional (3D) machines and evolve as linear strings of one-dimensional (1D) sequences. Here, we leverage pLMs to simultaneously model both modalities in a single model. We encode protein structures as token sequences using the 3Di-alphabet introduced by the 3D-alignment method <i>Foldseek</i>. For training, we built a non-redundant dataset from AlphaFoldDB and fine-tuned an existing pLM (ProtT5) to translate between 3Di and amino acid sequences. As a proof-of-concept for our novel approach, dubbed Protein 'structure-sequence' T5 (<i>ProstT5</i>), we showed improved performance for subsequent, structure-related prediction tasks, leading to three orders of magnitude speedup for deriving 3Di. This will be crucial for future applications trying to search metagenomic sequence databases at the sensitivity of structure comparisons. Our work showcased the potential of pLMs to tap into the information-rich protein structure revolution fueled by AlphaFold2. <i>ProstT5</i> paves the way to develop new tools integrating the vast resource of 3D predictions and opens new research avenues in the post-AlphaFold2 era.
BibTeX
@inproceedings{heinzinger2024bilingual,
  author = {Michael Heinzinger and Konstantin Weißenow and Joaquín Gómez Sánchez and Adrian Henkel and Milot Mirdita and Martin Steinegger and Burkhard Rost},
  title = {{Bilingual language model for protein sequence and structure}},
  booktitle = {NAR},
  year = {2024},
  doi = {10.1093/nargab/lqae150},
}
Figure