LM2Protein: A Structure-to-Token Protein Large Language Model

Chang Zhou, Yuheng Shan, Pengan Chen, Xiangyu Shi, Zikang Wang, Yanting Li, Jiyue JiangEMNLP 2025 · OpenReview

TL;DR

This paper introduces LM2Protein, a novel large language model that unifies protein sequence and 3D structural information by converting structural coordinates into discrete "tokens." This allows LLMs to process both sequence and structure in a unified sequence-to-sequence framework, enabling impressive performance in tasks like inverse protein folding and RNA design.

What Problem Does This Paper Address?

Proteins are the workhorses of biology, and their function is intimately tied to their complex 3D structures. However, the relationship between a protein's amino acid sequence and its final folded structure is not a simple one-to-one mapping; multiple sequences can fold into similar structures. This complexity makes predicting, designing, and modifying proteins challenging, especially as sequences grow longer and more intricate. Furthermore, understanding how proteins interact with other biomolecules, like RNA (RNA-binding proteins or RBPs), is crucial. While Large Language Models (LLMs) have excelled at processing biological sequences as text, integrating the 3D spatial information of proteins has been a hurdle, often requiring separate, complex computer vision or 3D modeling pipelines. This paper aims to bridge this gap by creating a unified framework that can handle both protein sequences and their 3D structures seamlessly.

Methods

The core innovation of LM2Protein lies in its method for representing protein 3D structures in a way that LLMs can understand: structural tokenization.

  1. SE(3)-Equivariant Structural Tokenization: Instead of directly feeding raw 3D coordinates, the model first uses a lightweight SE(3)-equivariant encoder (an Equiformer) to process local 3D atomic point clouds around each residue. This encoder generates continuous "latent descriptors" that capture both local and global structural patterns, ensuring consistency regardless of the protein's overall pose.
  2. Vector Quantization (VQ-VAE): These continuous descriptors are then mapped to the nearest entries in a pre-trained codebook of 4096 "structural tokens." This process, inspired by VQ-VAEs, discretizes the structural information into a sequence of integer IDs.
  3. Unified Sequence-to-Sequence Modeling: The resulting sequence of structural tokens, along with the amino acid sequence, is fed into an augmented decoder-only LLM (specifically, a fine-tuned Qwen 2.5 model). The tokenizer of this LLM is extended to include these structural tokens, along with task delimiters and residue tags. This allows the model to process both sequence and structure as a single, unified sequence, enabling it to perform various tasks within a standard sequence-to-sequence paradigm.

Datasets

  • Protein Data: The model was trained on protein structures from the AlphaFold DB and tested on data from the BFVD database.
  • RBP Data: Complex structures for RNA-binding protein (RBP) tasks were gathered from EMDB and PDB.

The paper does not introduce new benchmarks but rather applies its model to existing tasks using these established datasets.

Key Insights and Ideas

  • Structure as Language: The fundamental idea is to treat protein 3D structure as a form of "language" that LLMs can learn. By discretizing coordinates into tokens, the model bypasses the need for complex 3D processing pipelines, making integration much simpler.
  • Unified Framework: The ability to handle both sequence and structure within a single Seq2Seq model is a significant step towards a more holistic understanding and manipulation of biomolecules. This unification simplifies training and inference for diverse tasks.
  • Bridging Geometric and Textual Data: LM2Protein effectively bridges the gap between geometric point cloud data and sequence-based language models, opening new avenues for multimodal biological modeling.
  • RNA Design Potential: The model shows remarkable promise in RNA design tasks, suggesting that understanding protein structure-sequence relationships can directly inform the design of interacting molecules.

Key Results

  • Protein Structure Prediction: LM2Protein achieved a TM-score of 54.80% and an RMSD of 6.9428 Å. While slightly underperforming AlphaFold3, it demonstrates usable predictions with significantly fewer parameters (1.5 billion vs. AlphaFold3's massive scale).
  • Protein Sequence Design (Inverse Folding): LM2Protein significantly outperformed baselines like ProteinMPNN, PiFold, and ESM-IF, achieving a recovery rate of 51.00% and sequence similarity of 39.30%. This highlights its strength in reconstructing sequences from structures.
  • RNA Design based on RBP Sequence: The model achieved an impressive recovery rate of 73.7%, a more than threefold improvement over the baseline RNAFlow (21.0%). This indicates its effectiveness in generating RNA sequences that meet binding requirements based on protein sequences.
  • RNA Design based on RBP Structure: LM2Protein also showed strong performance here, with a recovery rate of 72.1% (compared to RNAFlow's 26.0%), demonstrating its ability to leverage protein structural information for RNA design.

Limitations and Future Work

  • Structural Accuracy: While LM2Protein excels at sequence generation and capturing structural motifs, it still faces challenges in accurately predicting or generating high-fidelity tertiary structures for RNA. The paper notes that the generated RNA sequences have an RMSD of 10.0282 Å and a TM-score of 42.32% in sequence-to-structure evaluation.
  • Topology Representation: The reliance on VQ-VAE for tokenization might underrepresent rare or anomalous protein topologies.
  • Resolution Limits: The structural resolution of the tokenized features might limit the model's ability to capture very fine-grained 3D details.
  • Future Directions: The authors suggest future work will focus on integrating biophysical priors to enhance structural accuracy, extending the framework to more complex molecular systems, and developing a universal model for in silico molecular engineering. They also acknowledge that the model struggles with global folding constraints, capturing local sequence motifs more effectively.

Summary updated

Abstract

Proteins are critical for various molecular functions, relying on their precise tertiary structures. This structure-sequence relationship is complex and degenerate, meaning multiple sequences can fold into a similar structure. The challenges in protein prediction, design, and modification increase with sequence complexity, while research on RNA-protein interactions, especially RNA-binding proteins (RBPs), is gaining importance. Large-scale pre-trained language models (LLMs) have shown promising results in handling biological sequences by treating them as natural language, though integrating spatial structures remains complex due to the need for specialized visual and 3D modeling approaches. We introduce a method to integrate protein 3D structural data within a sequence processing framework, converting 3D coordinates into discrete structure tokens using a VQ-VAE-like network. This simplifies the handling of 3D data, avoiding complex pipelines and facilitating a unified sequence-to-sequence processing model. Our approach demonstrates strong performance across a range of tasks, achieving high sequence recovery in inverse folding and protein-conditioned RNA design. These outstanding results demonstrate significant potential for application in complex biological systems research.
BibTeX
@inproceedings{zhou2025lm,
  author = {Chang Zhou and Yuheng Shan and Pengan Chen and Xiangyu Shi and Zikang Wang and Yanting Li and Jiyue Jiang},
  title = {{LM2Protein: A Structure-to-Token Protein Large Language Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=iYwTeNcf3Z},
}
Figure