xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein

Bo Chen, Xingyi Cheng, Li Pan, Yangli‐ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, Chi-Ming Liu, Aohan Zeng, Yuxiao Dong, Jie Tang, Le Song

Abstract

Protein language models have shown remarkable success in learning biological information from protein sequences. However, most existing models are limited by either autoencoding or autoregressive pre-training objectives, which makes them struggle to handle protein understanding and generation tasks concurrently. We propose a unified protein language model, xTrimoPGLM, to address these two types of tasks simultaneously through an innovative pre-training framework. Our key technical contribution is an exploration of the compatibility and the potential for joint optimization of the two types of objectives, which has led to a strategy for training xTrimoPGLM at an unprecedented scale of 100 billion parameters and 1 trillion training tokens. Our extensive experiments reveal that 1) xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. The model also facilitates an atomic-resolution view of protein structures, leading to an advanced 3D structural prediction model that surpasses existing language model-based tools. 2) xTrimoPGLM not only can generate de novo protein sequences following the principles of natural ones, but also can perform programmable generation after supervised fine-tuning (SFT) on curated sequences. These results highlight the substantial capability and versatility of xTrimoPGLM in understanding and generating protein sequences, contributing to the evolving landscape of foundation models in protein science.

Introduction

The development of Protein Language Models (PLMs) has traditionally been bifurcated into two distinct paradigms: autoencoding models (e.g., ESM, BERT) and autoregressive models (e.g., ProGen, GPT). Autoencoding models excel at "understanding" tasks, such as predicting protein structure or function from sequences, but lack generative capabilities. Conversely, autoregressive models are proficient at generating new protein sequences but often lag in providing high-quality representations for downstream prediction tasks. xTrimoPGLM bridges this divide, serving as a unified foundation model that handles both understanding and generation concurrently at an unprecedented scale of 100 billion parameters.

Methodology

xTrimoPGLM is built upon the General Language Model (GLM) framework, which employs a "blank-filling" objective. This architecture allows for bidirectional attention—essential for robust representation learning—while maintaining the autoregressive nature required for sequence generation.

Key technical innovations include:

  • Unified Training Strategy: The model combines Masked Language Modeling (MLM) for random token prediction with the GLM objective for predicting consecutive spans of tokens.
  • Two-Stage Training and Smooth Transition: Initial pre-training focused on MLM to build strong representations, followed by a joint MLM+GLM objective. To maintain stability at the 100B scale, a "Smooth Transition" (ST) strategy gradually increased the ratio of the generation loss.
  • Metagenomic Data Integration: In addition to Uniref90, the model was trained on the ColAbFoldDB metagenomic dataset, capturing a vast array of "evolutionary experiments" from environmental sequences.

Scaling Laws in Protein Modeling

A core finding of the xTrimoPGLM project is that scaling laws observed in natural language processing are equally applicable to the biological "language" of proteins. There is a consistent linear relationship between model performance and the logarithm of computational FLOPs across diverse biological tasks.

Scaling laws for xTrimoPGLM across four major protein task categories: Structure, Function, Interaction, and Developability.

As shown in the scaling plots, performance improves predictably as model size and training compute increase. The 100B parameter model (xT100B) demonstrates emergent abilities, particularly in complex tasks where smaller models plateau.

Functional Representation and Benchmarks

xTrimoPGLM outperformed advanced baselines on 13 out of 15 downstream benchmarks. The quality of its learned representations is further evidenced by its ability to organize the protein universe into biologically meaningful clusters without explicit functional supervision.

t-SNE visualization of protein representations from xTrimoPGLM, showing distinct clusters corresponding to specific Gene Ontology (GO) terms.

The t-SNE visualization demonstrates that the model's latent space naturally aligns with Gene Ontology (GO) terms, such as transmembrane transporter activity and DNA-binding transcription repressor activity. This indicates that the model has internalized deep biological principles purely from sequence data.

Accelerating Antibody Engineering

The practical utility of xTrimoPGLM is exemplified by its specialized antibody variant, xTrimoPGLM-Ab. When applied to structural prediction (xTrimoPGLM-AbFold), it achieves accuracy on par with AlphaFold2 but at 6,000 times the inference speed. This breakthrough allows for the high-throughput screening of antibody libraries, a critical bottleneck in modern drug discovery.

Conclusion

xTrimoPGLM represents a significant milestone in protein science, demonstrating that "understanding" and "generation" objectives are not only compatible but synergistic. While the computational cost of training such massive models remains a challenge, the success of xTrimoPGLM paves the way for more capable and versatile foundation models in biotechnology. Future work will focus on improving conditional generation for targeted protein design and enhancing model efficiency through quantization and advanced attention mechanisms.

Summary updated

Abstract

Protein language models have shown remarkable success in learning biological information from protein sequences. However, most existing models are limited by either autoencoding or autoregressive pre-training objectives, which makes them struggle to handle protein understanding and generation tasks concurrently. We propose a unified protein language model, xTrimoPGLM, to address these two types of tasks simultaneously through an innovative pre-training framework. Our key technical contribution is an exploration of the compatibility and the potential for joint optimization of the two types of objectives, which has led to a strategy for training xTrimoPGLM at an unprecedented scale of 100 billion parameters and 1 trillion training tokens. Our extensive experiments reveal that 1) xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. The model also facilitates an atomic-resolution view of protein structures, leading to an advanced 3D structural prediction model that surpasses existing language model-based tools. 2) xTrimoPGLM not only can generate de novo protein sequences following the principles of natural ones, but also can perform programmable generation after supervised fine-tuning (SFT) on curated sequences. These results highlight the substantial capability and versatility of xTrimoPGLM in understanding and generating protein sequences, contributing to the evolving landscape of foundation models in protein science. Trained weight for the xTrimoPGLM model, and downstream datasets are available at https://huggingface.co/proteinglm .
BibTeX
@misc{chen2024xtrimopglm,
  author = {Bo Chen and Xingyi Cheng and Li Pan and Yangli‐ao Geng and Jing Gong and Shen Li and Zhilei Bei and Xu Tan and Boyan Wang and Xin Zeng and Chi-Ming Liu and Aohan Zeng and Yuxiao Dong and Jie Tang and Le Song},
  title = {{xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein}},
  year = {2024},
  eprint = {2401.06199},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.06199},
}
Figure