Abstract
Protein language models have shown remarkable success in learning biological information from protein sequences. However, most existing models are limited by either autoencoding or autoregressive pre-training objectives, which makes them struggle to handle protein understanding and generation tasks concurrently. We propose a unified protein language model, xTrimoPGLM, to address these two types of tasks simultaneously through an innovative pre-training framework. Our key technical contribution is an exploration of the compatibility and the potential for joint optimization of the two types of objectives, which has led to a strategy for training xTrimoPGLM at an unprecedented scale of 100 billion parameters and 1 trillion training tokens. Our extensive experiments reveal that 1) xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. The model also facilitates an atomic-resolution view of protein structures, leading to an advanced 3D structural prediction model that surpasses existing language model-based tools. 2) xTrimoPGLM not only can generate de novo protein sequences following the principles of natural ones, but also can perform programmable generation after supervised fine-tuning (SFT) on curated sequences. These results highlight the substantial capability and versatility of xTrimoPGLM in understanding and generating protein sequences, contributing to the evolving landscape of foundation models in protein science.
Introduction
The development of Protein Language Models (PLMs) has traditionally been bifurcated into two distinct paradigms: autoencoding models (e.g., ESM, BERT) and autoregressive models (e.g., ProGen, GPT). Autoencoding models excel at "understanding" tasks, such as predicting protein structure or function from sequences, but lack generative capabilities. Conversely, autoregressive models are proficient at generating new protein sequences but often lag in providing high-quality representations for downstream prediction tasks. xTrimoPGLM bridges this divide, serving as a unified foundation model that handles both understanding and generation concurrently at an unprecedented scale of 100 billion parameters.
Methodology
xTrimoPGLM is built upon the General Language Model (GLM) framework, which employs a "blank-filling" objective. This architecture allows for bidirectional attention—essential for robust representation learning—while maintaining the autoregressive nature required for sequence generation.
Key technical innovations include:
- Unified Training Strategy: The model combines Masked Language Modeling (MLM) for random token prediction with the GLM objective for predicting consecutive spans of tokens.
- Two-Stage Training and Smooth Transition: Initial pre-training focused on MLM to build strong representations, followed by a joint MLM+GLM objective. To maintain stability at the 100B scale, a "Smooth Transition" (ST) strategy gradually increased the ratio of the generation loss.
- Metagenomic Data Integration: In addition to Uniref90, the model was trained on the ColAbFoldDB metagenomic dataset, capturing a vast array of "evolutionary experiments" from environmental sequences.
Scaling Laws in Protein Modeling
A core finding of the xTrimoPGLM project is that scaling laws observed in natural language processing are equally applicable to the biological "language" of proteins. There is a consistent linear relationship between model performance and the logarithm of computational FLOPs across diverse biological tasks.
As shown in the scaling plots, performance improves predictably as model size and training compute increase. The 100B parameter model (xT100B) demonstrates emergent abilities, particularly in complex tasks where smaller models plateau.
Functional Representation and Benchmarks
xTrimoPGLM outperformed advanced baselines on 13 out of 15 downstream benchmarks. The quality of its learned representations is further evidenced by its ability to organize the protein universe into biologically meaningful clusters without explicit functional supervision.
The t-SNE visualization demonstrates that the model's latent space naturally aligns with Gene Ontology (GO) terms, such as transmembrane transporter activity and DNA-binding transcription repressor activity. This indicates that the model has internalized deep biological principles purely from sequence data.
Accelerating Antibody Engineering
The practical utility of xTrimoPGLM is exemplified by its specialized antibody variant, xTrimoPGLM-Ab. When applied to structural prediction (xTrimoPGLM-AbFold), it achieves accuracy on par with AlphaFold2 but at 6,000 times the inference speed. This breakthrough allows for the high-throughput screening of antibody libraries, a critical bottleneck in modern drug discovery.
Conclusion
xTrimoPGLM represents a significant milestone in protein science, demonstrating that "understanding" and "generation" objectives are not only compatible but synergistic. While the computational cost of training such massive models remains a challenge, the success of xTrimoPGLM paves the way for more capable and versatile foundation models in biotechnology. Future work will focus on improving conditional generation for targeted protein design and enhancing model efficiency through quantization and advanced attention mechanisms.
Summary updated