Endowing Protein Language Models with Structural Knowledge

Dexiong Chen, Philip Hartout, Paolo Pellizzoni, Carlos Oliver, Karsten BorgwardtSubmission to ICLR 2024 · Rejected

Abstract

Protein language models have shown strong performance in predicting function and structure across diverse tasks. These models undergo unsupervised pretraining on vast sequence databases to generate rich protein representations, followed by finetuning with labeled data on specific downstream tasks. The recent surge in computationally predicted protein structures opens new opportunities in protein representation learning. In our study, we introduce a novel framework to enhance transformer protein language models specifically on protein structures. Drawing from recent advances in graph transformers, our approach refines the self-attention mechanisms of pretrained language transformers by integrating structural information with structure extractor modules. This refined model, termed the Protein Structure Transformer (PST), is further pretrained on a protein structure database such as AlphaFoldDB, using the same masked language modeling objective as traditional protein language models. Our empirical findings show superior performance on several benchmark datasets. Notably, PST consistently outperforms the foundation model for protein sequences, ESM-2, upon which it is built. Our code and pretrained models will be released upon publication.
BibTeX
@article{chen2025endowing,
  author = {Dexiong Chen and Philip Hartout and Paolo Pellizzoni and Carlos Oliver and Karsten Borgwardt},
  title = {{Endowing Protein Language Models with Structural Knowledge}},
  journal = {Bioinformatics},
  year = {2025},
  doi = {10.1093/bioinformatics/btaf582},
}
Figure