GOProteinGNN: Leveraging Protein Knowledge Graphs for Protein Representation Learning

Dan Kalifa, Uriel Singer, Kira Radinsky

TL;DR

GOProteinGNN is a novel architecture that bridges the gap between protein language models (PLMs) and structured biological knowledge graphs. By injecting holistic graph information directly into the encoder stage via the [CLS] token, it captures both sequence-level and functional context simultaneously. The model achieves state-of-the-art results across multiple bioinformatics benchmarks and has been successfully deployed in the real world to design lipid nanoparticles that cross the blood-brain barrier in mice.

What Problem Does This Paper Address?

Protein Language Models (PLMs) like ProtBert and ESM have revolutionized bioinformatics by treating amino acid sequences like text, learning the "grammar" of proteins. However, sequence data alone doesn't explicitly capture factual, high-level biological knowledge—such as how a protein interacts with other molecules, its cellular location, or its role in metabolic pathways.

Previous attempts to inject external knowledge from sources like Gene Ontology (GO) knowledge graphs (KGs) suffered from two major flaws:

  1. Simplification: They reduced complex, interconnected knowledge graphs into isolated triplets (Protein, Relation, GO Term), losing the broader topological context.
  2. Late Injection: They incorporated knowledge after the encoder stage via loss objectives. Because language models struggle to memorize "long-tail" (rare) factual knowledge, this post-encoder approach limited the model's ability to truly fuse sequence and structure.

Furthermore, existing models optimized for either amino-acid-level tasks or protein-level tasks, but rarely both at once.

Methods

GOProteinGNN introduces a framework that alternates between transformer-based sequence modeling and graph-based context propagation during the encoding phase.

Figure 1: The GOProteinGNN pre-training architecture incorporates a protein’s knowledge graph with relational and GO term information. First, encoder layers process the amino acid sequence to produce amino acid representations and a protein-level representation (via the [CLS] token). The GKI module then refines the protein representation using graph learning, resulting in a knowledge-enhanced representation that captures key biological context. Finally, subsequent encoder layers further refine the amino acid representations, and the model is trained using masked language modeling (MLM) to predict masked residues.

The core innovation is the GNN Knowledge Injection (GKI) mechanism, which leverages the standard [CLS] token. Instead of using the [CLS] token only at the very end for classification, GOProteinGNN uses it as a dynamic, two-way information highway:

  • Step 1: Early transformer layers process the amino acid sequence, and the [CLS] token aggregates a global summary of the sequence.
  • Step 2: The [CLS] representation is mapped onto its corresponding node in a protein knowledge graph.
  • Step 3: A Relational Graph Convolutional Network (RGCN) propagates functional and relational data from neighboring nodes (like GO terms encoded via PubMedBERT) into the [CLS] token.
  • Step 4: The updated, knowledge-enriched [CLS] token is passed back into the next transformer layers, where self-attention distributes this new context across the individual amino acid representations.

This process repeats $k$ times throughout the $L$ encoder layers. For pre-training, the model uses a standard Masked Language Modeling (MLM) objective. Crucially, during inference, the GKI component is omitted to prevent label leakage and allow the model to generalize to entirely unseen proteins without requiring manual annotations.

Datasets

  • Pre-training: ProteinKG25, which contains roughly 5 million triplets, 600k proteins, 50k attribute terms, and 31 relation terms. The authors strictly removed any sequences present in downstream tasks to ensure a true inductive evaluation setup.
  • Downstream Tasks:
  • PROBE: Used for evaluating semantic similarity inference.
  • SHS27K & SHS148K: Used for protein-protein interaction (PPI) identification.
  • TAPE Benchmark: Used for amino acid contact prediction and remote homology detection.
  • In-House BBBP Dataset: ~2,000 molecules and 500 candidate proteins used for real-world blood-brain barrier permeability validation.

Key Insights and Ideas

  • The [CLS] Token as a Knowledge Bridge: Repurposing the [CLS] token as an iterative aggregator and distributor allows sequence and graph representations to co-evolve during training, rather than forcing a late-stage fusion.
  • Holistic Graph Topology Matters: Moving away from simple triplet matching to full graph convolution (via RGCN) allows the model to capture broader relational nuances and dependencies between functional annotations.
  • In-Encoder Injection Captures the Long-Tail: Injecting knowledge directly into the hidden states helps the model retain rare biological annotations that standard language models typically fail to memorize through text alone.

Key Results

GOProteinGNN consistently outperformed sequence-only models (ProtBert, ESM-2) and the previous knowledge-enhanced state-of-the-art (KeAP) across all evaluated tasks:

  • Semantic Similarity: Achieved a 23.8% performance increase (0.52 Spearman's rank correlation vs. KeAP's 0.41) in capturing biomolecular functional similarities.
  • Contact Prediction & PPI: Consistently achieved the highest precision and F1 scores, proving that high-level functional knowledge refines low-level structural understanding.
  • Real-World Deployment: The model was used to identify optimal proteins for decorating lipid nanoparticles (LNPs) to deliver drugs across the blood-brain barrier (BBB). It predicted Transferrin and Insulin as top candidates with >97% probability.
Figure 2: Distribution of BBB Penetration Scores.

In vivo laboratory validation in mice confirmed that Transferrin-functionalized liposomes achieved a sevenfold increase in drug concentration within brain cells compared to conventional delivery methods.

Limitations and Future Work

  • Base Model Dependency: GOProteinGNN's scalability to exceptionally long proteins or massive datasets remains bound to the context window and computational limits of its underlying PLM (ProtBert in this study).
  • Graph Scaling: While optimizations like limiting protein hops keep the computational complexity manageable ($O(|V|)$), scaling to dynamically updating, multi-million node knowledge graphs will require advanced caching techniques.
  • Future Directions: The authors suggest extending this framework to other advanced PLMs and exploring its utility in virtual screening for drug discovery by prioritizing compounds for specific protein targets.

Summary updated

Abstract

Proteins play a vital role in biological processes and are indispensable for living organisms. Accurate representation of proteins is crucial, especially in drug development. Recently, there has been a notable increase in interest in utilizing machine learning and deep learning techniques for unsupervised learning of protein representations. However, these approaches often focus solely on the amino acid sequence of proteins and lack factual knowledge about proteins and their interactions, thus limiting their performance. In this study, we present GOProteinGNN, a novel architecture that enhances protein language models by integrating protein knowledge graph information during the creation of amino acid level representations. Our approach allows for the integration of information at both the individual amino acid level and the entire protein level, enabling a comprehensive and effective learning process through graph-based learning. By doing so, we can capture complex relationships and dependencies between proteins and their functional annotations, resulting in more robust and contextually enriched protein representations. Unlike previous methods, GOProteinGNN uniquely learns the entire protein knowledge graph during training, which allows it to capture broader relational nuances and dependencies beyond mere triplets as done in previous work. We perform a comprehensive evaluation on several downstream tasks demonstrating that GOProteinGNN consistently outperforms previous methods, showcasing its effectiveness and establishing it as a state-of-the-art solution for protein representation learning.
BibTeX
@inproceedings{kalifa2025goproteingnn,
  author = {Dan Kalifa and Uriel Singer and Kira Radinsky},
  title = {{GOProteinGNN: Leveraging Protein Knowledge Graphs for Protein Representation Learning}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2025},
  doi = {10.1145/3746252.3761500},
}
Figure