Exploring the Knowledge of An Outstanding Protein to Protein Interaction Transformer

Sen Yang, Dawei Feng, Peng Cheng, Yang Liu, Shengqi Wang

Abstract

Protein-to-protein interaction (PPI) prediction is vital for understanding biological processes and drug discovery. Traditional deep learning models often struggle with "Out-of-Distribution" (OOD) scenarios, failing to generalize to species not seen during training. This paper introduces PPITrans, a Transformer-based model that leverages the pre-trained ProtT5 language model within a Siamese architecture. By training on human PPI data and testing across diverse species (Mouse, Fly, Worm, Yeast, and E. coli), PPITrans demonstrates superior cross-species generalization, achieving up to a 33.9% absolute improvement in AUPR on Fly PPIs. Furthermore, through a series of probing experiments, we reveal that the model captures emergent biological knowledge, such as binding affinity and interaction types, despite being trained only on binary interaction labels.

Introduction

Predicting PPIs is a fundamental challenge in bioinformatics. While deep learning has advanced the field, most models are limited by their inability to handle the evolutionary variance of protein sequences across different species. A model trained on human data often fails on yeast or bacteria because it has only learned the specific "vocabulary" of human proteins. Additionally, these models are frequently treated as black boxes, leaving it unclear whether they are learning true biological principles or merely statistical correlations. PPITrans addresses these issues by utilizing massive pre-trained protein embeddings and a robust Transformer architecture designed for generalization.

Methods: The PPITrans Architecture

PPITrans is built upon three core components designed to maximize contextual understanding and sequence processing:

  1. Pre-trained Embedder: The model uses ProtT5, a language model with 3 billion parameters pre-trained on billions of protein sequences. This provides a rich, evolutionary-aware representation of residues before task-specific training begins.
  2. Protein Encoder: An 8-layer Transformer encoder processes the embeddings. Unlike CNNs, Transformers maintain a constant sequence length and can capture long-range dependencies, which is critical for the subsequent probing analysis.
  3. Prediction Module: Utilizing a Siamese architecture, two identical encoders process a pair of proteins. Their representations are combined via a Hadamard product (element-wise multiplication) and passed to a classifier to predict the interaction probability.
Figure 2: The architecture of PPITrans, featuring the pre-trained ProtT5 embedder, Siamese Transformer encoders, and the final classifier module.

Key Results and Generalization

PPITrans was trained exclusively on Human PPI data (~38k positive samples) and evaluated on five other species to test OOD performance. It consistently outperformed state-of-the-art baselines like PIPR and D-SCRIPT.

The most significant gains were observed in the Fly dataset, where AUPR increased from 0.562 to 0.901. Even for E. coli, which is evolutionarily distant from humans, the model showed a 14.3 percentage point improvement in AUPR.

The Homology Threshold

A critical finding of this study is the relationship between sequence similarity and model performance. As shown in the figure below, the model's performance (AUPR) remains remarkably stable across species until the maximum protein identity (similarity to the human training set) drops below 30%. Below this threshold, the predictive power degrades sharply, highlighting a clear boundary for reliable cross-species inference.

Figure 4: Performance (AUPR) and sample distribution across different species as a function of maximum protein identity to the training set.

Probing Emergent Knowledge

To look inside the "black box," the authors used auxiliary classifiers (probes) to test what information is stored in the model's layers. Key insights include:

  • Forgetting Hierarchy: The model gradually discards fine-grained residue details in higher layers to focus on coarse-grained features necessary for interaction prediction.
  • Functional vs. Structural Knowledge: PPITrans effectively predicts binding affinity and interaction types (e.g., reaction vs. binding) even though it was never explicitly trained for these tasks. However, it performs poorly at predicting 3D spatial contact maps, suggesting it relies more on sequence-based co-occurrence than explicit geometry.

Conclusion

PPITrans represents a significant step forward in cross-species PPI prediction. By combining large-scale pre-training with a Transformer architecture, it overcomes many OOD challenges and reveals that deep models can "accidentally" learn complex biophysical properties like binding affinity. Future work will focus on integrating structural awareness into the training process to improve performance on proteins with low sequence identity.

Summary updated

Abstract

Abstract Protein-to-protein interaction (PPI) prediction aims to predict whether two given proteins interact or not. Compared with traditional experimental methods of high cost and low efficiency, the current deep learning based approach makes it possible to discover massive potential PPIs from large-scale databases. However, deep PPI prediction models perform poorly on unseen species, as their proteins are not in the training set. Targetting on this issue, the paper first proposes PPITrans, a Transformer based PPI prediction model that exploits a language model pre-trained on proteins to conduct binary PPI prediction. To validate the effectiveness on unseen species, PPITrans is trained with Human PPIs and tested on PPIs of other species. Experimental results show that PPITrans significantly outperforms the previous state-of-the-art on various metrics, especially on PPIs of unseen species. For example, the AUPR improves 0.339 absolutely on Fly PPIs. Aiming to explore the knowledge learned by PPITrans from PPI data, this paper also designs a series of probes belonging to three categories. Their results reveal several interesting findings, like that although PPITrans cannot capture the spatial structure of proteins, it can obtain knowledge of PPI type and binding affinity, learning more than binary PPI.
BibTeX
@article{yang2024exploring,
  author = {Sen Yang and Dawei Feng and Peng Cheng and Yang Liu and Shengqi Wang},
  title = {{Exploring the Knowledge of An Outstanding Protein to Protein Interaction Transformer}},
  journal = {IEEE ACM Trans. Comput. Biol. Bioinform.},
  year = {2024},
  url = {https://openreview.net/forum?id=pqTguBuehd},
}
Figure