Abstract
Protein-to-protein interaction (PPI) prediction is vital for understanding biological processes and drug discovery. Traditional deep learning models often struggle with "Out-of-Distribution" (OOD) scenarios, failing to generalize to species not seen during training. This paper introduces PPITrans, a Transformer-based model that leverages the pre-trained ProtT5 language model within a Siamese architecture. By training on human PPI data and testing across diverse species (Mouse, Fly, Worm, Yeast, and E. coli), PPITrans demonstrates superior cross-species generalization, achieving up to a 33.9% absolute improvement in AUPR on Fly PPIs. Furthermore, through a series of probing experiments, we reveal that the model captures emergent biological knowledge, such as binding affinity and interaction types, despite being trained only on binary interaction labels.
Introduction
Predicting PPIs is a fundamental challenge in bioinformatics. While deep learning has advanced the field, most models are limited by their inability to handle the evolutionary variance of protein sequences across different species. A model trained on human data often fails on yeast or bacteria because it has only learned the specific "vocabulary" of human proteins. Additionally, these models are frequently treated as black boxes, leaving it unclear whether they are learning true biological principles or merely statistical correlations. PPITrans addresses these issues by utilizing massive pre-trained protein embeddings and a robust Transformer architecture designed for generalization.
Methods: The PPITrans Architecture
PPITrans is built upon three core components designed to maximize contextual understanding and sequence processing:
- Pre-trained Embedder: The model uses ProtT5, a language model with 3 billion parameters pre-trained on billions of protein sequences. This provides a rich, evolutionary-aware representation of residues before task-specific training begins.
- Protein Encoder: An 8-layer Transformer encoder processes the embeddings. Unlike CNNs, Transformers maintain a constant sequence length and can capture long-range dependencies, which is critical for the subsequent probing analysis.
- Prediction Module: Utilizing a Siamese architecture, two identical encoders process a pair of proteins. Their representations are combined via a Hadamard product (element-wise multiplication) and passed to a classifier to predict the interaction probability.
Key Results and Generalization
PPITrans was trained exclusively on Human PPI data (~38k positive samples) and evaluated on five other species to test OOD performance. It consistently outperformed state-of-the-art baselines like PIPR and D-SCRIPT.
The most significant gains were observed in the Fly dataset, where AUPR increased from 0.562 to 0.901. Even for E. coli, which is evolutionarily distant from humans, the model showed a 14.3 percentage point improvement in AUPR.
The Homology Threshold
A critical finding of this study is the relationship between sequence similarity and model performance. As shown in the figure below, the model's performance (AUPR) remains remarkably stable across species until the maximum protein identity (similarity to the human training set) drops below 30%. Below this threshold, the predictive power degrades sharply, highlighting a clear boundary for reliable cross-species inference.
Probing Emergent Knowledge
To look inside the "black box," the authors used auxiliary classifiers (probes) to test what information is stored in the model's layers. Key insights include:
- Forgetting Hierarchy: The model gradually discards fine-grained residue details in higher layers to focus on coarse-grained features necessary for interaction prediction.
- Functional vs. Structural Knowledge: PPITrans effectively predicts binding affinity and interaction types (e.g., reaction vs. binding) even though it was never explicitly trained for these tasks. However, it performs poorly at predicting 3D spatial contact maps, suggesting it relies more on sequence-based co-occurrence than explicit geometry.
Conclusion
PPITrans represents a significant step forward in cross-species PPI prediction. By combining large-scale pre-training with a Transformer architecture, it overcomes many OOD challenges and reveals that deep models can "accidentally" learn complex biophysical properties like binding affinity. Future work will focus on integrating structural awareness into the training process to improve performance on proteins with low sequence identity.
Summary updated