Abstract
Protein-protein interactions (PPIs) are critical to normal cellular function and are related to many disease pathways. A range of protein functions are mediated and regulated by protein interactions through post-translational modifications (PTM). However, only 4% of PPIs are annotated with PTMs in biological knowledge databases such as IntAct, mainly performed through manual curation, which is neither time- nor cost-effective. Here we aim to facilitate annotation by extracting PPIs along with their pairwise PTM from the literature by using distantly supervised training data using deep learning to aid human curation. We further assessed model generalisation in a real-world scenario, evaluating its performance on a randomly sampled subset of predictions from 18 million PubMed abstracts.
Introduction
Proteins are the primary functional units of the cell, but their activity is rarely static. Post-translational modifications (PTMs)—such as phosphorylation, acetylation, or ubiquitination—act as molecular switches that regulate protein-protein interactions (PPIs). Despite their importance in drug discovery and disease modeling, our knowledge of these modifications is largely trapped in unstructured scientific literature. Manual curation is a significant bottleneck; currently, only about 4% of known human PPIs in the IntAct database include PTM annotations.
To address this, researchers developed PPI-BioBERT-x10, an ensemble deep learning approach designed to automate the extraction of PTM-specific interactions from millions of scientific abstracts.
Methodology: Distant Supervision and Ensemble Learning
The researchers employed a three-pronged strategy to overcome the lack of labeled training data and the inherent uncertainty in natural language processing.
1. Distant Supervision
Instead of manually labeling thousands of sentences, the team used "distant supervision." They cross-referenced the IntAct database with PubMed abstracts. If a database entry indicated that Protein A modifies Protein B, and an abstract mentioned both proteins along with a PTM keyword, the system automatically labeled that abstract as a positive training example. This allows for the creation of massive datasets without the cost of manual annotation.
2. BioBERT Ensemble (PPI-BioBERT-x10)
The core model is based on BioBERT, a version of the BERT language model pre-trained on large-scale biomedical corpora. To improve reliability and handle the "noisy" nature of distantly supervised data, the authors trained an ensemble of ten models.
3. Confidence Calibration
Deep learning models are often "overconfident," providing high probability scores even for incorrect predictions. By using an ensemble, the researchers could measure the "agreement" between models. High-quality predictions were defined as those where all ten models produced high scores with low variance (standard deviation). This calibration helps filter out ambiguous text where the model might otherwise guess incorrectly.
Key Results and Real-World Performance
When evaluated on a standard test set, the model achieved a modest F1-micro score of 41.3. However, the true test was applying it to the entire PubMed corpus of 18 million abstracts.
- Extraction Scale: The model identified 1.6 million raw PTM-PPI predictions across the corpus.
- High-Confidence Filtering: By applying strict confidence and variance thresholds, the team narrowed this down to 5,708 high-confidence interactions (representing 4,584 unique triplets).
- The Generalization Gap: Human evaluation of a random sample revealed a significant "precision drop" in the wild, falling to 33.7%. This highlights a "generalization gap" where models learn specific quirks of the training data that do not always translate to the diverse language used across all of PubMed.
- The Power of Consensus: Precision was significantly improved (to 58.8%) by requiring "consensus" — only accepting predictions that appeared in multiple different papers.
Conclusion
This work highlights both the immense potential and the persistent challenges of using AI for biological text mining. While PPI-BioBERT-x10 identified 200% more PTM-PPIs than previous rule-based systems, the gap between lab performance and real-world accuracy remains a hurdle. Ultimately, these tools are best viewed as "human augmentation" systems, helping curators prioritize the most relevant literature in an ever-expanding sea of data.
Summary updated