Large-Scale Protein-Protein Post-Translational Modification Extraction With Distant Supervision And Confidence Calibrated BioBERT

Aparna Elangovan, Yuan Li, Douglas E. V. Pires, Melissa J. Davis, Karin VerspoorResearch 2021 · DOI

Abstract

Protein-protein interactions (PPIs) are critical to normal cellular function and are related to many disease pathways. A range of protein functions are mediated and regulated by protein interactions through post-translational modifications (PTM). However, only 4% of PPIs are annotated with PTMs in biological knowledge databases such as IntAct, mainly performed through manual curation, which is neither time- nor cost-effective. Here we aim to facilitate annotation by extracting PPIs along with their pairwise PTM from the literature by using distantly supervised training data using deep learning to aid human curation. We further assessed model generalisation in a real-world scenario, evaluating its performance on a randomly sampled subset of predictions from 18 million PubMed abstracts.

Introduction

Proteins are the primary functional units of the cell, but their activity is rarely static. Post-translational modifications (PTMs)—such as phosphorylation, acetylation, or ubiquitination—act as molecular switches that regulate protein-protein interactions (PPIs). Despite their importance in drug discovery and disease modeling, our knowledge of these modifications is largely trapped in unstructured scientific literature. Manual curation is a significant bottleneck; currently, only about 4% of known human PPIs in the IntAct database include PTM annotations.

To address this, researchers developed PPI-BioBERT-x10, an ensemble deep learning approach designed to automate the extraction of PTM-specific interactions from millions of scientific abstracts.

Methodology: Distant Supervision and Ensemble Learning

The researchers employed a three-pronged strategy to overcome the lack of labeled training data and the inherent uncertainty in natural language processing.

1. Distant Supervision

Instead of manually labeling thousands of sentences, the team used "distant supervision." They cross-referenced the IntAct database with PubMed abstracts. If a database entry indicated that Protein A modifies Protein B, and an abstract mentioned both proteins along with a PTM keyword, the system automatically labeled that abstract as a positive training example. This allows for the creation of massive datasets without the cost of manual annotation.

Example of the distant supervision pipeline, showing the mapping of gene mentions to NCBI and Uniprot identifiers within a normalized abstract.

2. BioBERT Ensemble (PPI-BioBERT-x10)

The core model is based on BioBERT, a version of the BERT language model pre-trained on large-scale biomedical corpora. To improve reliability and handle the "noisy" nature of distantly supervised data, the authors trained an ensemble of ten models.

3. Confidence Calibration

Deep learning models are often "overconfident," providing high probability scores even for incorrect predictions. By using an ensemble, the researchers could measure the "agreement" between models. High-quality predictions were defined as those where all ten models produced high scores with low variance (standard deviation). This calibration helps filter out ambiguous text where the model might otherwise guess incorrectly.

Key Results and Real-World Performance

When evaluated on a standard test set, the model achieved a modest F1-micro score of 41.3. However, the true test was applying it to the entire PubMed corpus of 18 million abstracts.

  • Extraction Scale: The model identified 1.6 million raw PTM-PPI predictions across the corpus.
  • High-Confidence Filtering: By applying strict confidence and variance thresholds, the team narrowed this down to 5,708 high-confidence interactions (representing 4,584 unique triplets).
  • The Generalization Gap: Human evaluation of a random sample revealed a significant "precision drop" in the wild, falling to 33.7%. This highlights a "generalization gap" where models learn specific quirks of the training data that do not always translate to the diverse language used across all of PubMed.
  • The Power of Consensus: Precision was significantly improved (to 58.8%) by requiring "consensus" — only accepting predictions that appeared in multiple different papers.

Conclusion

This work highlights both the immense potential and the persistent challenges of using AI for biological text mining. While PPI-BioBERT-x10 identified 200% more PTM-PPIs than previous rule-based systems, the gap between lab performance and real-world accuracy remains a hurdle. Ultimately, these tools are best viewed as "human augmentation" systems, helping curators prioritize the most relevant literature in an ever-expanding sea of data.

Summary updated

Abstract

Abstract Motivation: Protein-protein interactions (PPIs) are critical to normal cellular function and are related to many disease pathways. A range of protein functions are mediated and regulated by protein interactions through post-translational modifications (PTM). However, only 4% of PPIs are annotated with PTMs in biological knowledge databases such as IntAct, mainly performed through manual curation, which is neither time- nor cost-effective. Here we aim to facilitate annotation by extracting PPIs along with their pairwise PTM from the literature by using distantly supervised training data using deep learning to aid human curation. We further assessed model generalisation in a real-world scenario, evaluating its performance on a randomly sampled subset of predictions from 18 million PubMed abstracts. Method: We use the IntAct PPI database to create a distant supervised dataset annotated with interacting protein pairs, their corresponding PTM type, and associated abstracts from the PubMed database. We train an ensemble of BioBERT models – dubbed PPI-BioBERT-x10 – to improve confidence calibration. We extend the use of ensemble average confidence approach with confidence variation to counteract the effects of class imbalance to extract high confidence predictions. Results and conclusion: The PPI-BioBERT-x10 model evaluated on the test set resulted in a modest F1-micro 41.3 (P=58.1, R=32.1). However, by combining high confidence and low variation to identify high quality predictions, tuning the predictions for precision, we retained 19% of the test predictions with 100% precision. We evaluated PPI-BioBERT-x10 on 18 million PubMed abstracts and extracted 1.6 million (546507 unique PTM-PPI triplets) PTM-PPI predictions, and filter ≈5,700 (4584 unique) high confidence predictions. Of the 5700, human evaluation on a small randomly sampled subset shows that the precision drops to 33.7% despite confidence calibration and highlights the challenges of generalisability beyond the test set even with confidence calibration. We circumvent the problem by only including predictions associated with multiple papers, improving the precision to 58.8%. In this work, we highlight the benefits and challenges of deep learning-based text mining in practice, and the need for increased emphasis on confidence calibration to facilitate human curation efforts.
BibTeX
@inproceedings{elangovan2021large,
  author = {Aparna Elangovan and Yuan Li and Douglas E. V. Pires and Melissa J. Davis and Karin Verspoor},
  title = {{Large-Scale Protein-Protein Post-Translational Modification Extraction With Distant Supervision And Confidence Calibrated BioBERT}},
  booktitle = {Research},
  year = {2021},
  doi = {10.21203/rs.3.rs-898489/v1},
}
Figure