CU-UD: text-mining drug and chemical-protein interactions with ensembles of BERT-based models

Mehmet Efruz Karabulut, K. Vijay-Shanker, Yifan PengarXiv 2021 · OpenReview

TL;DR

This paper presents CU-UD, an ensemble system for automatically identifying drug-protein interactions from biomedical text. By combining multiple BERT-based language models through majority voting and a meta-learner, their approach significantly outperforms the average participant in the BioCreative VII DrugProt challenge, demonstrating the power of ensemble methods for this complex task.

What Problem Does This Paper Address?

The paper tackles the crucial task of relation extraction in the biomedical domain, specifically focusing on identifying interactions between chemical compounds (drugs) and genes/proteins. This is vital for advancing drug discovery, database curation, and question-answering systems. The BioCreative VII DrugProt challenge aims to push the boundaries of automated systems for this task, which is challenging due to the complexity of biological language and the need for high accuracy. Existing methods often struggle to capture the nuances of these interactions, necessitating more robust approaches.

Methods

The core of CU-UD is an ensemble system built upon multiple BERT-based language models. The authors explored several strategies to achieve diversity among their ensemble members:

  • Diverse BERT Architectures: They utilized three distinct BERT models:
  • BioBERT: Pre-trained on PubMed and PMC abstracts.
  • PubMedBERT: Pre-trained solely on PubMed abstracts from scratch.
  • BioM-ELECTRA: Based on the ELECTRA architecture, pre-trained on PubMed abstracts. These models differ in their pre-training data, vocabulary, and learning objectives, contributing to ensemble diversity.

  • Fine-tuning Strategies: For relation extraction, they experimented with different ways to leverage BERT's output:

  • Using the [CLS] token's final layer output (aggregating sentence information).
  • Applying a Long Short-Term Memory (LSTM) layer on top of the last layer.
  • Employing an attention mechanism across all units of the last layer.
  • Entity Tagging Schemes: Two schemes were used: one anonymizing entities with generic tags (<DRUG>, <PROTEIN>) and another using explicit start/end markers (<DRUG-B>, <DRUG-E>).
  • Handling Class Imbalance: A class-weighted loss function was implemented to address the significant imbalance between positive and negative relation instances in the dataset.
  • Ensemble Fusion Techniques: Two primary methods were used to combine the predictions of individual models:
  • Majority Voting: The most straightforward approach, selecting the relation type with the most votes.
  • Meta-Learning (MLP): A Multilayer Perceptron (MLP) was trained as a meta-learner to combine the outputs of individual models, learning optimal fusion strategies.

Datasets

The primary dataset used is the DrugProt corpus, manually annotated from 5,000 PubMed abstracts. This dataset was split into:

  • Training set: 3,500 abstracts.
  • Development set: 750 abstracts.
  • Test set: 750 abstracts.

The paper also mentions that for training, a 70-20-10 split (training-development-ensemble training) was used, and this process was repeated five times for some runs to ensure robust hyperparameter tuning and ensemble model training.

Key Insights and Ideas

  • Ensembles are Key: The paper strongly supports the hypothesis that ensemble systems, by combining diverse models, achieve superior performance compared to single models for complex biomedical relation extraction tasks.
  • BERT's Power in Biomedical NLP: The continued success of BERT-based models (BioBERT, PubMedBERT, BioM-ELECTRA) highlights their effectiveness in understanding the nuances of biomedical text.
  • Diversity Matters: The choice of different pre-training strategies and architectures for the individual BERT models was crucial for the ensemble's success.
  • Handling Imbalance is Crucial: The use of class-weighted loss functions was effective in mitigating the negative impact of a large number of non-interacting entity pairs.
  • Surprising Extraction Performance: The system achieved a remarkably high F1 score for the ANTAGONIST relation, which is one of the least frequent in the dataset, suggesting that even with limited examples, sophisticated models can learn to extract these specific interactions effectively.

Key Results

The CU-UD system achieved impressive results in the BioCreative VII DrugProt task:

  • Overall Performance: The system obtained a Precision of 0.7708, Recall of 0.7770, and an F1 score of 0.7739 on the official test set.
  • Comparison to Participants: This F1 score was substantially better than the average F1 score of participant teams (0.6196).
  • Ensemble vs. Single Model: The ensemble approaches (Runs 3 and 4) consistently outperformed single models (Run 5, BioM-ELECTRAL), demonstrating the benefit of combining predictions.
  • Individual Relation Performance: The system achieved its best performance on the ANTAGONIST relation (F1 of 0.8902) and the INHIBITOR relation (F1 of 0.8570).

Limitations and Future Work

  • Limitations:
    • The paper notes that for some runs, the ensemble training used only 10% of the data, which might limit the full potential of the meta-learner.
  • While the system performs well, further error analysis is needed to understand why certain relations (like ANTAGONIST) are extracted so effectively with few examples, and to improve performance on less frequent relations.
  • Future Work:
    • The authors suggest exploring more sophisticated ensemble techniques and architectures.
  • Investigating methods to further enhance diversity among ensemble members could lead to even better performance.
  • Expanding the scope to include interactions beyond sentence boundaries could be a valuable next step.

Summary updated

Abstract

Identifying the relations between chemicals and proteins is an important text mining task. BioCreative VII track 1 DrugProt task aims to promote the development and evaluation of systems that can automatically detect relations between chemical compounds/drugs and genes/proteins in PubMed abstracts. In this paper, we describe our submission, which is an ensemble system, including multiple BERT-based language models. We combine the outputs of individual models using majority voting and multilayer perceptron. Our system obtained 0.7708 in precision and 0.7770 in recall, for an F1 score of 0.7739, demonstrating the effectiveness of using ensembles of BERT-based language models for automatically detecting relations between chemicals and proteins. Our code is available at https://github.com/bionlplab/drugprot_bcvii.
BibTeX
@misc{karabulut2021cu,
  author = {Mehmet Efruz Karabulut and K. Vijay-Shanker and Yifan Peng},
  title = {{CU-UD: text-mining drug and chemical-protein interactions with ensembles of BERT-based models}},
  year = {2021},
  eprint = {2112.03004},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.03004},
}
Figure