An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining

Yifan Peng, Qingyu Chen, Zhiyong Lu

TL;DR

This paper explores how to effectively train a single BERT model to perform multiple biomedical and clinical text mining tasks simultaneously. They demonstrate that this multi-task learning (MTL) approach, when properly refined and fine-tuned, can outperform individual task-specific models, achieving state-of-the-art results on several benchmarks and offering a more efficient way to build models for new problems.

What Problem Does This Paper Address?

Deep learning models, especially large transformer models like BERT, are incredibly powerful for natural language processing (NLP). However, they are notoriously "data-hungry," requiring vast amounts of labeled data for each specific task. In specialized domains like biomedical and clinical text mining, obtaining such large, task-specific datasets can be challenging and expensive. Multi-task learning (MTL) offers a potential solution by allowing a single model to learn from multiple related tasks simultaneously, leveraging shared knowledge to improve performance and generalization, especially when data is scarce. While MTL has shown promise in general NLP, its application and effectiveness across diverse biomedical and clinical tasks, particularly with advanced models like BERT, needed a more thorough empirical investigation. This paper aims to fill that gap by systematically studying MTL with BERT on a range of biomedical and clinical NLP tasks.

Methods

The authors propose an MT-BERT model architecture that builds upon the pre-trained BERT model. The core idea is to share the BERT encoder layers across all tasks, allowing it to learn a general representation of the text. On top of these shared layers, they introduce task-specific layers (decoders) for each individual task.

Their training strategy involves three stages:

  1. Pre-training BERT: This is the standard BERT pre-training process using masked language modeling.
  2. Refining via Multi-task Learning (MT-BERT-Refinement): Here, the entire BERT model (shared layers) and the task-specific layers are updated simultaneously using a combined objective function that considers all tasks. This stage aims to learn a generalized MTL model.
  3. Fine-tuning (MT-BERT-Fine-Tune): After the refinement stage, the model is further fine-tuned on each individual task using its specific dataset, but with a smaller learning rate. This step allows the model to specialize further for each task while retaining the benefits of MTL.

They also investigate "pairwise MTL" to understand how specific task combinations affect each other, identifying which tasks help or hinder others.

Datasets

The study utilizes datasets from the Biomedical Language Understanding Evaluation (BLUE) benchmark, which comprises 8 tasks across diverse text genres (biomedical and clinical). These tasks include:

  • Clinical Tasks:
    • ClinicalSTS (Sentence Similarity)
    • i2b2 2010 (Relation Extraction)
    • MedNLI (Inference)
    • ShARe/CLEFE (Named Entity Recognition)
  • Biomedical Tasks:
    • BC5CDR disease (Named Entity Recognition)
    • BC5CDR chemical (Named Entity Recognition)
    • DDI (Relation Extraction)
    • ChemProt (Relation Extraction)

Key Insights and Ideas

  • MTL Boosts Performance: Jointly training BERT on multiple biomedical and clinical tasks leads to significant performance improvements, often surpassing single-task fine-tuned BERT models.
  • Refinement and Fine-tuning Synergy: A two-stage approach (MTL refinement followed by task-specific fine-tuning) yields the best results, suggesting that initial joint learning helps generalize, while subsequent fine-tuning allows for task-specific optimization.
  • Task Relationships Matter: Not all task combinations are beneficial. Pairwise MTL analysis reveals that some tasks can improve others, while some can even degrade performance, highlighting the importance of understanding task dependencies.
  • Domain-Specific Pre-training is Crucial: Using BERT models pre-trained on domain-specific corpora (e.g., BlueBERT for clinical, BioBERT for biomedical) as a starting point for MTL is more effective than using general-domain BERT.
  • Efficiency for New Problems: The MT-BERT-Fine-Tune model, trained on multiple tasks, can be a valuable starting point for researchers facing new problems with limited labeled data, as it already possesses generalized features.

Key Results

  • The MT-BERT-Fine-Tune model achieved state-of-the-art results on average across the BLUE benchmark, with an average improvement of 2.0% in the biomedical domain and 1.3% in the clinical domain compared to single-task BERT models.
  • Specifically, MT-BERT-Fine-Tune achieved new state-of-the-art results on four BLUE benchmark tasks.
  • On the DDI task (biomedical relation extraction), the model saw a significant improvement of 8.1%.
  • In clinical tasks, MT-BlueBERT-Fine-Tuneclinical pushed the benchmark to 81.9% on average, a 1.3% absolute improvement over its single-task counterpart.
  • The study also showed that using clinical notes during the pre-training phase of BERT (e.g., ClinicalBERT vs. BlueBERT) significantly impacts performance on clinical tasks.

Limitations and Future Work

  • MTL Effectiveness Varies: The paper acknowledges that MTL is not always effective, and the success depends on the specific task relationships and data characteristics. They are interested in further exploring what makes MTL successful.
  • Task Relationship Characterization: A deeper understanding of task relationships is needed to predict when MTL will be beneficial.
  • Alternative MTL Approaches: The study focused on sharing the encoder with task-specific decoders. Future work could explore other MTL strategies like soft parameter sharing or knowledge distillation.
  • Domain Adaptation: While the study shows benefits of MTL for domain adaptation, further investigation into training separate deep neural networks for different text genres might be beneficial.

Summary updated

BibTeX
@inproceedings{peng2020empirical,
  author = {Yifan Peng and Qingyu Chen and Zhiyong Lu},
  title = {{An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining}},
  booktitle = {BioNLP},
  year = {2020},
  url = {https://openreview.net/forum?id=wRx0ZTWNTyX},
}
Figure