An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining

Yifan Peng, Qingyu Chen, Zhiyong Lu

TL;DR

This paper explores the effectiveness of Multi-Task Learning (MTL) with BERT for biomedical and clinical text mining tasks. They demonstrate that a jointly trained BERT model (MT-BERT) can outperform single-task BERT models by up to 2% on average, offering a more efficient way to achieve state-of-the-art results across various NLP tasks in these specialized domains.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has seen tremendous success with transformer models like BERT. However, applying these models to specialized domains like biomedical and clinical text mining often requires extensive labeled data for each specific task (e.g., named entity recognition, relation extraction, text similarity). This can be a significant bottleneck, especially when datasets are scarce. Multi-Task Learning (MTL) offers a potential solution by allowing a single model to learn multiple related tasks simultaneously, leveraging shared representations to improve generalization and reduce data requirements. While MTL has shown promise in general NLP, its application and effectiveness across a diverse set of biomedical and clinical tasks, particularly with state-of-the-art transformer architectures, needed a thorough empirical investigation.

Methods

The core of this work is the development and evaluation of a Multi-Task BERT (MT-BERT) model. The architecture consists of a shared BERT encoder that processes input text and generates contextualized embeddings. On top of this shared encoder, task-specific layers (decoders) are added for each NLP task. The training process involves three stages:

  1. Pretraining: Standard BERT pretraining using masked language modeling.
  2. MTL Refinement: All layers of the BERT model are refined using a multi-task objective, where mini-batches from different tasks are merged and processed. This stage aims to create a general-purpose biomedical/clinical NLP model.
  3. Task-Specific Fine-tuning: The refined MT-BERT model is then further fine-tuned on individual tasks using a smaller learning rate. This stage aims to optimize performance for each specific task.

The authors also investigate "pairwise MTL" to understand how specific task combinations influence each other.

Datasets

The study utilizes the Biomedical Language Understanding Evaluation (BLUE) benchmark, which comprises 8 tasks across diverse text genres in both biomedical and clinical domains. The specific tasks and their domains are:

  • Clinical Tasks:
    • ClinicalSTS (Sentence similarity)
    • i2b2 2010 re (Relation extraction)
    • MedNLI (Inference)
    • ShARe/CLEFE (Named entity recognition)
  • Biomedical Tasks:
    • ChemProt (Relation extraction)
    • DDI (Relation extraction)
    • BC5CDR disease (Named entity recognition)
    • BC5CDR chemical (Named entity recognition)

Key Insights and Ideas

  • MTL Boosts Performance: Jointly training BERT on multiple biomedical and clinical tasks leads to significant performance improvements compared to single-task BERT models.
  • Domain Specialization Matters: Models pre-trained on biomedical literature perform better on biomedical tasks, while models trained on both biomedical and clinical data excel in clinical tasks. This suggests that separate deep neural networks might be beneficial for different text genres in BioNLP.
  • MTL Refinement is Valuable: Even after task-specific fine-tuning, the MTL refinement stage provides a benefit, suggesting that shared representations learned through MTL are robust.
  • Understanding Task Relationships is Crucial: Pairwise MTL analysis reveals which tasks benefit from or harm others, providing valuable insights for researchers when selecting models or designing new MTL setups. For instance, NER tasks that require more data can benefit from being trained with other tasks.

Key Results

  • The MT-BERT fine-tuned models achieved over 2.0% average performance improvement in the biomedical domain and 1.3% in the clinical domain compared to state-of-the-art single-task BERT models.
  • MT-BERT achieved new state-of-the-art results on four BLUE benchmark tasks.
  • For example, on the clinical tasks, MT-BlueBERT-Fine-Tuneclinical pushed the benchmark to 81.9%, a 1.3% improvement over its baseline.
  • In biomedical tasks, MT-BlueBERT-Fine-Tunebiomedical achieved a 2.0% absolute improvement over its baseline, reaching 83.6% on one task.
  • The DDI task saw the largest improvement in the biomedical domain, with an 8.1% gain.

Limitations and Future Work

The authors acknowledge two main limitations:

  • MTL Effectiveness Varies: MTL is not universally effective across all task combinations. Further research is needed to characterize task relationships and identify data characteristics that predict MTL success.
  • Architectural Exploration: The study focused on a specific MTL architecture (shared encoder, task-specific decoders). Future work should explore other MTL approaches, such as fine-tuning only task-specific layers, soft parameter sharing, and knowledge distillation.

The authors suggest that future work should investigate these alternative MTL strategies and delve deeper into understanding the nuanced relationships between tasks to build more generalizable and efficient NLP models for the biomedical and clinical domains.

Summary updated

Abstract

Multi-task learning (MTL) has achieved remarkable success in natural language processing applications. In this work, we study a multi-task learning model with multiple decoders on varieties of biomedical and clinical natural language processing tasks such as text similarity, relation extraction, named entity recognition, and text inference. Our empirical results demonstrate that the MTL fine-tuned models outperform state-of-the-art transformer models (e.g., BERT and its variants) by 2.0% and 1.3% in biomedical and clinical domains, respectively. Pairwise MTL further demonstrates more details about which tasks can improve or decrease others. This is particularly helpful in the context that researchers are in the hassle of choosing a suitable model for new problems. The code and models are publicly available at https://github.com/ncbi-nlp/bluebert
BibTeX
@inproceedings{peng2020empirical,
  author = {Yifan Peng and Qingyu Chen and Zhiyong Lu},
  title = {{An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining}},
  booktitle = {BioNLP},
  year = {2020},
  url = {https://openreview.net/forum?id=wRx0ZTWNTyX},
}
Figure