BCH-NLP at BioCreative VII Track 3: medications detection in tweets using transformer networks and multi-task learning

Dongfang Xu, Shan Chen, Timothy MillerarXiv 2021 · OpenReview

TL;DR

This paper tackles the challenge of automatically identifying medication and dietary supplement mentions in noisy social media text. They developed a multi-task learning model combining text classification and sequence labeling, achieving a top score in the BioCreative VII competition by outperforming the average participant by over 10 F1 points. Their work highlights the effectiveness of ensemble methods, multi-task learning, and data augmentation for this task.

What Problem Does This Paper Address?

Social media platforms like Twitter are rich sources of real-time health information, but the text is often messy. It's filled with jargon, misspellings, abbreviations, and other informal language, making it difficult for traditional Natural Language Processing (NLP) techniques to accurately extract specific information. This paper focuses on a critical sub-problem: automatically identifying mentions of medications and dietary supplements within these tweets. This is important for various applications, such as pharmacovigilance (monitoring drug safety) and understanding public health trends. The specific challenge is to accurately pinpoint these mentions in a highly imbalanced dataset where drug mentions are rare, and the text is inherently noisy.

Methods

The core of their approach is a multi-task learning model built upon a shared transformer network. This model is designed to perform two tasks simultaneously:

  1. Binary Classification: A classifier determines whether a tweet contains any medication or dietary supplement mention. This uses the representation of the special <CLS> token from the transformer.
  2. Sequence Labeling: A classifier identifies the exact spans of medication/supplement mentions within the tweet. This uses the representations of all tokens.

By jointly training these tasks, the model can leverage shared representations and context, potentially improving performance on both. They also employ an ensemble technique, combining the predictions of eight different pre-trained transformer models (including BERT variants, BioBERT, BioRedditBERT, and PubMedBERT) to further boost performance. Data augmentation strategies were crucial for handling the imbalanced nature of the dataset.

Datasets

The primary dataset used is from the BioCreative VII Track 3 shared task (2021 dataset), which consists of tweets from Twitter users during their pregnancy. This dataset is characterized by a natural, highly imbalanced distribution of drug mentions (around 0.2%).

To address data imbalance and improve robustness, they augmented the training data by:

  • Including the 2018 SMM4H shared task dataset, which was used for binary classification and had a more balanced distribution but less granular annotations.
  • Extracting additional tweets mentioning medications from Twitter using keyword searches based on frequent drug mentions from both the 2021 and 2018 datasets. They filtered these tweets to be shorter than 128 characters, mimicking the typical length in the 2021 dataset.
  • A further augmentation step involved manually annotating drug use categories and replacing drug mentions in extracted tweets with similar drugs from the same categories.

Key Insights and Ideas

  • Multi-task learning is beneficial: Jointly training text classification and sequence labeling for medication detection proved more effective than a single-task approach, suggesting that understanding if a tweet mentions a drug helps in identifying where those mentions are.
  • Ensembling is powerful: Combining predictions from multiple transformer models significantly improved performance, indicating that different models capture different aspects of the data.
  • Data augmentation is crucial for imbalance: Strategies like incorporating older datasets and actively extracting more relevant tweets were vital for overcoming the scarcity of positive examples.
  • Case-insensitivity helps: Models that ignored case information generally performed better, suggesting that the specific casing of drug names in tweets is less important than their presence and context.
  • Biomedical pre-training matters: Models pre-trained on biomedical texts (like BioBERT) showed an advantage, as these terms are often strong indicators of drug mentions.

Key Results

The authors' best system run achieved a strict F1 score of 80.4 on the test set. This performance was:

  • 4.6 points higher than the baseline system.
  • 10.6 points higher than the average score of all participants.
  • This result ranked them first in the competition.

Table II provides a detailed breakdown of individual model performances across different data combinations, showing that the ensemble system consistently outperformed individual models. Table III further demonstrates that the multi-task objective (text classification + sequence labeling) yielded better results than sequence labeling alone for their top models.

Limitations and Future Work

  • Data Imbalance: While data augmentation helped, the inherent imbalance of social media data remains a challenge. The authors note that upsampling or downsampling techniques on the 2021 dataset alone resulted in worse performance and sensitivity to hyperparameters.
  • Ambiguous Mentions: The 2018 dataset contained ambiguous mentions that could be confused with medications. While incorporating this dataset helped precision for some models, it highlights the difficulty in distinguishing true drug mentions from similar terms.
  • Future Work: The paper implicitly suggests that further exploration of data augmentation techniques, more sophisticated ensemble strategies, and potentially domain-specific pre-training on even larger or more diverse biomedical text corpora could lead to further improvements. The success of the multi-task approach also opens avenues for exploring other related NLP tasks in conjunction with medication detection.

Summary updated

Abstract

In this paper, we present our work participating in the BioCreative VII Track 3 - automatic extraction of medication names in tweets, where we implemented a multi-task learning model that is jointly trained on text classification and sequence labelling. Our best system run achieved a strict F1 of 80.4, ranking first and more than 10 points higher than the average score of all participants. Our analyses show that the ensemble technique, multi-task learning, and data augmentation are all beneficial for medication detection in tweets.
BibTeX
@misc{xu2021bch,
  author = {Dongfang Xu and Shan Chen and Timothy Miller},
  title = {{BCH-NLP at BioCreative VII Track 3: medications detection in tweets using transformer networks and multi-task learning}},
  year = {2021},
  eprint = {2111.13726},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.13726},
}
Figure