Text Augmentation Techniques in Drug Adverse Effect Detection Task

Pavel BlinovNAACL 2021

TL;DR

This paper explores text augmentation techniques to improve drug adverse effect detection from social media. By translating existing text and generating new text using language models, they significantly increased their training data, aiming to boost classifier performance on this challenging NLP task. While augmentation showed promise in cross-validation, its impact on the final test set was less pronounced, highlighting the need for further research into robust augmentation strategies.

What Problem Does This Paper Address?

Detecting drug adverse effects (AEs) from informal text sources like social media is a crucial but difficult task in Natural Language Processing (NLP). Existing state-of-the-art models struggle to differentiate between actual AEs and mentions of drug use or indications for use, especially when dealing with noisy, user-generated content. The performance of deep learning models, particularly transformer-based ones, is heavily reliant on the size and quality of the training data. This paper tackles the challenge of insufficient training data for Russian-language drug AE detection by investigating text augmentation techniques.

Methods

The authors employed two primary text augmentation techniques:

  • Translation Augmentation: They utilized back-translation, a common technique in machine translation. Specifically, they translated existing English data (from a previous workshop iteration) into Russian and then back into English. This process creates slightly varied versions of the original text, aiming to introduce diversity without drastically altering the meaning.
  • Generation Augmentation: They leveraged generative pre-trained language models, specifically fine-tuning a GPT-2 model for Russian tweet generation. By providing random starting tokens, the model generates plausible and coherent text fragments. These generated messages were then labeled using their existing model, and high-confidence samples were selected to augment the training set.

For the core classification task, they used the RuBERT model, a Russian-language BERT variant. This model was fine-tuned on a combination of real and augmented data using a mean pooling strategy over token embeddings and a binary cross-entropy loss function. They trained five models using cross-validation and ensembled their predictions for the final output.

Datasets

The primary datasets used were the Train and Dev sets released by the SMM4H workshop organizers, containing user messages from Twitter with corresponding labels.

  • Original Data:
    • Train: 8,184 samples
    • Dev: 3,425 samples
    • Test: 9,095 samples
  • Augmented Data:
    • Augm_Transl: 25,678 samples generated via translation.
  • Augm_Gen: 51,152 samples generated via language model.
  • Total Data: 97,534 samples (including original and augmented).

The augmented data (Augm_* rows) were added to the training set for model fine-tuning.

Key Insights and Ideas

  • Augmentation's Double-Edged Sword: While text augmentation techniques like translation and generation can significantly increase the volume of training data, their effectiveness in improving downstream task performance isn't guaranteed. The paper highlights that simply increasing data size doesn't automatically translate to better generalization.
  • Data Distribution Shift: The authors observed a potential shift in data distribution after augmentation. Their t-SNE plot of tweet embeddings suggests that synthetically generated messages might occupy different regions of the feature space compared to real data, potentially leading to a performance gap.
  • NLP Augmentation Challenges: Unlike image augmentation, which has well-established and effective techniques, NLP augmentation is more complex. Simple word manipulations can easily alter text meaning, necessitating the use of sophisticated language models to ensure generated or translated text remains semantically sound and grammatically correct.

Key Results

  • Data Volume Increase: The augmentation techniques successfully increased the training dataset size by over 9 times, from 8,184 samples to 97,534 samples in total.
  • Cross-Validation Improvement: The models trained with augmented data showed improved performance metrics during cross-validation compared to the model trained on real data alone. For instance, the Real+Augm_* system achieved an F1 score of 56.6±2.5, outperforming the Real system's 55.4±2.2.
  • Test Set Performance Lag: Despite cross-validation gains, the augmented systems did not outperform the baseline on the test set. In fact, their best system was inferior to the median performance of participants in the SMM4H workshop, only excelling in Recall by a small margin. This suggests that the augmentation strategies, while increasing data quantity, may not have effectively captured the nuances required for robust generalization on unseen data.

Limitations and Future Work

  • Data Distribution Mismatch: The authors hypothesize that a significant shift in data distribution between the training (including augmented data) and test sets is a primary reason for the lack of test performance improvement.
  • Augmentation Strategy Refinement: The paper concludes that while text data augmentation is a promising avenue for obtaining more data, its successful practical application for AE detection requires further research. Future work should focus on developing more sophisticated and robust augmentation techniques that better preserve data distribution and semantic integrity.
  • Investigating Augmentation Impact: Further experiments are needed to understand precisely why certain augmentation strategies fail to translate to test performance and to develop methods that mitigate distribution shifts.

Summary updated

BibTeX
@inproceedings{blinov2021text,
  author = {Pavel Blinov},
  title = {{Text Augmentation Techniques in Drug Adverse Effect Detection Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure