CAISA@SMM4H'22: Robust Cross-Lingual Detection of Disease Mentions on Social Media with Adversarial Methods

Akbar Karimi, Lucie FlekSMM4H @ COLING 2022 · OpenReview

TL;DR

This paper tackles the challenge of detecting disease mentions in Spanish social media text using an English-trained model. They demonstrate that adversarial training and data augmentation techniques can significantly improve robustness, especially in low-data scenarios, and surprisingly, a cross-lingual approach using an English model can be highly effective.

What Problem Does This Paper Address?

Social media is a treasure trove of information about public health, offering insights into disease trends, public perception, and drug effects. Automatically extracting this information, specifically identifying disease mentions, is crucial for informed decision-making. However, building robust models for this task, especially across different languages, is challenging. Many existing approaches rely on models trained in the same language as the target data. This paper investigates whether a powerful pre-trained model from one language (English) can be effectively adapted for a task in another language (Spanish) and how adversarial methods can enhance its performance, particularly when data is scarce.

Methods

The authors propose an ensemble system that leverages several techniques to improve disease mention detection. The core of their approach is the English BioBERT model, a BERT-based model pre-trained on both general and biomedical text. To enhance its robustness, they employ two main adversarial strategies:

  • Adversarial Training (AT): This technique generates "adversarial examples" in the embedding space. By training the model on these slightly perturbed examples, it becomes more resilient to small variations in the input, making it more robust.
  • AEDA (Adversarial Data Augmentation): This method artificially expands the training data by inserting punctuation marks randomly into sentences. This creates more training instances, which can be particularly beneficial in low-data settings.

Additionally, they incorporate PSUM (Parallel Aggregation), a module that adds an extra BERT layer to improve performance by aggregating losses. Their final system combines these methods, often pairing BioBERT with AT and PSUM, and also uses AEDA as a data augmentation technique. For evaluation, they convert the BIO (Beginning, Inside, Outside) tagging scheme to a character span format and vice-versa, noting potential performance losses due to this conversion.

Datasets

The primary dataset used in this study is from the SocialDisNER challenge at SMM4H'22. This dataset consists of Spanish tweets annotated for disease mentions. The organizers provided an initial 5K tweet training set, which was later expanded to 90K for the full training set. The development (dev) and test sets contained 2.5K and 23.5K tweets, respectively. The paper notes that "Covid19" was the most frequently discussed disease in the dataset.

Key Insights and Ideas

  • Cross-Lingual Transfer is Viable: Surprisingly, the English BioBERT model, even without explicit cross-lingual training, achieved strong performance on the Spanish dataset. This highlights the power of pre-trained models and suggests that the medical domain might have significant lexical overlap between English and Spanish.
  • Adversarial Methods Shine in Low-Data Regimes: While the full 90K Spanish dataset was large enough for the English BioBERT to perform well on its own, adversarial training and data augmentation techniques showed substantial improvements when the training data was very limited (100 or 1000 tweets). This indicates their value for resource-scarce scenarios.
  • Augmentation Benefits Diminish with More Data: The authors observed that the benefits of data augmentation methods like AEDA became less pronounced when applied to the larger 90K training set, suggesting that the model had already learned sufficient patterns from the extensive data.
  • Heuristic Baselines are Outperformed: Both the proposed adversarial methods and the English BioBERT model significantly outperformed a simple heuristic vocabulary-based baseline, underscoring the effectiveness of deep learning approaches.

Key Results

  • English BioBERT Dominates: On the full 90K Spanish dataset, the English BioBERT model achieved the best recall, outperforming data augmentation methods.
  • Significant Gains with Limited Data: In low-data settings (100 and 1000 training tweets), adversarial methods like PSUM and AT, especially when combined with AEDA, showed notable improvements. For instance, with 100 tweets, PSUM improved the F1 score to 66.8% from BioBERT's 61.1%, and AT+AEDA reached 74.4% F1 with 1000 tweets.
  • Error Analysis: The analysis revealed that false positives were more common than false negatives. Top misclassified tokens were often prepositions rather than actual disease mentions, suggesting that subtle linguistic nuances can still pose a challenge.

Limitations and Future Work

  • Cross-Lingual Transfer Assumptions: The success of the English BioBERT model relies on the assumption of significant lexical overlap in the medical domain between English and Spanish. This might not hold true for all language pairs or domains.
  • Data Conversion Impact: The conversion between BIO and character span formats introduced some performance loss, indicating that a more seamless integration of these formats could be beneficial.
  • Future Directions: The authors suggest that their adversarial and augmentation strategies could be applied to other languages, provided there are some lexical similarities. Further research could explore more sophisticated cross-lingual transfer techniques and investigate the impact of these methods on different types of social media content and disease mentions.

Summary updated

Abstract

We propose adversarial methods for increasing the robustness of disease mention detection on social media. Our method applies adversarial data augmentation on the input and the embedding spaces to the English BioBERT model. We evaluate our method in the SocialDisNER challenge at SMM4H’22 on an annotated dataset of disease mentions in Spanish tweets. We find that both methods outperform a heuristic vocabulary-based baseline by a large margin. Additionally, utilizing the English BioBERT model shows a strong performance and outperforms the data augmentation methods even when applied to the Spanish dataset, which has a large amount of data, while augmentation methods show a significant advantage in a low-data setting.
BibTeX
@inproceedings{karimi2022caisa,
  author = {Akbar Karimi and Lucie Flek},
  title = {{CAISA@SMM4H'22: Robust Cross-Lingual Detection of Disease Mentions on Social Media with Adversarial Methods}},
  booktitle = {SMM4H @ COLING},
  year = {2022},
  url = {https://openreview.net/forum?id=a6gCyrEYnM},
}
Figure