AIR-JPMC@SMM4H'22: BERT + Ensembling = Too Cool: Using Multiple BERT Models Together for Various COVID-19 Tweet Identification Tasks

Leung Wai Liu, Akshat Gupta, Saheed Obitayo, Xiaomo Liu, Sameena ShahSMM4H @ COLING 2022 · OpenReview

Abstract

This paper details a winning strategy for the Social Media Mining of Health (SMM4H) 2022 competition, focusing on identifying adverse drug events (ADEs) and public sentiment toward COVID-19 mandates in tweets. By leveraging an ensemble of fine-tuned BERT and RoBERTa models, the authors consistently outperformed competition averages, proving that even in the era of massive singular models, strategic ensembling remains a highly effective tool for processing noisy social media data.

Introduction

Social media platforms like Twitter provide a massive, real-time stream of public sentiment and health-related experiences, but the data is notoriously "messy." Tweets are informal, lack uniformity, and are filled with slang, making it difficult to automate tasks like pharmacovigilance (detecting side effects of drugs) or stance detection (understanding if the public supports or opposes health mandates like mask-wearing).

The authors tackle two specific challenges from the SMM4H 2022 Shared Tasks:

  1. Task 1 (Pharmacovigilance): Classifying tweets that mention Adverse Drug Events (ADEs), pinpointing the specific text span describing the effect, and mapping that effect to standardized medical terminology (MedDRA).
  2. Task 2 (Health Mandates): Determining the public's stance (Favor, Against, or None) and the presence of a premise (reasoning) regarding COVID-19 mandates like school closures and stay-at-home orders.

Methods

The core of the approach is Model Ensembling. Rather than relying on a single "best" model, the team combined the strengths of multiple architectures.

Base Models and Fine-Tuning

The system utilized BERT and RoBERTa in both base and large versions. RoBERTa was chosen specifically because it builds upon BERT’s architecture with optimized training tweaks that often lead to better performance on downstream tasks. The models were fine-tuned on task-specific data using standard sequence classification heads for classification and BERTForTokenClassification for Task 1b (Span Detection).

Comparison of F1-scores across different base model architectures for Tasks 1a, 2a, and 2b.

As shown in the preliminary model evaluations, "Large" versions of BERT and RoBERTa significantly outperformed their "Base" counterparts, justifying the extra computational cost for these specific NLP tasks.

Ensembling Strategies

Performance gains were achieved by testing three ensembling methods: majority-vote, unweighted average of prediction scores, and weighted average (giving more "voting power" to models with higher validation F1-scores). The final submission utilized an ensemble of 10 "Large" models.

For span detection, a custom post-processing script converted token-level predictions back into character-level indices in the original tweet, handling complexities like partial words and punctuation.

Datasets

The study utilized two primary datasets provided by the competition organizers:

  • DeepADEMiner (Version 2): Used for Task 1. This dataset is highly imbalanced, with significantly more "noADE" tweets than those containing actual adverse events.
  • Stance Detection Dataset: Used for Task 2. This dataset covers three specific mandates: face masks, stay-at-home orders, and school closures, with a relatively equal distribution across stance and premise classes.

Key Results

The ensemble approach proved highly successful, beating the competition's mean and median scores in nearly every classification category.

Final competition results for Task 1 and Task 2, comparing the ensemble's performance against competition means.

For Task 1a (ADE Class), the system achieved an F1-score of 0.693, significantly higher than the competition mean of 0.562. In Task 2, the system outperformed the mean in both stance detection (0.577 vs 0.491) and premise identification (0.701 vs 0.574).

For Task 1b (Span Detection), the system achieved an F1-score of 0.568 on overlapping metrics, above the competition mean of 0.527. However, the strict metric was lower (0.091) because the span reconstruction logic included any gaps or non-relevant words between the first and last predicted tokens.

Discussion and Conclusion

The authors found that combining multiple large models consistently yielded higher accuracy than any single model, suggesting that different models learn different nuances of the noisy Twitter landscape.

However, several areas for improvement were identified. Task 1c (MedDRA mapping) used a simple "first-match" approach that could be improved with semantic mapping. Additionally, the span detection logic could be refined to exclude "gaps" between predicted tokens, which would drastically improve performance on strict evaluation metrics. Future work will also explore more extensive pre-processing to help models focus on core semantic content.

Summary updated

BibTeX
@inproceedings{liu2022air,
  author = {Leung Wai Liu and Akshat Gupta and Saheed Obitayo and Xiaomo Liu and Sameena Shah},
  title = {{AIR-JPMC@SMM4H'22: BERT + Ensembling = Too Cool: Using Multiple BERT Models Together for Various COVID-19 Tweet Identification Tasks}},
  booktitle = {SMM4H @ COLING},
  year = {2022},
  url = {https://openreview.net/forum?id=j9TP2IAXkM},
}
Figure