Overview of the Seventh Social Media Mining for Health Applications (#SMM4H) Shared Tasks at COLING 2022

Davy Weissenbacher, Juan M. Banda, Vera Davydova, Darryl Estrada-Zavala, Luis Gascó Sánchez, Yao Ge, Yuting Guo, Ari Z. Klein, Martin Krallinger, Mathias Leddin, Arjun Magge, Raul Rodriguez-Esteban, Abeed Sarker, Lucia Schmidt, Elena Tutubalina, Graciela Gonzalez-HernandezSMM4H @ COLING 2022 · OpenReview

Abstract

This paper provides an overview of the Seventh Social Media Mining for Health Applications (#SMM4H) Shared Tasks, a competitive workshop held at COLING 2022. The initiative aims to bridge the gap between the messy, informal world of social media and the structured needs of public health surveillance by challenging researchers to develop advanced Natural Language Processing (NLP) systems.

Introduction: Bridging the Gap in Public Health Surveillance

Traditional public health monitoring often relies on clinical records or official reports, which can suffer from significant time lags and miss the "patient's voice." Social media offers a real-time, massive-scale alternative for pharmacovigilance and disease tracking. However, social media data is notoriously difficult to process: it is informal, rife with slang, linguistically diverse, and highly imbalanced. The #SMM4H shared tasks provide standardized datasets and benchmarks to help the research community tackle these specific hurdles in a reproducible way.

Methodology: The Era of Transformers

The 54 participating teams utilized a variety of state-of-the-art NLP techniques, with a clear consensus on the winning architecture. Nearly all top-performing systems used Transformer-based models, with RoBERTa and BERTweet (pre-trained specifically on English tweets) being the most popular choices.

For tasks involving medical terminology or COVID-19, models like BioBERT, PubMedBERT, and COVID-Twitter-BERT often outperformed general-purpose models. Many winning teams employed "majority voting" or "weighted ensembles" of multiple models to improve robustness. To deal with the rarity of certain health events, teams also utilized techniques like R-drop, adversarial training, data augmentation, and pseudo-labeling.

Leaderboard for Task 1a: Classification of tweets containing Adverse Drug Events (ADE).

Diverse Datasets and Global Reach

The workshop utilized data from Twitter, Reddit, and WebMD in both English and Spanish. Key datasets included:

  • Adverse Drug Events (ADE): Tweets annotated for mentions of side effects, including spans of text and their normalization to the MedDRA ontology.
  • COVID-19: Tweets regarding stance/premise on health mandates, self-reported symptoms (in Spanish), and vaccination status.
  • Demographics & Mental Health: Datasets for self-reported exact age, chronic stress, and intimate partner violence (IPV).
  • SocialDisNER: A large-scale corpus of 9,500 Spanish tweets for detecting disease mentions.

Key Insights: Successes and Bottlenecks

  • The Normalization Bottleneck: While models are quite good at detecting that a user is talking about a side effect (Task 1a), they struggle to normalize that informal text to a formal medical term (Task 1c). The best F1-score for normalization was only 0.387, compared to 0.698 for classification.
  • Age is "Easy," Stance is Hard: Detecting a user's exact age saw very high performance (F1 > 0.92). Conversely, argument mining—detecting the stance and premise behind a tweet—remains difficult (F1 ~0.64-0.70), likely due to the nuance and sarcasm inherent in health debates.
  • Spanish NLP Progress: The introduction of tasks like SocialDisNER highlights a growing and successful effort to move health NLP beyond English-centric models.
Leaderboard for Task 10: SocialDisNER, focusing on disease mention detection in Spanish tweets.

Key Results

Interest in the workshop is surging, with a 35% increase in participating teams compared to the previous year. Top performance metrics included:

  • Exact Age Detection: Reached 0.956 on Reddit and 0.92 on Twitter.
  • Chronic Stress: The best system achieved an F1 of 0.792.
  • Adverse Events: Classification reached 0.698, but normalization lagged significantly.
  • COVID-19 Symptoms (Spanish): Interestingly, the baseline (0.90) outperformed most participants, suggesting that weak supervision and massive data augmentation are highly effective for this specific task.

Limitations and Future Work

Many tasks, such as IPV detection, suffer from "needle in a haystack" problems where the positive class is extremely rare. Future work needs better synthetic data generation or few-shot learning approaches. Additionally, the gap in normalization suggests a need for better ways to link the "layman's language" of social media to professional medical vocabularies. Finally, the authors suggest a need for more complex tasks like temporal extraction and causal reasoning in health social media.

Summary updated

BibTeX
@inproceedings{weissenbacher2022overview,
  author = {Davy Weissenbacher and Juan M. Banda and Vera Davydova and Darryl Estrada-Zavala and Luis Gascó Sánchez and Yao Ge and Yuting Guo and Ari Z. Klein and Martin Krallinger and Mathias Leddin and Arjun Magge and Raul Rodriguez-Esteban and Abeed Sarker and Lucia Schmidt and Elena Tutubalina and Graciela Gonzalez-Hernandez},
  title = {{Overview of the Seventh Social Media Mining for Health Applications (\#SMM4H) Shared Tasks at COLING 2022}},
  booktitle = {SMM4H @ COLING},
  year = {2022},
  url = {https://openreview.net/forum?id=tgBWovmSl5},
}
Figure