Overview of the Fourth Social Media Mining for Health (SMM4H) Shared Tasks at ACL 2019

Davy Weissenbacher, Abeed Sarker, Arjun Magge, Ashlynn Daughton, Karen O’Connor, Michael J. Paul, Graciela Gonzalez-HernandezACL 2019

TL;DR

This paper presents the results of the fourth SMM4H shared task, focusing on extracting and normalizing adverse drug reactions (ADRs) from social media and classifying personal health mentions. The challenge highlighted the continued dominance of deep learning architectures, particularly those leveraging pre-trained transformer embeddings like BERT, while also underscoring the persistent difficulties in end-to-end ADR resolution.

What Problem Does This Paper Address?

Social media platforms like Twitter have become a vast repository of public health information, offering real-time insights into health trends, medication side effects, and personal health experiences. However, the sheer volume, noise, and creative language used on these platforms present significant challenges for automated analysis. Researchers need robust methods to extract meaningful health-related information for public health surveillance and biomedical research. The SMM4H shared task series aims to address these challenges by providing standardized tasks and datasets to drive progress in this domain. This year's iteration focused on two main areas: pharmacovigilance (identifying and understanding adverse drug reactions) and general health mention classification.

Methods

The SMM4H 2019 shared task featured four distinct sub-tasks:

  • Task 1: ADR Classification: A binary classification task to identify tweets mentioning adverse drug reactions (ADRs). This is a foundational step for filtering relevant content.
  • Task 2: ADR Span Extraction: A named entity recognition (NER) task to pinpoint the exact text spans within tweets that describe ADRs.
  • Task 3: ADR Extraction and Normalization: An end-to-end task combining ADR span extraction with normalization. This involves mapping the extracted ADR mentions to standardized concept IDs within the MedDRA (Medical Dictionary for Regulatory Activities) vocabulary.
  • Task 4: Personal Health Mention Classification: A classification task to distinguish between tweets reporting personal health status or opinions, general health discussions, or unrelated mentions. This task specifically tested the generalizability of models across different health contexts.

The competition leveraged state-of-the-art NLP techniques, with a notable trend towards deep learning architectures. Many participants utilized pre-trained word embeddings, particularly those from transformer models like BERT, which can capture contextual nuances of language more effectively than traditional methods.

Datasets

The corpora for all tasks were composed of public tweets collected via Twitter's streaming API, made available to participants under Twitter's data use policy.

  • Task 1: Used a training set of 25,678 tweets and a test set of 4,575 tweets.
  • Task 2 & 3: Utilized a training set of 2,276 tweets (1,300 positive for ADRs, 976 negative) and a test set of 1,573 tweets. For Task 3, additional MedDRA annotations (LLT and PT identifiers) were provided.
  • Task 4: Employed datasets related to influenza (vaccination and infection) and Zika virus (travel changes and mosquito exposure mitigation).

The datasets are publicly available at https://competitions.codalab.org/competitions/22521.

Key Insights and Ideas

  • Dominance of Deep Learning and Transformers: The results strongly reinforce the trend of deep learning, particularly transformer-based models like BERT, outperforming traditional machine learning approaches for social media text analysis. These models excel at capturing complex linguistic nuances.
  • ADR Resolution Remains Challenging: While ADR detection and extraction have seen improvements, the end-to-end resolution (extraction and normalization to standardized terms) remains a significant hurdle. This suggests that current systems still require human oversight for accurate medication safety monitoring.
  • Generalizability is Promising but Context-Dependent: Task 4 demonstrated that models can generalize to new health domains, which is crucial for building scalable health surveillance tools. However, performance varied, with models performing better on contexts similar to their training data and less well on entirely novel domains.
  • Data Quantity Matters: The authors note that the limited quantity of data for Task 4, while sufficient for initial insights, suggests that larger datasets would likely lead to improved classifier performance and generalization.

Key Results

  • Task 1 (ADR Classification): The best systems achieved an F1-score of 0.646, a notable improvement from previous years' scores of 0.522.
  • Task 2 (ADR Extraction): The best system achieved a strict matching F1-score of 0.432, indicating that precise span identification is still difficult. Relaxed F1-scores were higher, suggesting that some overlap was often achieved.
  • Task 3 (ADR Resolution): The best system achieved a strict matching F1-score of 0.432 for ADR resolution, highlighting the ongoing challenges in mapping extracted mentions to correct MedDRA terms.
  • Task 4 (Health Mention Classification): Systems showed reasonable performance across various contexts, with average F1-scores ranging from 0.41 to 0.87 depending on the context. Performance was generally better on contexts similar to the training data.
  • Participation: 34 teams registered, with 19 teams from 12 countries submitting runs across the four tasks.

Limitations and Future Work

  • ADR Resolution Complexity: The paper explicitly states that end-to-end ADR resolution remains a major challenge, requiring further research. The low strict F1-scores for extraction and normalization underscore this.
  • Data Limitations: The authors acknowledge that the quantity of data for Task 4 was limited, and more data would likely improve model performance and generalization capabilities.
  • Generalization Gaps: While Task 4 showed promise, the performance drop on novel health contexts suggests that further work is needed to develop truly robust and adaptable health classification models.
  • Future Work: The authors suggest continued research into improving ADR resolution techniques and exploring methods for better generalization across diverse health domains and social media data. The success of transformer models also points towards further exploration of their capabilities in this domain.

Summary updated

BibTeX
@inproceedings{weissenbacher2019overview,
  author = {Davy Weissenbacher and Abeed Sarker and Arjun Magge and Ashlynn Daughton and Karen O’Connor and Michael J. Paul and Graciela Gonzalez-Hernandez},
  title = {{Overview of the Fourth Social Media Mining for Health (SMM4H) Shared Tasks at ACL 2019}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure