Proceedings of the 2018 EMNLP Workshop SMM4H: The 3rd Social Media Mining for Health Applications Workshop & Shared Task

Michael Davy, Graciela Gonzalez‐Hernandez, Roland Roller, Philippe Thomas, Sven Schmeier, Maria Skeppstedt, Manfred Stede, Andreas Kerren, Daniel Lee, Heissl Sebastian, Fabian Pedregosa, Gael Varoquaux, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexan- Dre Passos, David Cournapeau, Matthieu Brucher, Timothy R. Tangherlini, Vwani Roychowdhury, Beth A. Glenn, Inna Pirina, Çağrı Çöltekin, Davy Weissenbacher, Abeed Sarker, Michael Paul, † Dbei, Lucie Flek, Vasileios Lampos, Ingemar J. Cox, Takeshi Onishi, Ari Z Klein, Karen O’Connor, Graciela Gonzalez, Sofiane Abbar, Yelena Mejova, Ingmar Weber, Peter Anderson, Avalon De Bruijn, Kathryn Angus, Ross Gordon, Gerard Hastings, Aron Culotta, Brenda Curtis, Salvatore Giorgi, E Anneke, Lyle Buffone, Robert Ungar, Jessie Ashford, Dan Hemmons, Casey Summers, H Hamilton, Schwartz, P Eric, David Baumer, Shion Mimno, Emily Guha, Geri Quan, Gay, ar ltekin, Taraka Rama, Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, Margaret Mitchell, Munmun De, Choudhury, Sushovan De, Veronica Lynn, Alissa Goodman, Kate Niederhoffer, Kate Loveys, Philip Resnik, Gal Varoquaux, Alexandre Gramfort, Matthieu Perrot, douard Duchesnay, James Pennebaker, Ian M. CampbellOpenalex Unknown 2018 · DOI · OpenAlex

Introduction

As social media usage continues to expand globally, these platforms have evolved into significant repositories of public health data. Recent estimates suggest that 26% of social media users discuss health information online, and 42% of those users discuss specific medical conditions. This shift presents a unique opportunity for the biomedical and public health communities to leverage real-time, "real-world" data that traditional sources—such as clinical records or pharmacy claims—often miss due to reporting delays and high costs.

The 3rd Social Media Mining for Health Applications (SMM4H) workshop, held at EMNLP 2018, focused on the methodological challenges inherent in this medium. Mining social media requires navigating informal language, sarcasm, slang, and a high noise-to-signal ratio. This paper summarizes the key contributions, shared tasks, and methodological shifts presented during the workshop.

The SMM4H Shared Tasks

A central component of the 2018 workshop was a series of four shared tasks designed to benchmark NLP systems on annotated Twitter data:

  1. Automatic Classification of Drug Mentions: Identifying tweets that mention a specific drug name.
  2. Medication Intake Reports: Distinguishing between a casual mention of a drug and a first-person report of actually taking the medication.
  3. Adverse Drug Reaction (ADR) Detection: Identifying self-reports of side effects, a critical task for pharmacovigilance.
  4. Vaccine Behavior Mentions: Classifying tweets related to vaccination intent and behavior, specifically regarding the flu vaccine.

A total of 14 teams submitted 78 system runs, providing a comprehensive look at the state-of-the-art in health-related text classification.

The 2018 proceedings marked a definitive shift in the field, with deep learning architectures largely superseding traditional feature-engineering approaches.

Hierarchical and Attention-Based Models

The most successful systems, such as those from the THU_NGN team, utilized hierarchical neural networks. These models first learn word representations from characters—a vital step for handling the frequent misspellings (e.g., "aspirn") common in social media. These representations are then processed through Bi-directional Long Short-Term Memory (Bi-LSTM) networks. The integration of Multi-Head Self-Attention was particularly impactful, allowing models to dynamically weight the importance of specific words within a tweet to capture long-distance context.

Psycholinguistic and Topic Modeling

Beyond classification, researchers employed Linguistic Inquiry and Word Count (LIWC) to track shifts in word categories (such as first-person pronouns) during illness cycles. For vaccine hesitancy, Non-negative Matrix Factorization (NMF) was used to extract recurring themes from large-scale discussions, helping to map the landscape of public sentiment.

The Persistence of Traditional ML

While deep learning dominated, Support Vector Machines (SVMs) using n-gram features remained competitive. These traditional models proved particularly useful for tasks with smaller datasets or highly specific domains, such as identifying emergency requests for blood donations.

Data Sources and Benchmarks

The workshop highlighted the importance of diverse, specialized corpora:

  • Twitter Benchmarks: Thousands of annotated tweets for the four shared tasks.
  • Reddit Corpora: Researchers utilized subreddits like r/depression to study mental health markers, noting that Reddit’s anonymity and lack of character limits often yield more honest and detailed self-disclosures than Twitter.
  • The Pregnancy Corpus: A massive dataset of 4 million tweets from 112,500 timelines was used to study medication adherence during pregnancy.
  • Online Forums: Data from specialized forums like ivf.ca provided insights into the sentiment of patients undergoing complex medical treatments.

Key Insights and Performance

The workshop results demonstrated both the potential and the remaining hurdles of health-related NLP:

  • High Accuracy in Identification: Drug mention detection (Task 1) reached an impressive F1-score of 0.918, and vaccine behavior classification (Task 4) achieved an F1-score of 0.887.
  • The ADR Challenge: Detecting adverse drug reactions (Task 3) remains difficult, with the top system reaching only an F1-score of 0.522. The rarity of ADR reports in general chatter makes this a "needle in a haystack" problem.
  • Social Activity and Recovery: One study found a correlation between recovery from the flu and an increase in "Friends" and "Family" related vocabulary, suggesting that social engagement may be a linguistic marker of improving health status.
  • Character-Level Robustness: Because users rarely use correct medical terminology or spelling, character-level modeling is no longer optional; it is a requirement for robust health-tech NLP.

Limitations and Future Directions

Despite significant progress, several challenges persist. Data imbalance remains a primary obstacle, particularly for rare events like ADRs. Furthermore, models trained on specific platforms (like Reddit) often struggle to generalize to the broader environment of Twitter. Finally, the nuances of sarcasm and irony in patient discussions continue to challenge sentiment analysis models.

Future research is moving toward multi-task learning, where models are trained on several related health tasks simultaneously. This approach aims to build a more holistic understanding of medical language, intent, and the complex ways in which users express their health experiences online.

Summary updated

Abstract

Welcome to the 3rd Social Media Mining for Health Applications Workshop and Shared Task -SMM4H.The total number of users of social media continues to grow worldwide, resulting in the generation of vast amounts of data.With nearly half of adults worldwide and two-thirds of all American adults using social networking, the latest Pew Research Report estimates that 26% of the total users have discussed health information and, of those, 42% have even discussed current medical conditions.Advances in automated NLP and machine learning present the possibility of utilizing this massive data source for biomedical and public health applications, if researchers address the methodological challenges unique to this media.For its third iteration, the SMM4H workshop takes place in Brussels, Belgium, on November 1, 2018, and is co-located with the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).Following on the success of a session and accompanying Workshop on the topic that was hosted at the Pacific Symposium in Biocomputing (PSB) in 2016 and the AMIA Annual Conference in 2017, this workshop aims to provide a forum for the ACL community members to present and discuss NLP advances specific to social media use in the particularly challenging area of health applications, with a special focus given to automatic methods for the collection, extraction, representation, analysis, and validation of social media data for health informatics.As for the previous years, the workshop includes shared tasks with a particular interest on social media mining for pharmacovigilance.This third execution of the SMM4H shared tasks comprises four subtasks.These subtasks involve annotated user posts from Twitter (tweets) and focus on the (i) automatic classification of tweets mentioning a drug name, (ii) automatic classification of tweets containing reports of first-person medication intake, (iii) automatic classification of tweets presenting self-reports of adverse drug reaction (ADR) detection, and (iv) automatic classification of vaccine behavior mentions in tweets.A total of 14 teams participated and 78 system runs were submitted.Deep learning-based classifiers were the primary approach, but feature-based classifiers and a few ensemble learning systems were also used.We received very high quality submissions, and present 19 as long and short talks and posters.The organizing committee would like to thank the program committee, consisting of 13 researchers, for their thoughtful input on the submissions, as well as the organizers of EMNLP for their support and management.Finally, a huge thanks
BibTeX
@inproceedings{davy2018proceedings,
  author = {Michael Davy and Graciela Gonzalez‐Hernandez and Roland Roller and Philippe Thomas and Sven Schmeier and Maria Skeppstedt and Manfred Stede and Andreas Kerren and Daniel Lee and Heissl Sebastian and Fabian Pedregosa and Gael Varoquaux and Vincent Michel and Bertrand Thirion and Olivier Grisel and Mathieu Blondel and Peter Prettenhofer and Ron Weiss and Vincent Dubourg and Jake Vanderplas and Alexan- Dre Passos and David Cournapeau and Matthieu Brucher and Timothy R. Tangherlini and Vwani Roychowdhury and Beth A. Glenn and Inna Pirina and Çağrı Çöltekin and Davy Weissenbacher and Abeed Sarker and Michael Paul and † Dbei and Lucie Flek and Vasileios Lampos and Ingemar J. Cox and Takeshi Onishi and Ari Z Klein and Karen O’Connor and Graciela Gonzalez‐Hernandez and Graciela Gonzalez and Sofiane Abbar and Yelena Mejova and Ingmar Weber and Peter Anderson and Avalon De Bruijn and Kathryn Angus and Ross Gordon and Gerard Hastings and Aron Culotta and Brenda Curtis and Salvatore Giorgi and E Anneke and Lyle Buffone and Robert Ungar and Jessie Ashford and Dan Hemmons and Casey Summers and H Hamilton and Schwartz and P Eric and David Baumer and Shion Mimno and Emily Guha and Geri Quan and Gay and ar ltekin and Taraka Rama and ar ltekin and Taraka Rama and Glen Coppersmith and Mark Dredze and Craig Harman and Kristy Hollingshead and Margaret Mitchell and Munmun De and Choudhury and Sushovan De and Veronica Lynn and Alissa Goodman and Kate Niederhoffer and Kate Loveys and Philip Resnik and H Schwartz and Fabian Pedregosa and Gal Varoquaux and Alexandre Gramfort and Vincent Michel and Bertrand Thirion and Olivier Grisel and Mathieu Blondel and Peter Prettenhofer and Ron Weiss and Vincent Dubourg and Jake Vanderplas and Alexandre Passos and David Cournapeau and Matthieu Brucher and Matthieu Perrot and douard Duchesnay and James Pennebaker and Ian M. Campbell},
  title = {{Proceedings of the 2018 EMNLP Workshop SMM4H: The 3rd Social Media Mining for Health Applications Workshop \& Shared Task}},
  booktitle = {Openalex Unknown},
  year = {2018},
  doi = {10.18653/v1/w18-59},
}
Figure