Transfer Learning for Health-related Twitter Data

Anne Dirkson, Suzan VerberneSMM4H @ ACL 2019 · OpenReview

Abstract

Transfer learning is promising for many NLP applications, especially in tasks with limited labeled data. This paper describes the methods developed by team TMRLeiden for the 2019 Social Media Mining for Health Applications (SMM4H) Shared Task. Our methods use state-of-the-art transfer learning methods to classify, extract and normalise adverse drug effects (ADRs) and to classify personal health mentions from health-related tweets. The code and fine-tuned models are publicly available.

Introduction

Social media, particularly Twitter, is a goldmine for public health surveillance, but it is notoriously difficult to analyze. Users use slang, misspell drug names, and describe symptoms informally (e.g., "my head is spinning" instead of "vertigo").

The authors address the Social Media Mining for Health Applications (SMM4H) 2019 Shared Task, which involves four specific challenges:

  1. S1 (Classification): Does this tweet mention an Adverse Drug Reaction (ADR)?
  2. S2 (Extraction): Which specific words in the tweet refer to the ADR?
  3. S3 (Normalization): Can we map that informal mention to a standardized medical ID (MedDRA)?
  4. S4 (Personal Health Mentions): Is the user talking about their own health (or someone they know), or just discussing health in general?

The core challenge is the scarcity of labeled data. Traditional machine learning requires massive datasets; transfer learning allows these models to "start smart" by using knowledge gained from Wikipedia or other large text sources.

Methods

The authors employed a multi-stage pipeline centered on transfer learning:

  • Preprocessing: A specialized lexical normalization pipeline and an in-house spelling correction tool were used to handle "Twitter-speak" before feeding data into the models.
  • Classification (S1 & S4): The team utilized ULMfit (Universal Language Model Fine-tuning). This involves taking a model trained on Wikipedia, fine-tuning its "language DNA" on a large collection of health-related tweets, and then finally training it on the specific classification task.
  • Named Entity Recognition (S2): For extracting ADR mentions, they experimented with Flair embeddings combined with BERT and GloVe. This "stacked" approach allows the model to understand both character-level nuances (helpful for typos) and broader sentence context.
  • Normalization (S3): A Recurrent Neural Network (RNN) with Docembeddings was used to map extracted text to MedDRA concept IDs.

Datasets

The study used several datasets provided by the SMM4H workshop, supplemented by external sources. A key aspect of the work was managing the varying sizes of these datasets across tasks, particularly for the supplemented S4+ task.

Dataset statistics for SMM4H 2019 tasks, including the supplemented S4+ dataset.
  • SMM4H 2019 Task Data: Labeled tweets for ADRs and personal health mentions.
  • TwitterHealth Corpus: A custom, larger unsupervised corpus created by the authors. It combined the task data with the DIEGO Drug Chatter corpus (approx. 28,000 Wikipedia articles) to help the models learn the "language" of health on Twitter.
  • S4+: A supplemented dataset for Task 4 that added six labeled datasets from different disease domains to improve the model's ability to generalize.
  • MedDRA & CHV: Standardized medical dictionaries used for the normalization task.

Key Insights

  • The Power of "Unsupervised" Fine-tuning: Fine-tuning the language model on a large, unlabeled corpus of health tweets (the "TwitterHealth" corpus) before training on the specific task significantly boosted performance.
  • More Data Isn't Always Better: In Task 3 (Normalization), adding "aliases" from the Consumer Health Vocabulary (CHV) actually decreased performance, suggesting that noisy external data can be counterproductive.
  • Recall vs. Precision Trade-offs: For ADR classification, a simple "absence of ADRs" (noADE) approach yielded the highest recall, while the ULMfit model was superior in precision.
  • Domain Generalization: For personal health mentions, training on a diverse set of diseases (S4+) outperformed training on a larger but less diverse health corpus.

Results

The transfer learning systems consistently outperformed the average scores of all participants in the SMM4H workshop.

Performance results for Task S2 (ADR Extraction) comparing various BERT-based runs against the task average.
  • ADR Classification (S1): Achieved an F1-score of 0.533, significantly higher than the task average of 0.502.
  • ADR Extraction (S2): The combination of BERT + Flair achieved a "relaxed" F1-score of 0.625, outperforming the average of 0.538.
  • Personal Health Mentions (S4): The ULMfit model trained on the diverse S4+ dataset achieved an F1-score of 0.726, effectively distinguishing personal experiences from general health talk.

Conclusion

The authors demonstrate that transfer learning is highly effective for noisy social media data. While powerful, they suggest future work should focus on more rigorous hyperparameter tuning and deeper domain-specific adaptation, such as training embeddings from scratch on massive medical databases.

Summary updated

Abstract

Transfer learning is promising for many NLP applications, especially in tasks with limited labeled data. This paper describes the methods developed by team TMRLeiden for the 2019 Social Media Mining for Health Applications (SMM4H) Shared Task. Our methods use state-of-the-art transfer learning methods to classify, extract and normalise adverse drug effects (ADRs) and to classify personal health mentions from health-related tweets. The code and fine-tuned models are publicly available.
BibTeX
@inproceedings{dirkson2019transfer,
  author = {Anne Dirkson and Suzan Verberne},
  title = {{Transfer Learning for Health-related Twitter Data}},
  booktitle = {SMM4H @ ACL},
  year = {2019},
  url = {https://openreview.net/forum?id=dlHBBuSg7s},
}
Figure