Benchmarking for Public Health Surveillance tasks on Social Media with a Domain-Specific Pretrained Language Model

Usman Naseem, Byoung Chan Lee, Matloob Khushi, Jinman Kim, Adam DunnACL 2022

TL;DR

This paper introduces PHS-BERT, a novel domain-specific pre-trained language model (PLM) specifically designed for public health surveillance (PHS) tasks on social media. PHS-BERT significantly outperforms existing state-of-the-art PLMs across 25 diverse datasets and 7 PHS tasks, demonstrating its robustness and generalizability in analyzing health-related social media content.

What Problem Does This Paper Address?

Public health surveillance (PHS) is crucial for monitoring disease outbreaks, forecasting trends, and informing health interventions. Social media offers a vast, real-time source of information for PHS, but analyzing this unstructured text is challenging. While general-purpose PLMs like BERT have advanced natural language processing, they often struggle with the nuances of domain-specific language. Existing domain-specific PLMs exist for areas like biomedical text or general social media, but there's a notable gap for a PLM tailored to the unique challenges of public health surveillance on social media. This paper aims to bridge this gap by developing and benchmarking a PHS-specific PLM.

Methods

The core of this work is the development of PHS-BERT, a domain-specific pre-trained language model. PHS-BERT builds upon the established BERT architecture but is pre-trained on a custom corpus of health-related tweets. This corpus was curated using keywords related to diseases, symptoms, vaccines, and mental health. The pre-training process involved masked language modeling (MLM) and next sentence prediction (NSP), similar to the original BERT.

For fine-tuning on downstream tasks, PHS-BERT was adapted to various classification tasks. The model utilizes the embedding of the [CLS] token from the last hidden layer as the final input feature, followed by a multi-layer perceptron (MLP) with a hyperbolic tangent activation function. An Adam optimizer and a one-cycle learning rate policy were employed during fine-tuning.

Datasets

The researchers evaluated PHS-BERT on a comprehensive set of 25 datasets spanning 7 distinct public health surveillance tasks. These tasks include:

  • Suicide Ideation: R-SSD
  • Stress Detection: Dreaddit, SAD
  • Health Mention: PHM (Multi-class & Binary), HMC2019, RHMD
  • Vaccine Sentiment: VS1, VS2
  • COVID Related: Covid Lies, Covid Category, COVIDSentiA, COVIDSentiB, COVIDSentiC
  • Depression Detection: eRisk T3, Depression_Reddit_1, eRisk T1, Depression_Reddit_2, Depression_Twitter_1, Depression_Twitter_2
  • Other Health Related Tasks: PubHealth, Abortion, Amazon Health, SMM4H T1, SMM4H T2, HRT

The datasets were sourced from popular social media platforms like Twitter and Reddit. The paper notes that a stratified 5-fold cross-validation strategy was used for datasets lacking official train/test splits.

Key Insights and Ideas

  • Domain-Specific Pre-training is Crucial: The study strongly suggests that pre-training language models on domain-specific corpora (like health-related tweets) significantly improves performance on downstream tasks within that domain, outperforming general-domain models.
  • Social Media Nuances Matter: PHS-BERT's success highlights the importance of understanding the unique language, slang, and context present in social media for effective public health analysis.
  • Generalizability of PHS-BERT: The model's consistent high performance across a wide range of PHS tasks and datasets indicates its robustness and potential for broad application in the field.
  • Bridging the Gap: The paper effectively addresses the lack of a dedicated PLM for social media-based PHS, providing a valuable resource for researchers and practitioners.

Key Results

PHS-BERT achieved state-of-the-art performance on all 25 tested datasets across the 7 PHS tasks. Specifically, the paper reports:

  • Suicide Ideation: Marginal increase of 18.45% compared to BERT and 12.79% compared to the second-best PLM.
  • Stress Detection: Marginal increase of 3.80% compared to BERT and 2.00% compared to the second-best PLM.
  • Health Mention: Marginal increase of 3.34% compared to BERT and 1.76% compared to the second-best PLM.
  • Depression Detection: Marginal increase of 6.03% compared to BERT and 2.76% compared to the second-best PLM.
  • Vaccine Sentiment: Marginal increase of 7.70% compared to BERT and 0.34% compared to the second-best PLM.
  • COVID Related: Average marginal increase of 11.82% compared to BERT and 4.471% compared to the second-best PLM.
  • Other Health Related Tasks: Average marginal increase of 11.82% compared to BERT and 4.71% compared to the second-best PLM.

The paper also notes that while general-domain PLMs like BERT can achieve competitive results, domain-specific PLMs like CT-BERT and BERTweet (trained on social media) performed better than general PLMs, and PHS-BERT further surpassed these by being specifically trained for PHS.

Limitations and Future Work

  • Limitations: The paper acknowledges that while PHS-BERT shows strong performance, general domain PLMs might require more extensive fine-tuning on relevant corpora to achieve comparable results on domain-specific tasks. The study also mentions that no additional ethics approval was sought as the data was drawn from already published studies.
  • Future Work: The authors aim to facilitate the community by releasing PHS-BERT, which they expect will reduce computational costs and establish new baselines for future PHS-related research. The work is intended to accelerate the identification of PHS-related tasks on social media for early intervention screening, not as a diagnostic tool. Reproducibility is ensured by making PHS-BERT publicly available.

Summary updated

BibTeX
@inproceedings{naseem2022benchmarking,
  author = {Usman Naseem and Byoung Chan Lee and Matloob Khushi and Jinman Kim and Adam Dunn},
  title = {{Benchmarking for Public Health Surveillance tasks on Social Media with a Domain-Specific Pretrained Language Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure