TL;DR
This paper introduces PHS-BERT, a novel domain-specific pre-trained language model (PLM) specifically designed for public health surveillance (PHS) tasks on social media. PHS-BERT significantly outperforms existing state-of-the-art PLMs across 25 diverse datasets and 7 PHS tasks, demonstrating its robustness and generalizability in analyzing health-related social media content.
What Problem Does This Paper Address?
Public health surveillance (PHS) is crucial for monitoring disease outbreaks, forecasting trends, and informing health interventions. Social media offers a vast, real-time source of information for PHS, but analyzing this unstructured text is challenging. While general-purpose PLMs like BERT have advanced natural language processing, they often struggle with the nuances of domain-specific language. Existing domain-specific PLMs exist for areas like biomedical text or general social media, but there's a notable gap for a PLM tailored to the unique challenges of public health surveillance on social media. This paper aims to bridge this gap by developing and benchmarking a PHS-specific PLM.
Methods
The core of this work is the development of PHS-BERT, a domain-specific pre-trained language model. PHS-BERT builds upon the established BERT architecture but is pre-trained on a custom corpus of health-related tweets. This corpus was curated using keywords related to diseases, symptoms, vaccines, and mental health. The pre-training process involved masked language modeling (MLM) and next sentence prediction (NSP), similar to the original BERT.
For fine-tuning on downstream tasks, PHS-BERT was adapted to various classification tasks. The model utilizes the embedding of the [CLS] token from the last hidden layer as the final input feature, followed by a multi-layer perceptron (MLP) with a hyperbolic tangent activation function. An Adam optimizer and a one-cycle learning rate policy were employed during fine-tuning.
Datasets
The researchers evaluated PHS-BERT on a comprehensive set of 25 datasets spanning 7 distinct public health surveillance tasks. These tasks include:
- Suicide Ideation: R-SSD
- Stress Detection: Dreaddit, SAD
- Health Mention: PHM (Multi-class & Binary), HMC2019, RHMD
- Vaccine Sentiment: VS1, VS2
- COVID Related: Covid Lies, Covid Category, COVIDSentiA, COVIDSentiB, COVIDSentiC
- Depression Detection: eRisk T3, Depression_Reddit_1, eRisk T1, Depression_Reddit_2, Depression_Twitter_1, Depression_Twitter_2
- Other Health Related Tasks: PubHealth, Abortion, Amazon Health, SMM4H T1, SMM4H T2, HRT
The datasets were sourced from popular social media platforms like Twitter and Reddit. The paper notes that a stratified 5-fold cross-validation strategy was used for datasets lacking official train/test splits.
Key Insights and Ideas
- Domain-Specific Pre-training is Crucial: The study strongly suggests that pre-training language models on domain-specific corpora (like health-related tweets) significantly improves performance on downstream tasks within that domain, outperforming general-domain models.
- Social Media Nuances Matter: PHS-BERT's success highlights the importance of understanding the unique language, slang, and context present in social media for effective public health analysis.
- Generalizability of PHS-BERT: The model's consistent high performance across a wide range of PHS tasks and datasets indicates its robustness and potential for broad application in the field.
- Bridging the Gap: The paper effectively addresses the lack of a dedicated PLM for social media-based PHS, providing a valuable resource for researchers and practitioners.
Key Results
PHS-BERT achieved state-of-the-art performance on all 25 tested datasets across the 7 PHS tasks. Specifically, the paper reports:
- Suicide Ideation: Marginal increase of 18.45% compared to BERT and 12.79% compared to the second-best PLM.
- Stress Detection: Marginal increase of 3.80% compared to BERT and 2.00% compared to the second-best PLM.
- Health Mention: Marginal increase of 3.34% compared to BERT and 1.76% compared to the second-best PLM.
- Depression Detection: Marginal increase of 6.03% compared to BERT and 2.76% compared to the second-best PLM.
- Vaccine Sentiment: Marginal increase of 7.70% compared to BERT and 0.34% compared to the second-best PLM.
- COVID Related: Average marginal increase of 11.82% compared to BERT and 4.471% compared to the second-best PLM.
- Other Health Related Tasks: Average marginal increase of 11.82% compared to BERT and 4.71% compared to the second-best PLM.
The paper also notes that while general-domain PLMs like BERT can achieve competitive results, domain-specific PLMs like CT-BERT and BERTweet (trained on social media) performed better than general PLMs, and PHS-BERT further surpassed these by being specifically trained for PHS.
Limitations and Future Work
- Limitations: The paper acknowledges that while PHS-BERT shows strong performance, general domain PLMs might require more extensive fine-tuning on relevant corpora to achieve comparable results on domain-specific tasks. The study also mentions that no additional ethics approval was sought as the data was drawn from already published studies.
- Future Work: The authors aim to facilitate the community by releasing PHS-BERT, which they expect will reduce computational costs and establish new baselines for future PHS-related research. The work is intended to accelerate the identification of PHS-related tasks on social media for early intervention screening, not as a diagnostic tool. Reproducibility is ensured by making PHS-BERT publicly available.
Summary updated