TL;DR
This paper explores how well different transformer-based language models can detect influenza and COVID-19 related health experiences from social media (Twitter) in Saskatchewan, Canada. They found that domain-specific models, particularly CT-BERT for COVID-19, generally outperform general models, and that data balancing techniques didn't significantly improve performance.
What Problem Does This Paper Address?
Traditional public health surveillance for diseases like influenza and COVID-19 often relies on methods like lab reports and doctor visits, which can be slow to detect outbreaks in real-time. Social media offers a potential complementary data source for faster, more granular insights. However, extracting meaningful health information from the noisy, informal text of social media posts is challenging. This paper specifically addresses the gap in understanding how well different advanced language models can identify user-reported symptoms and health experiences related to influenza and COVID-19 from Canadian social media data, and whether specialized models trained on health or social media data offer an advantage over general-purpose models.
Methods
The researchers employed a multi-stage approach:
- Data Collection: They collected over 50 million tweets from Saskatchewan, Canada, using the Twitter API between December 2016 and December 2022.
- Keyword-Based Filtering: Tweets were filtered using specific keyword lists for influenza (December 2016 - March 2018) and COVID-19 symptoms (January 2020 - June 2021).
- Manual Annotation: A team of 12 annotators manually labeled tweets as indicating plausible influenza or COVID-19 cases. This involved rigorous guidelines to exclude news, advertisements, and general comments.
- Data Preprocessing: Tweets were cleaned by removing emojis, punctuation, links, mentions, hashtags, and extra spaces.
- Model Training and Evaluation: They fine-tuned eight pre-trained transformer-based language models: BERT-base, BERT-large, RoBERTa-base, RoBERTa-large, BERTweet-base, BERTweet-large, BERTweet-covid-base, and CT-BERT.
- Handling Class Imbalance: To address the skewed distribution of positive cases in the datasets, they experimented with random oversampling and undersampling techniques on the training data.
- Performance Metrics: Models were evaluated using recall, F1 score, accuracy, and Area Under the ROC Curve (AUC).
Datasets
The study utilized two newly constructed, manually annotated Twitter datasets specific to Saskatchewan, Canada:
- Influenza Twitter Dataset: Comprised of 6,444 tweets filtered with influenza-related keywords from December 13, 2016, to March 17, 2018. Of these, 1,129 (17.5%) were labeled as indicating plausible influenza cases.
- COVID-19 Twitter Dataset: Comprised of 11,939 tweets filtered with COVID-19 symptom-related keywords from January 1, 2020, to June 22, 2021. Of these, 924 (7.7%) were labeled as indicating plausible symptomatic COVID-19 cases.
These datasets represent a novel benchmark for analyzing self-reported health experiences related to these diseases within a specific Canadian province.
Key Insights and Ideas
- Domain-Specific Models Shine: Models specifically pre-trained on social media data (BERTweet) or health-related data (CT-BERT for COVID-19) generally outperformed general-domain models like BERT and RoBERTa. This suggests that tailoring models to the specific characteristics of social media text and the domain of health is crucial for better performance.
- CT-BERT Dominates for COVID-19: CT-BERT, a model specifically designed for COVID-19 Twitter data, achieved the highest scores for recall, F1, and accuracy on the COVID-19 dataset. This highlights the benefit of highly specialized models when sufficient domain-specific pre-training data is available.
- Influenza Detection is More Nuanced: For influenza detection, BERTweet models showed strong performance, but CT-BERT did not consistently outperform BERTweet-large, suggesting that the overlap in symptoms between influenza and COVID-19 might make it harder for a COVID-19-specific model to excel at influenza detection without specific fine-tuning.
- Data Balancing is Not a Silver Bullet: Surprisingly, the study found that applying oversampling or undersampling techniques to address class imbalance did not lead to improved model performance. This indicates that for these specific datasets and models, the inherent data characteristics or the models' ability to handle imbalance might be sufficient, or that these techniques were not optimally applied.
- Large Models Don't Always Mean Better: While BERT-large generally outperformed BERT-base, the performance difference between "base" and "large" versions of RoBERTa and BERTweet was less consistent. This could be due to the relatively small size of the fine-tuning datasets, which might not fully leverage the capacity of larger models, potentially leading to overfitting.
Key Results
- COVID-19 Dataset: CT-BERT achieved the highest scores, with recall at 94.8%, F1 at 94.4%, and accuracy at 94.6% when no sampling was applied (for the "large" model size group).
- Influenza Dataset: BERTweet models generally performed better. For the "large" model size group, BERTweet-large achieved the best mean scores in recall, F1, and accuracy under no sampling or oversampling methods.
- Sampling Impact: In most cases, data balancing techniques (oversampling and undersampling) did not lead to statistically significant improvements and sometimes even resulted in slightly worse performance compared to no sampling.
- Base vs. Large Models: BERT-large consistently outperformed BERT-base. However, the performance difference between base and large versions of RoBERTa and BERTweet was mixed, with no consistent trend observed across all metrics and datasets.
Limitations and Future Work
- Geographic and Language Specificity: The study was limited to English-language tweets from Saskatchewan, Canada. Findings may not generalize to other regions or languages.
- Keyword Limitations: The use of symptom-related keywords for COVID-19 filtering might have missed non-symptomatic cases or reports using different terminology.
- Asymptomatic Cases: The method cannot detect asymptomatic COVID-19 infections, which are a significant portion of cases.
- Twitterverse Evolution: Changes in Twitter's platform and the increasing cost of large-scale monitoring are noted concerns.
- Incubation Periods: The temporal resolution of symptom-based monitoring might be limited by disease incubation periods, suggesting a need to combine with less lagged surveillance data.
Future work could involve expanding the geographic scope, incorporating multilingual data, exploring more comprehensive keyword strategies, and investigating the integration of these models into broader public health surveillance systems. The authors also suggest that base models might be sufficient for many practical applications due to the computational costs associated with large models.
Summary updated