Introduction
During global health crises, public health organizations like the CDC and WHO face a significant "one-to-many" communication challenge on social media. A single tweet can trigger a vast and often hostile range of responses. While sentiment analysis is commonly used to evaluate message reception retrospectively, there is a critical need for proactive tools. This research introduces a "virtual focus group" approach, fine-tuning generative models to predict the semantic and sentiment distribution of public reactions before a message is posted.
Methodology
The researchers framed response prediction as a generative task. They utilized a GPT-2 (762M parameter) model, fine-tuned on triples of <|message|>, <|author|>, and <|response|>. Unlike traditional chatbots, this model is designed to capture the distribution of typical responses rather than a single "correct" reply.
At inference time, the model is prompted with a draft tweet and an author (e.g., @CDCgov) and generates 30 candidate responses using beam sampling. To evaluate the model's performance, the authors developed a framework comparing generated responses to ground truth across two dimensions:
- Semantics: Using MiniLM embeddings to calculate cosine similarity.
- Sentiment: Using a RoBERTa-based classifier to compare polarity distributions via Chi-Square tests.
Datasets
The study leveraged two distinct Twitter datasets focusing on high-impact accounts, including the WHO and various branches of the CDC:
- COVID-19 Dataset: ~8,500 original messages and ~70,000 responses from March to September 2020.
- Vaccines Dataset: ~3,000 original messages and ~61,000 responses from October 2021 to January 2022.
Key Results
The model demonstrated a strong ability to replicate the "vibe" of public discourse, including specific themes like mistrust of international organizations and concerns over political interference.
Semantic Accuracy
The model significantly outperformed random baselines in capturing the semantic space of human responses. Using Area Under Regression Error Characteristic (AUC-REC) curves, the researchers found that the model achieved a "Model % Difference" (relative to ground truth) of 71.7% for COVID-19 and 66.7% for Vaccines.
Sentiment Alignment
In the COVID-19 dataset, the model’s predicted sentiment distribution matched the primary ground truth in 55.5% of cases, compared to a random-chance baseline of 43.9%. The model also successfully identified account-specific nuances, such as the generally more negative reception of @CDCgov compared to @WHO.
Discussion and Future Work
While promising, the model has limitations. As a snapshot in time, it is subject to temporal drift; a model trained on 2020 data cannot predict 2024 reactions as public sentiment evolves. Additionally, like other LLMs, it can "hallucinate" irrelevant tags or hashtags.
Future research directions include integrating Retrieval-Augmented Generation (RAG) to improve factuality and conditioning the model on specific responder demographics to provide more granular, targeted insights for public health communicators.
Summary updated