TL;DR
This paper tackles the challenge of comparing U.S. state-level COVID-19 policy announcements with public discussions on Twitter, requiring models that can accurately classify both. They demonstrate that combining event-extraction with a co-training approach significantly boosts transfer learning accuracy for RoBERTa models, achieving an 11% improvement on this cross-domain task.
What Problem Does This Paper Address?
Understanding the relationship between government policy and public sentiment is crucial, especially during crises like the COVID-19 pandemic. However, directly comparing official policy documents with informal social media discussions presents a significant challenge. These two types of text have vastly different styles, vocabularies, and intents, making it difficult for NLP models to transfer knowledge effectively between them. Researchers need methods to bridge this gap to enable robust social science studies that can analyze both policy pronouncements and public reactions.
Methods
The core of this paper lies in its novel application of event-extraction and co-training for cross-domain transfer learning.
- Event-Extraction: They hypothesize that extracting structured event information (agents, patients, anchors) can help abstract the core meaning from text, making it more comparable across different domains. This is used as one "view" of the data.
- Co-Training: This is a semi-supervised learning technique where two classifiers are trained on different "views" of the data. Each classifier labels unlabeled data for the other, iteratively expanding the training set. In this work, the two views are:
- Text with extracted events.
-
Text with event-related words removed. They also explore other baseline views like noun-phrases and verb-phrases to validate the effectiveness of event-extraction.
-
Classifiers: They compare two classifiers:
- Logistic Regression: A fast, interpretable baseline using TF-IDF features.
- RoBERTa-base: A powerful pre-trained transformer model, chosen for its strong generalization capabilities across transfer tasks.
Datasets
- Policy Announcements: Approximately 2,100 U.S. state-level government policy announcements scraped from the National Governors Association website. A subset of 576 of these were hand-labeled into 6 hierarchical categories.
- Tweets: A large dataset of roughly 137 million tweets related to COVID-19. A subset of 310 tweets were hand-labeled as policy-related for evaluation.
Key Insights and Ideas
- Event-extraction as a unifying signal: The paper strongly suggests that extracting structured event information is a powerful way to bridge the stylistic gap between formal policy documents and informal tweets. It helps focus on the "what happened" rather than the "how it was said."
- Co-training amplifies RoBERTa's strengths: While event-extraction alone showed limited gains, when combined with co-training, it significantly boosted the performance of the RoBERTa model, especially for the challenging cross-domain transfer task. This indicates that co-training effectively leverages the signal from event-extraction.
- Domain-specific pre-training matters: The authors note that while RoBERTa is a strong general-purpose model, a model pre-trained specifically on Twitter or legal text might yield even better results, highlighting the importance of domain adaptation in pre-training.
- Co-training's impact on vocabulary: The analysis of vocabulary expansion during co-training suggests that for models like Logistic Regression, the introduction of new vocabulary from tweets can introduce sparsity that hinders performance. Pre-trained models like RoBERTa are better equipped to handle this expansion.
Key Results
- 11% improvement: Co-training with event-extraction views improved the transfer accuracy of the RoBERTa classifier by 11% over a baseline.
- Maximum transfer accuracy: The co-trained RoBERTa classifier achieved a maximum transfer accuracy of .77 f1-score on a hand-validated set of policy-related tweets, using only 576 labeled policy announcements.
- Event-extraction benefits co-training: While event-extraction alone didn't significantly improve accuracy without co-training, it was the most effective view for co-training, especially on the transfer task.
- Logistic Regression vs. RoBERTa: Co-training showed a more pronounced benefit for RoBERTa, suggesting that pre-trained models are better positioned to leverage the added signal from co-training and event-extraction.
Limitations and Future Work
- Domain-specific pre-training: The authors acknowledge that using a more domain-specific pre-trained model (e.g., Twitter-specific RoBERTa) could further enhance performance.
- Exploring other views: Future work could explore additional "views" for co-training, such as selective lemmatization, to potentially improve accuracy.
- Conditional independence: While they assume conditional independence between views, they acknowledge this might not be perfectly met and could be an area for further investigation.
- Imbalanced classes: The paper notes that imbalanced classes can degrade co-training performance, a limitation that might affect future applications.
- Future research: The work paves the way for future studies examining the interplay between policy and public conversation, leveraging the developed methodology and datasets.
Summary updated