TL;DR
This paper tackles the challenge of extracting fine-grained protest attendance figures from news articles, given only coarse, order-of-magnitude labels and a few precisely annotated examples. They introduce a new task and benchmark dataset, demonstrating that even a simple rule-based model can outperform a zero-shot transformer, and that fine-tuning on a small set of precise examples significantly boosts performance.
What Problem Does This Paper Address?
Social scientists often face the challenge of extracting detailed information from large collections of documents. A common scenario involves having many documents with broad, "coarse" labels (e.g., "protest happened") and a small number of documents with highly specific, "fine-grained" annotations (e.g., "exactly 30-40 people attended"). The problem is how to effectively "upsample" the coarse labels to generate more detailed, fine-grained data for the entire collection. This is particularly relevant in fields like social science research, where understanding the scale of events like protests is crucial for analyzing their impact. Existing NLP tools often struggle with this, as training a classifier on coarse labels doesn't directly yield fine-grained answers. This paper focuses on the specific task of identifying the exact crowd size at US protests from news text, using coarse attendance categories and a limited set of precisely labeled spans.
Methods
The authors propose a question-answering (QA) framework for this task. Their approach involves several key components:
- Heuristic Keyword Model: A rule-based system that identifies number-containing phrases, converts them to numeric form, and filters them based on keywords (like "crowd," "protesters") and their proximity to the protest event. Dependency parsing is used to expand the identified span.
- Zero-Shot QA Model: A pre-trained RoBERTa model, fine-tuned on the general SQuAD 2.0 dataset, is used "off-the-shelf" without any further training on the protest data. This serves as a baseline to see how well a powerful pre-trained model performs without specific task adaptation.
- Fine-tuned QA Model (Multitask): A more advanced model that combines the standard QA objective with an additional objective to predict the coarse crowd size category. This multitask approach aims to leverage both types of labels. They also explore fine-tuning this model on:
- Coarse Labels Only: Training the model using only the order-of-magnitude information.
- Gold Spans Only: Fine-tuning on the small set of precisely annotated protest size spans.
- Coarse Labels + Gold Spans: Combining both types of information.
- Coarse Labels + Heuristic Spans: Using the heuristically generated spans as additional training data.
Datasets
The primary dataset used is derived from the Crowd Counting Consortium (CCC) dataset, which compiles structured data about protests from news reports.
- Newly Introduced Benchmark: The paper constructs a new benchmark dataset for protest size detection. This dataset consists of:
- Coarse Label Training Set: ~2,694 full articles with order-of-magnitude labels but no exact spans.
- Gold Span Training Set: 25 short text snippets with exact answer spans but no coarse labels.
- Validation Set: 200 full articles with both coarse labels and exact spans.
- Test Set: 930 full articles with both coarse labels and exact spans.
Key Insights and Ideas
- Heuristics Can Be Powerful: Surprisingly, a carefully crafted rule-based heuristic model can outperform a powerful zero-shot transformer QA model on this specific task. This suggests that domain-specific knowledge and careful feature engineering can be highly effective.
- Few-Shot Learning is Crucial: Fine-tuning even a standard QA model on a very small set of precisely annotated "gold spans" (just 25 examples) leads to a dramatic improvement in performance, highlighting the power of few-shot learning for specialized extraction tasks.
- Leveraging Coarse Labels is Key: The multitask model, trained to predict both coarse labels and fine-grained spans, shows significant gains over the zero-shot model, demonstrating that the abundant coarse labels can indeed be used to improve fine-grained extraction.
- Combining Data Sources is Beneficial: Models that combine coarse labels with heuristically generated spans achieve performance close to models trained on the limited gold spans, suggesting that combining different sources of weak supervision can be very effective.
Key Results
- Heuristic Model Performance: The heuristic keyword model achieved 0.54 exact match accuracy and 0.61 F1 score on the test set.
- Zero-Shot QA Performance: The RoBERTa QA model, without any fine-tuning on the protest data, achieved a low 0.17 exact match and 0.27 F1.
- Fine-tuning Gains:
- Fine-tuning the multitask model on coarse labels alone improved performance to 0.48 exact match and 0.54 F1.
- Fine-tuning the QA model on just the 25 gold spans yielded the best results: 0.67 exact match and 0.65 F1.
- A multitask model trained on coarse labels and heuristic spans achieved 0.66 exact match and 0.63 F1, very close to the top-performing model.
Limitations and Future Work
- Data Scarcity: The primary limitation is the small number of precisely annotated "gold spans" available for fine-tuning.
- Copyrighted Text: The use of scraped copyrighted news text raises concerns, and the authors propose making their scraping code and URL lists available to allow for replication.
- Future Work: The authors suggest further integrating weak/distant supervision techniques and modeling label noisiness. They also highlight the potential for this approach in other domains where similar upsampling challenges exist, such as extracting policy proposals from manifestos or identifying armed actors in conflict reports.
Summary updated