TL;DR
This paper tackles the challenge of automatically identifying and classifying political statements within unstructured text, like parliamentary speeches and party manifestos. They propose and evaluate several novel methods, including a CRF-based approach and fine-tuned large language models, demonstrating competitive performance and paving the way for large-scale computational analysis of political discourse.
What Problem Does This Paper Address?
Analyzing political texts, such as parliamentary speeches and party manifestos, is crucial for understanding political discourse. However, the sheer volume of these texts makes manual analysis infeasible. While previous work has focused on analyzing manifestos with pre-defined statement boundaries, this paper addresses the more challenging problem of automatically segmenting and labeling statements directly from raw, unstructured text. This is particularly relevant for parliamentary debates, where such pre-defined boundaries are often absent. The goal is to enable more scalable and comprehensive computational analysis of political communication.
Methods
The authors explore three main strategies for jointly segmenting and labeling political statements:
- Linear-chain Conditional Random Fields (CRFs): This approach uses a pre-trained XLM-ROBERTa encoder to generate token embeddings, which are then fed into a linear-chain CRF layer. This model learns to predict statement boundaries and MARPOR labels simultaneously. It's noted for its efficiency and speed.
- Fine-tuned Text-to-Text Models (Flan-T5 XL): A pre-trained Flan-T5 XL model is fine-tuned to perform the segmentation and labeling task. This model is more computationally demanding but offers greater expressiveness.
- In-context Learning with Large Language Models (Llama 3.1 8B Instruct): This method leverages a large language model without fine-tuning. It uses few-shot learning by providing the model with a system message and a set of example prompt-response pairs to guide its segmentation and labeling of new text. Decoding-time constraints are employed to ensure the output adheres to the desired format.
Datasets
The primary dataset used is the MARPOR dataset, a collection of manually annotated party manifestos from over 60 countries. For their analysis, they use a subset of this dataset, comprising 1314 manifestos from 41 countries, with texts in 27 languages. For the parliamentary debate analysis, they use the UK House of Commons Hansard records.
Key Insights and Ideas
- Unified Segmentation and Labeling: The paper demonstrates that it's possible to effectively segment and label political statements simultaneously, moving beyond approaches that rely on pre-existing statement boundaries.
- CRF Efficiency: The CRF-based model offers a compelling trade-off between performance and computational efficiency, making it suitable for large-scale analysis where inference speed is critical.
- LLM Potential and Limitations: While LLMs show promise, particularly with in-context learning, their high computational demands and slow inference times currently limit their practical application for massive datasets.
- Error Type Matters for Scaling: For downstream tasks like political scaling (e.g., RILE scores), the type of errors made by a model can be more important than its raw accuracy in statement classification. Some errors might not significantly distort overall political positioning.
- Parliamentary Data Challenges: Applying manifesto-trained models to parliamentary data requires careful consideration, as topic sequences and statement structures can differ, potentially leading to misclassifications (e.g., the 'Political authority' category).
Key Results
- The CRF-based model achieved competitive F1-scores for statement segmentation and MARPOR label prediction, differing by less than four percentage points from a baseline that assumed gold statement boundaries.
- When provided with oracle statement boundaries, the CRF model outperformed the classifier-based baseline by over four percentage points, suggesting the benefit of modeling label sequence interactions.
- Fine-tuned Flan-T5 XL models generally achieved the best in-domain results for statement segmentation and labeling.
- For political scaling (RILE scores), the CRF-based model, despite lower F1-scores, provided better estimates than some other models.
- Inference speed: The CRF model processed over 3000 tokens per second, while Flan T5 XL processed 26 tokens/sec, and Llama 3.1 8B averaged under 3 tokens/sec, highlighting the significant speed advantage of the CRF.
- Analysis of UK parliamentary debates revealed distinct political trajectories for major parties over three decades, with the CRF model enabling this large-scale exploratory study.
Limitations and Future Work
- In-domain Qualitative Analysis: The breadth of languages in the MARPOR corpus made in-depth qualitative analysis challenging for the authors.
- Computational Costs for LLMs: Full-scale comparisons of autoregressive models across all 41 MARPOR countries were prevented by computational costs.
- Out-of-Domain Evaluation: Due to a lack of labeled parliamentary debate data, the models' out-of-domain performance could not be quantitatively evaluated. The analysis of UK and Australian debates was therefore exploratory.
- Future Work: The authors suggest exploring ways to accelerate inference on high-volume raw-text segmentation and analysis as a crucial avenue for future research. They also note that improving the performance of in-context learning for multi-label classification might be achieved with larger models or longer contexts.
Summary updated