TL;DR
This paper investigates how well Large Language Models (LLMs) can identify political actors making claims in text, a crucial step for understanding public discourse. Surprisingly, LLMs struggle with generating the canonical actor name, even when they correctly identify the actor. A hybrid approach combining LLMs with traditional NLP methods significantly improves performance.
What Problem Does This Paper Address?
Understanding political debates is essential for democratic analysis. One way to do this is by building "discourse networks," which map out who is saying what about specific issues. A key challenge in automatically constructing these networks is "actor identification": figuring out which specific person or group (the actor) is behind a particular statement or claim. This is tricky because texts often refer to actors using pronouns or vague descriptions (e.g., "He said...", "A U.S. official stated..."). To build accurate discourse networks, we need to link these mentions to a consistent, canonical name (e.g., "Joe Biden" instead of "He"). While LLMs are powerful text generators, it's unclear if they can reliably perform this nuanced task of identifying and standardizing actor names.
Methods
The authors compare two main approaches for actor identification:
- Traditional Pipeline: This method uses a sequence of dedicated NLP components. First, a Conditional Random Field (CRF) model, trained with an XLM-ROBERTa encoder, extracts mentions of actors associated with a claim. Second, a neural classifier, also using XLM-ROBERTa, maps these mentions to canonical actor names. This classifier is trained on a predefined set of common canonical names, with a fallback for less frequent ones.
- LLM-Based Approach: This method treats actor identification as an end-to-end text generation task. An LLM (specifically, Llama 2) is prompted with the article's text and the target claim, and it's asked to directly output the canonical actor name. The authors experiment with zero-shot and few-shot prompting techniques, varying instructions and exemplar selection to find optimal performance.
- Hybrid Model: Motivated by the limitations of the individual approaches, a hybrid model is proposed. This model integrates the LLM's prediction as an additional input to the traditional pipeline, allowing the LLM to handle the initial actor identification and the pipeline to refine the canonicalization.
Datasets
The study uses the DEbateNet dataset. This is a German corpus of approximately 700 newspaper articles from the left-wing quality newspaper "taz" concerning the 2015 German debate on migration. It contains around 1,800 actor-attributed claims, with gold-standard canonical actor names provided by political science experts. The dataset is split into train, development, and test sets.
Key Insights and Ideas
- LLMs struggle with canonicalization: Despite their impressive text generation capabilities, LLMs are surprisingly poor at producing the exact canonical actor name. They often identify the correct entity but fail to format it correctly or use the precise standardized form.
- Traditional pipelines are surprisingly robust: The dedicated, multi-stage NLP pipeline, while seemingly more basic, outperforms the LLM in strict "exact match" evaluations. This suggests that specialized models can be more precise for specific tasks.
- Canonicalization is the hard part: The biggest performance gains for both models come when evaluating "up to canonicalization," indicating that correctly identifying the entity is easier than producing its standardized name.
- Hybrid models offer the best of both worlds: Combining the LLM's ability to identify the correct actor with the traditional pipeline's strength in canonicalization leads to the best overall performance.
Key Results
- Exact Match: The traditional pipeline achieved an F1 score of 53.47, while the LLM achieved 43.06.
- Up to Canonicalization: The LLM showed a significant improvement, reaching an F1 score of 62.96, while the traditional pipeline scored 60.21. This highlights the LLM's strength in identifying the correct entity but weakness in exact output.
- Hybrid Model: This combined approach achieved the highest scores, with an F1 score of 58.97 in the strict "exact match" setting and 70.21 in the "up to canonicalization" setting.
Limitations and Future Work
The authors acknowledge that their study is limited to one LLM (Llama 2), one corpus (DEbateNet), and one evaluation setting. Future work could involve:
- Extrinsic Evaluation: Evaluating the actor identifier's performance on the downstream task of generating full discourse networks.
- Broader LLM and Corpus Testing: Testing other LLMs and diverse corpora to see if these findings generalize.
- Retrieval-Augmented Generation (RAG): Exploring RAG techniques to give LLMs access to external information, potentially improving their ability to find and standardize actor names.
Summary updated