TL;DR
This paper introduces PLEDGETRACKER, a novel system that moves beyond simple document classification to monitor the fulfillment of political pledges by constructing dynamic, temporal event timelines. By integrating evidence retrieval, timeline construction, and a unique fulfillment filtering module, PLEDGETRACKER offers a more accurate and interpretable way to track government accountability, outperforming existing tools like Google Search and GPT-4.
What Problem Does This Paper Address?
Political pledges are central to democratic accountability, but tracking their fulfillment is a complex, resource-intensive task. Existing automated methods often simplify this by treating it as a static document classification problem – essentially asking if a single article supports a pledge. This overlooks the dynamic, multi-document, and temporal nature of pledge fulfillment. A pledge isn't just "fulfilled" or "not fulfilled"; it evolves through a series of actions, milestones, and sometimes even setbacks over time. This requires systems that can collect and reason over temporally distributed evidence from various sources to build a coherent picture of progress.
Methods
PLEDGETRACKER is a Retrieval Augmented Generation (RAG) system built around three core modules:
- Evidence Retrieval: This module uses a multi-round, question-driven approach to collect relevant documents. It starts with an initial web search, then iteratively refines queries by generating hypothetical documents and clarification questions based on retrieved evidence. This aims to capture a broader and more relevant set of information than standard search.
- Timeline Construction: Instead of relying on predefined schemas, this module uses a generative approach (GPT-4) to extract key event descriptions and their timestamps from the retrieved documents. It's designed to handle various temporal expressions and output them in a structured format.
- Fulfilment Filtering: This is a crucial novel component. It filters the extracted events to identify those that are genuinely informative and relevant to tracking pledge fulfillment, discarding background information or irrelevant details. This module was developed with the help of professional fact-checkers and a custom-annotated dataset.
Datasets
The paper introduces a new annotated dataset for pledge monitoring. This dataset consists of 1,559 event descriptions across 50 pledges, with each event labeled for its relevance to pledge fulfillment. This dataset was created by professional fact-checkers from Full Fact, who examined timelines generated by PLEDGETRACKER (without the filtering module) for pledges from the Labour Party's 2024 UK election manifesto.
Key Insights and Ideas
- Pledge monitoring is fundamentally a timeline construction problem: The dynamic and temporal nature of pledge fulfillment means that simply classifying documents isn't enough. Building a chronological sequence of events is key.
- Iterative, question-driven retrieval is more effective: The PLEDGETRACKER's evidence retrieval module, which uses generated questions to refine searches, significantly outperforms standard search in finding useful URLs.
- Fulfilment filtering is essential: A large majority of extracted events are not directly useful for tracking pledge progress. A dedicated filtering module is necessary to surface the truly informative events.
- Real-world evaluation with fact-checkers is critical: Integrating the system into the workflows of professional fact-checkers provided valuable qualitative feedback and demonstrated practical utility.
Key Results
- Fulfilment Filtering: The fulfillment filtering module achieved an F₁ score of 0.633 when evaluated using GPT-4, demonstrating its effectiveness in identifying useful events.
- Real-world Evaluation: In a real-world setting with Full Fact fact-checkers, PLEDGETRACKER achieved a precision of 0.764, recall of 0.553, and F₁ of 0.641 in identifying fulfillment events.
- Comparison with Existing Tools: PLEDGETRACKER significantly outperformed Google Search and GPT-4 with web search in retrieving useful URLs for pledge monitoring. It achieved a URL-level F₁ of 0.78 compared to Google Search's 0.23 and GPT-4's 0.03. Notably, PLEDGETRACKER also surfaced 36 unique, useful URLs that the other systems missed.
- Qualitative Feedback: Fact-checkers found PLEDGETRACKER helpful in surfacing overlooked news (like a High Court ruling), identifying events in a timely manner, and highlighting legislative signals.
Limitations and Future Work
The authors acknowledge several limitations:
- Pledge Identification: PLEDGETRACKER assumes pledges are already identified and normalized; it doesn't handle the extraction of pledges from manifestos.
- Contextual Bias: The evaluation focused on UK political parties, and its effectiveness in other linguistic or institutional contexts needs further exploration.
- Search API Reliance: The system's evidence coverage is limited by the Google Custom Search API, which can introduce ranking biases and quota constraints.
- Model Scale: Due to resource limitations, smaller models and LLM APIs were used, preventing large-scale training.
Future work could involve extending the system to automatically extract and decontextualize pledges, exploring its performance in different political contexts, and investigating alternative retrieval strategies to mitigate reliance on specific search APIs.