ClaimLens: Automated, Explainable Fact-Checking on Voting Claims Using Frame-Semantics

Jacob Devasier, Rishabh Mediratta, Phuong Anh Le, David Huang, Chengkai LiEMNLP 2024

TL;DR

This paper introduces ClaimLens, an automated fact-checking system for voting-related claims that prioritizes transparency and explainability. By leveraging frame-semantic parsing, ClaimLens breaks down claims into structured components, allowing for verifiable fact-checking against official voting records and providing clear explanations for its conclusions.

What Problem Does This Paper Address?

The proliferation of misinformation online, particularly concerning political claims, necessitates effective and efficient fact-checking solutions. While automated fact-checking systems exist, they often suffer from a critical lack of transparency, making it difficult for users to trust or understand their reasoning. This "black box" nature is especially problematic in high-stakes domains where credibility is paramount. Manual fact-checking is too slow to combat the rapid spread of false information. Existing automated methods, while improving, often provide post-hoc explanations that can be unreliable or even fabricated, leading to a trust deficit. ClaimLens aims to bridge this gap by offering an explainable system that clearly articulates its fact-checking process.

Methods

ClaimLens employs a novel approach centered around frame-semantic parsing. This technique breaks down a claim into its core components, identifying semantic frames (events, concepts) and frame elements (roles within the frame). For voting claims, they specifically utilize the "Vote" frame to extract key elements like the Agent (who voted), the Issue (what they voted on), and their Position (for/against).

The system then performs:

  • Agent Lookup: Matches the extracted Agent FE to specific members of Congress, using Wikipedia and nickname lists to handle variations.
  • Semantic Bill Search: Identifies relevant legislative bills based on the Issue FE, moving beyond simple keyword matching to semantic similarity.
  • Vote-Claim Alignment: Uses Large Language Models (LLMs) to compare the extracted bill information and voting records against the claim, determining if the vote supports, refutes, or is irrelevant to the claim. This step is crucial for providing the explanation.

To address the scarcity of annotated data for frame-semantic parsing, the authors developed two novel data augmentation techniques:

  • FE Interleaving: Swaps frame elements between two sentences with the same frame to create new training examples.
  • FE Permutation: Generates new sentences by reordering frame elements within an existing sentence, helping models generalize to different sentence structures.

Datasets

The paper utilizes and introduces several datasets:

  • United States Congress Dataset: A comprehensive collection of bills, votes, and congress members from official US voting records (1789-2024).
  • PolitiFact Fact-Checks: Used as a source for identifying voting-related claims.
  • Bill Matching Evaluation Set: A newly introduced benchmark of 79 voting-related factual claims and their corresponding bills, derived from PolitiFact fact-checks.
  • Vote Frame Dataset: An existing dataset of voting-related claims annotated with the Vote frame, augmented by the authors.
  • Vote GPT Negatives: Synthetically generated negative samples using GPT-3.5 to improve frame identification.

Key Insights and Ideas

  • Frame Semantics for Explainability: The core idea is that by structuring claims using frame semantics, the fact-checking process becomes inherently more interpretable. Each extracted element can be directly linked to evidence.
  • Beyond Keyword Matching for Bills: The realization that simply searching for keywords in bill titles or descriptions is insufficient, leading to the development of semantic search for better Issue FE matching.
  • LLMs for Nuanced Alignment: The use of LLMs for vote-claim alignment is a pragmatic approach to capture the complex relationships between legislative actions and factual claims, providing detailed explanations.
  • Data Augmentation is Crucial: The significant performance gains achieved through FE interleaving and permutation highlight the challenge of limited annotated data in frame-semantic parsing and the effectiveness of these novel augmentation strategies.

Key Results

  • The proposed data augmentation techniques for frame-semantic parsing led to a +9.5% macro F1 score on frame element identification over their baseline.
  • Combining FE interleaving (with high CoLA score filtering) and FE permutation yielded the best performance, achieving a macro F1 score of 0.708 for frame-semantic parsing.
  • Semantic search models significantly outperform the BM25 keyword search baseline, with some models achieving much higher recall.
  • Qualitative assessment of LLMs for claim alignment showed that GPT-4 and GPT-40, along with Claude 3 Opus, demonstrated higher concordance with human judgment compared to other models. GPT-40 was selected for implementation due to its comparable performance and cost-effectiveness.

Limitations and Future Work

  • Coreference Resolution: The system currently struggles with resolving pronouns (e.g., "he," "she") for Agent FEs, limiting its ability to process claims involving indirect references.
  • Final Vote Simplification: The system only considers the final vote on a bill, potentially overlooking important details from amendments or preliminary votes.
  • Agent Ambiguity: When multiple individuals share the same Agent FE, the system defaults to the most recent congress member, which might not always be accurate historically.
  • Database Maintenance: Keeping the voting records database up-to-date requires ongoing effort to monitor congressional APIs.
  • Claim Metadata: The system does not currently incorporate claim metadata like the date, which could be critical for time-sensitive claims.

Future work includes:

  • Expanding the system to support other semantic frames beyond "Vote."
  • Investigating alternatives to LLMs for vote-claim alignment to improve speed, potentially using textual entailment tasks.
  • Applying the data augmentation techniques to the broader FrameNet dataset to assess their generalizability.

Summary updated

BibTeX
@inproceedings{devasier2024claimlens,
  author = {Jacob Devasier and Rishabh Mediratta and Phuong Anh Le and David Huang and Chengkai Li},
  title = {{ClaimLens: Automated, Explainable Fact-Checking on Voting Claims Using Frame-Semantics}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure