Abstract
With the rise of generative AI, automated fact-checking methods to combat misinformation are becoming more and more important. However, factual claim detection, the first step in a fact-checking pipeline, suffers from two key issues that limit its scalability and generalizability: (1) inconsistency in definitions of the task and what a claim is, and (2) the high cost of manual annotation. To address (1), we review the definitions in related work and propose a unifying definition of factual claims that focuses on verifiability. To address (2), we introduce AFaCTA (Automatic Factual Claim deTection Annotator), a novel framework that assists in the annotation of factual claims with the help of large language models (LLMs). AFaCTA calibrates its annotation confidence with consistency along three predefined reasoning paths. Extensive evaluation and experiments in the domain of political speech reveal that AFaCTA can efficiently assist experts in annotating factual claims and training high-quality classifiers, and can work with or without expert supervision. Our analyses also result in PoliClaim, a comprehensive claim detection dataset spanning diverse political topics.
Introduction
Factual claim detection is the critical first step in any fact-checking pipeline. Before a statement can be debunked or verified, it must first be identified as a claim. However, this field faces two major hurdles. First, there is significant "definition drift"; researchers often conflate "check-worthiness" (subjective importance) with "verifiability" (objective checkability). Second, high-quality annotation is prohibitively expensive, requiring experts to manually label thousands of sentences, which often restricts datasets to narrow domains.
To overcome these challenges, we propose a framework centered on verifiability—whether a statement can be checked against evidence—and leverage Large Language Models (LLMs) to automate the annotation process reliably.
The AFaCTA Framework
The core of our approach is AFaCTA (Automatic Factual Claim deTection Annotator). Inspired by dual-process theory, AFaCTA employs three distinct prompting strategies to evaluate a statement's verifiability:
- Direct Classification (Fast Thinking): A simple binary prompt to identify objective information.
- Fact-Extraction Chain-of-Thought (Slow Thinking): The model decomposes sentences to separate subjective rhetoric from objective facts, providing reasoning for its verifiability determination.
- Reasoning with Debate (Multi-Perspective): Two LLM agents argue opposing sides (verifiable vs. unverifiable), and a third agent acts as a judge to determine the most convincing argument.
The framework's reliability stems from Self-Consistency. When all three reasoning paths agree, the label is categorized as "Silver" data, indicating high confidence. Disagreements flag samples for human review, effectively filtering out ambiguous cases.
Experimental Setup and Datasets
We evaluated AFaCTA using two primary datasets:
- PoliClaim: A new, comprehensive dataset of U.S. political speeches (State of the State addresses) from 1998 to 2022, containing over 11,000 labeled sentences.
- CheckThat!-2021-dev: A dataset of COVID-19 tweets re-annotated specifically for verifiability.
Key Results and Insights
Our experiments demonstrate that AFaCTA can significantly reduce the manual annotation burden while maintaining high accuracy.
"Silver" Data as a Proxy for "Gold"
A major finding is that "Silver" data—labels where the LLM achieved perfect consistency—serves as a robust substitute for human-annotated "Gold" data. Training a RoBERTa classifier on Silver data yielded performance nearly identical to training on Gold data, whereas "Bronze" data (inconsistent LLM labels) led to significantly lower performance.
Performance and Efficiency
- Accuracy: On the PoliClaim test set, GPT-4 AFaCTA achieved 98.49% accuracy on perfectly consistent samples, even outperforming human experts in some instances.
- Human-in-the-loop Efficiency: By auto-labeling consistent samples, AFaCTA can reduce manual annotation effort by approximately 48.78%.
- The Reliability Gap: GPT-4 showed a superior ability to identify implicit facts within complex opinions compared to GPT-3.5, which tended to be overly conservative.
Conclusion and Future Work
AFaCTA demonstrates that structured LLM reasoning can scale factual claim detection without sacrificing reliability. While the multi-step "Slow Thinking" process is more computationally expensive than direct prompting, the resulting data quality justifies the cost. Future work will explore reducing costs through context window optimization and fine-tuning open-source models to mitigate the position bias observed in current smaller models during the debate phase.
Summary updated