Introduction
Fact-checking is a vital tool in the fight against misinformation, yet it is often hindered by a significant bottleneck: factual claim detection. Before a statement can be verified, it must first be identified as containing a verifiable assertion. This task is complicated by two primary issues: the subjective conflation of "verifiability" with "check-worthiness" and the high cost of manual expert annotation.
The AFaCTA (Automatic Factual Claim deTection Annotator) framework addresses these challenges by focusing strictly on verifiability and utilizing Large Language Models (LLMs) to scale high-quality data collection.
The AFaCTA Framework
AFaCTA employs a multi-path reasoning approach inspired by the "Fast and Slow Thinking" paradigm. Rather than a simple binary classification, the pipeline processes statements through three distinct paths to ensure reliability:
- Direct Classification: A rapid "gut reaction" label without explanation.
- Fact-Extraction Chain-of-Thought (CoT): The model extracts objective components, reasons about verifiability, and assigns a category.
- Reasoning with Debate: Two LLM agents argue opposing sides (verifiable vs. unverifiable), while a third judge evaluates the arguments. To mitigate position bias, the judge reviews the debate twice with the agents' positions swapped.
Reliability Calibration
The framework uses self-consistency as a quality metric. If all three reasoning paths yield the same label, the sample is classified as "Silver" data (perfectly consistent). Disagreements result in "Bronze" data, which may require human review. This method allows researchers to identify high-confidence labels that can be automated without sacrificing accuracy.
Key Insights and Results
Consistency as a Proxy for Truth
A major finding of the research is that when GPT-4 achieves perfect consistency across all three paths, its accuracy (98.49%) actually exceeds that of the average human expert (94.85%). This suggests that diverse reasoning paths—rather than simple repetitive sampling—force the model into a more critical and accurate state.
Scaling Training with "Silver" Data
One of the most practical applications of AFaCTA is the generation of large-scale training sets. The authors introduced PoliClaim, a dataset of U.S. political speeches. They found that classifiers like RoBERTa, when trained on "Silver" data (fully automated, high-consistency labels), perform nearly as well as those trained on human-annotated "Gold" data.
As shown in the figure above, the accuracy of models trained on Silver data closely tracks the performance of those trained on expert-labeled Golden data, demonstrating that the human bottleneck in dataset creation can be significantly reduced.
Conclusion
AFaCTA demonstrates that LLMs, specifically GPT-4, can automate approximately 50% of the annotation process for factual claim detection while maintaining expert-level accuracy. By focusing on objective verifiability and using multi-path consistency checks, the framework provides a scalable solution for building robust fact-checking pipelines. Future work will focus on reducing dependency on expensive models like GPT-4 and exploring broader context windows for more complex claims.
Summary updated