Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers

TL;DR

This paper tackles the critical issue of ensuring LLM-generated information is factual by rigorously evaluating existing fact-verification models. They reveal that dataset imperfections (errors and ambiguity) significantly skew evaluation results and that simple few-shot prompting with large LLMs offers surprisingly strong performance, while also highlighting the need for more capable smaller models for complex reasoning tasks.

What Problem Does This Paper Address?

As Large Language Models (LLMs) become more integrated into applications, their tendency to "hallucinate" – generate incorrect or misleading information – poses a significant challenge to their reliability. Fact verification is a crucial component for mitigating this, acting as a gatekeeper to ensure LLM outputs are grounded in reality. However, existing fact-verification benchmarks and evaluation methodologies have limitations. This paper addresses the need for a deeper understanding of these fact verifiers by:

1. Identifying and quantifying the impact of dataset imperfections: Many existing fact-checking benchmarks contain annotation errors and ambiguous instances that can lead to misleading conclusions about model performance.
2. Evaluating a wide range of modern LLMs as fact verifiers: The study assesses both frontier LLMs (like GPT-4) and open-weight models, including those with few-shot capabilities, which are often overlooked.
3. Investigating the performance of smaller, fine-tuned fact verifiers: While large LLMs are powerful, their computational cost is a barrier. The paper explores the potential and limitations of smaller, specialized models.
4. Proposing methods to improve fact verification: The research aims to guide future development of more robust and reliable fact-verification systems.

Methods

The authors employ a multi-pronged approach:

• Dataset Curation and Refinement: They start with a collection of 14 fact-checking benchmarks. To address annotation errors and ambiguity, they develop an LLM-as-a-judge pipeline to systematically identify problematic examples. This pipeline helps filter out unverifiable statements and instances with ambiguous labels. The refined dataset is split into CLEARFACTS (corrected/unambiguous) and GRAYFACTS (ambiguous).
• LLM-as-a-Judge Framework: To scale the identification of errors and ambiguities, they leverage multiple frontier LLMs to evaluate model outputs. This allows for a more comprehensive assessment of correctness, logical coherence, and faithfulness.
• Model Evaluation: They evaluate 12 pre-trained LLMs and one specialized fact-verifier (MiniCheck 7B) using both zero-shot and few-shot prompting strategies.
• Synthetic Data Generation: To improve the reasoning capabilities of smaller models, they generate synthetic multi-hop reasoning data.
• Fine-tuning: They fine-tune a smaller LLM (Llama 3.1 8B Inst) on a combination of existing data (ANLI) and their newly generated synthetic multi-hop data to create a model named CLEARCHECK.

Datasets

• Initial Collection: 14 publicly available datasets for fact verification, including LLM-AggreFact (11 datasets), SciFact, Hover, and CoverBench.
• Refined Datasets:
  • CLEARFACTS: A cleaned dataset with corrected labels and removed ambiguous instances.
• GRAYFACTS: A dataset containing only the ambiguous instances.
• Training Data: ANLI dataset and synthetic multi-hop reasoning data generated by the authors.

Key Insights and Ideas

• Dataset Quality is Paramount: The study strongly emphasizes that even small percentages of mislabeled or ambiguous data (around 16% in their initial collection) can drastically alter model rankings and lead to incorrect conclusions about model performance. This highlights the critical need for robust dataset cleaning and validation pipelines.
• Few-Shot LLMs are Underestimated Baselines: Frontier LLMs, when prompted with a few examples (few-shot learning), achieve top-tier performance in fact verification, often outperforming more complex setups. This simple yet effective approach is often overlooked in favor of more intricate methods.
• Small Models Struggle with Complex Reasoning: While fine-tuned smaller models are computationally efficient, they significantly lag behind larger LLMs on tasks requiring multi-hop reasoning. This gap is particularly evident on benchmarks like CoverBench and Hover.
• Synthetic Data Can Boost Reasoning: Augmenting training data with synthetically generated multi-hop reasoning examples can significantly improve the performance of smaller fact verifiers on complex reasoning tasks, bridging some of the performance gap.

Key Results

• Impact of Data Cleaning: Model rankings shifted notably after refining the dataset. For instance, MiniCheck initially ranked higher than some larger models but fell after data cleaning.
• Few-Shot Performance: Frontier LLMs with few-shot prompting achieved top performance. For example, zero-shot o1 scored 9.4, while few-shot o1 achieved 88.7 Macro F1 on CLEARFACTS.
• Small Model Limitations: MiniCheck 7B, a fine-tuned model, performed well on simpler datasets but struggled on CoverBench and Hover, which require complex reasoning.
• Synthetic Data Improvement: Training with synthetic multi-hop reasoning data improved CLEARCHECK's performance on complex reasoning benchmarks like Hover and CoverBench.
• Error Analysis: The most common error types for fact verifiers were "Missing fine-grained detail of the statement" (47.1%) and "Misinterpretation of evidence" (36.8%).

Limitations and Future Work

• Unannotated Data: The authors acknowledge the possibility of annotation errors and ambiguous examples in the portions of the dataset they did not manually inspect. They chose not to manually annotate all examples to focus on developing a scalable pipeline for identification.
• Generalizability of Findings: The 16% mislabeled/ambiguous rate is based on their initial collection after some filtering, and they caution against generalizing this exact figure to all datasets.
• Ambiguity in Real-World Scenarios: They highlight that real-world statements are often ambiguous, and current evaluation efforts should better account for this.
• Closing the Gap for Small Models: While synthetic data helps, there's still a performance gap between smaller fine-tuned models and frontier LLMs. Future work could focus on developing more effective training algorithms to further improve smaller models' capabilities, especially for complex reasoning.