DiVA: Fine-grained Factuality Verification with Agentic-Discriminative Verifier

Hui Huang, Muyun Yang, Yuki ArasearXiv 2026 · arXiv

Abstract

Despite the significant advancements of Large Language Models (LLMs), their factuality remains a critical challenge, fueling growing interest in factuality verification. Existing research on factuality verification primarily conducts binary judgments (e.g., correct or incorrect), which fails to distinguish varying degrees of error severity. This limits its utility for applications such as fine-grained evaluation and preference optimization. To bridge this gap, we propose the Agentic Discriminative Verifier (DiVA), a hybrid framework that synergizes the agentic search capabilities of generative models with the precise scoring aptitude of discriminative models. We also construct a new benchmark, FGVeriBench, as a robust testbed for fine-grained factuality verification. Experimental results on FGVeriBench demonstrate that our DiVA significantly outperforms existing methods on factuality verification for both general and multi-hop questions.

Introduction

Large Language Models (LLMs) are notorious for "hallucinating"—generating plausible but false information. While tools exist to verify these claims, most current methods suffer from two major flaws: Binary Blindness and the Knowledge Gap.

Binary systems only provide a "Yes/No" label, failing to distinguish between a minor factual slip-up and a total fabrication. Furthermore, generative verifiers (like GPT-4) are often limited by their internal training data, while specialized discriminative scorers lack the ability to autonomously retrieve external evidence. DiVA aims to bridge this gap by creating a system that can autonomously search for evidence and provide a nuanced, continuous score of how factual a statement actually is.

Binary vs. Fine-grained Verification. While binary systems only mark answers as correct or incorrect, DiVA provides a continuous score reflecting the degree of truthfulness.

The DiVA Framework

DiVA employs a hybrid "Agentic-Discriminative" pipeline designed to combine the reasoning and search capabilities of generative agents with the precise grading of discriminative models. The process follows three distinct stages:

The DiVA inference pipeline. It consists of (1) Agentic Search for evidence retrieval, (2) Context Compression for noise reduction, and (3) Score Prediction via a discriminative module.

A generative LLM acts as an agent to verify a claim. It breaks the claim down into search queries and uses tools (such as Google Search or local Wikipedia dumps) to retrieve evidence. It operates in an iterative loop—thinking, searching, and observing—until it has gathered sufficient information to make a judgment.

2. Context Compression

Raw search results are often noisy. DiVA uses a generative module to condense the search trajectory into a "high-signal" summary. This summary contains only the useful facts and the necessary reasoning steps, ensuring the final scorer is not overwhelmed by irrelevant information.

3. Score Prediction

The compressed information is fed into a specialized discriminative module—an LLM augmented with a regression head. Instead of predicting the next word, this module outputs a continuous scalar score representing the factuality of the answer.

Training Strategy and FGVeriBench

To train DiVA without the need for manual absolute scoring, the researchers utilized pairwise ranking. The model was shown two different answers and taught to rank the more factual one higher using a margin ranking loss.

To evaluate this approach, the authors introduced FGVeriBench, a new benchmark spanning General QA (Natural Questions, TriviaQA) and Multi-hop QA (HotpotQA, Musique). The benchmark was constructed using an "LLM-as-a-judge" pipeline followed by rigorous human verification to ensure accurate relative rankings of candidate answers.

Key Results and Insights

Experimental results demonstrate that DiVA consistently outperforms state-of-the-art verifiers like FactScore and MiniCheck.

  • Superior Accuracy: On the NQ dataset, DiVA achieved a Precision@1 of 84.24%, significantly higher than FactScore’s 77.64%.
  • Multi-hop Mastery: The performance gap was even wider in complex scenarios. On the Musique dataset, DiVA (79.87%) far outperformed a Naive Generative Verifier (57.14%), proving that agentic search is essential for complex, multi-step questions.
  • The Importance of Compression: Removing the "Context Compression" step led to a substantial drop in performance, confirming that scorers perform better when provided with concise, high-signal evidence.
  • Best-of-N Sampling: DiVA can also be used to improve model generation. By generating 16 different answers and using DiVA to select the best one, the final accuracy of models like Llama-3 was significantly improved.

Conclusion

DiVA represents a significant step forward in factuality verification by moving beyond binary labels to a more nuanced, fine-grained scoring system. By separating the finding of facts from the scoring of facts, DiVA provides a robust framework for evaluating and optimizing the factuality of Large Language Models. Future work will look into unifying the generative and discriminative modules and integrating DiVA's scores into RLHF pipelines.

Summary updated

Abstract

Despite the significant advancements of Large Language Models (LLMs), their factuality remains a critical challenge, fueling growing interest in factuality verification. Existing research on factuality verification primarily conducts binary judgments (e.g., correct or incorrect), which fails to distinguish varying degrees of error severity. This limits its utility for applications such as fine-grained evaluation and preference optimization. To bridge this gap, we propose the Agentic Discriminative Verifier (DiVA), a hybrid framework that synergizes the agentic search capabilities of generative models with the precise scoring aptitude of discriminative models. We also construct a new benchmark, FGVeriBench, as a robust testbed for fine-grained factuality verification. Experimental results on FGVeriBench demonstrate that our DiVA significantly outperforms existing methods on factuality verification for both general and multi-hop questions.
BibTeX
@misc{huang2026diva,
  author = {Hui Huang and Muyun Yang and Yuki Arase},
  title = {{DiVA: Fine-grained Factuality Verification with Agentic-Discriminative Verifier}},
  year = {2026},
  eprint = {2601.03605},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.03605},
  howpublished = {arXiv},
}
Figure