From Detection to Diagnosis: Advancing Hallucination Analysis with Automated Data Synthesis

Yanyi Liu, Qingwen Yang, Tiezheng Guo, Feiyu Qu, Jun Liu, Yingyou Wen

TL;DR

This paper proposes shifting the focus of LLM hallucination research from simple binary "detection" to a comprehensive "diagnosis" framework that identifies, localizes, explains, and corrects errors. By using a novel automated data synthesis pipeline (HDG) and reinforcement learning (GRPO), the authors developed HDM-4B-RL, a 4-billion-parameter model that outperforms larger specialized models and rivals GPT-4 in diagnostic accuracy.

What Problem Does This Paper Address?

While Large Language Models (LLMs) are transformative, their tendency to "hallucinate"—generate factually incorrect or contextually inconsistent content—remains a major barrier to their use in high-stakes fields like medicine or law.

Currently, most research treats hallucination as a binary classification problem: Is this text a hallucination? Yes or No. The authors argue that this "detection" approach is insufficient because it provides no actionable feedback. To truly improve models or assist human editors, we need a "diagnosis" that answers:

  1. Where is the hallucination? (Localization)
  2. Why did it happen? (Explanation)
  3. How do we fix it? (Mitigation)

Methods

The researchers introduced two core innovations:

1. Hallucination Diagnosis Generator (HDG)

To train a model for diagnosis, you need high-quality, richly annotated data—which is notoriously expensive to produce manually. HDG is an automated pipeline that synthesizes this data from raw corpora (like Wikipedia) through four stages:

  • Seed Generation: Creating initial Q&A and summaries.
  • Multi-dimensional Augmentation: The "secret sauce." The pipeline intentionally injects errors by swapping entities (e.g., changing "red rose" to "pink rose"), deleting critical context, or perturbing logical reasoning chains.
  • Quality Verification: An ensemble of high-performing models (like GPT-4 and Lynx) verifies that the generated hallucinations and labels are accurate.
  • Metadata Enrichment: Adding sentence-level tags and "reasoning traces" to explain the errors.

2. HDM-4B-RL Model

Using the synthesized data, the authors fine-tuned a 4B-parameter model (Qwen3-4B) using Group Relative Policy Optimization (GRPO). Unlike standard fine-tuning, this reinforcement learning approach uses a complex reward function that scores the model on its ability to produce structured JSON outputs, its detection accuracy, and how precisely it localizes the error (overlap with ground truth).

Datasets

The authors utilized a wide array of benchmarks for evaluation:

  • QA Tasks: The HaluBench suite, including HaluEval, RAGTruth, FinanceBench, DROP, CovidQA, and PubMedQA.
  • Summarization: SummEval.
  • New Benchmark: They introduced a dedicated diagnosis benchmark derived from SummEval, featuring 285 samples with sentence-level annotations and ground-truth corrections.

Key Insights and Ideas

  • Reasoning is the Key to Detection: One of the most significant findings is that enabling "reasoning mode" (Chain-of-Thought) dramatically boosts a model's ability to spot hallucinations, particularly in math and logic-heavy tasks.
  • Specialization Trumps Scale: A 4B model specifically aligned for diagnosis can outperform 8B specialized detection models (like Lynx) and even challenge 32B general-purpose models. This suggests that for specific safety tasks, "small and smart" is a viable path.
  • The "Diagnosis" Trade-off: The authors found that while multi-step pipelines (calling different models for detection, then localization, then correction) are very accurate, they are slow and expensive. Their end-to-end HDM model provides a much more efficient alternative with comparable performance.

Key Results

  • SOTA Detection: HDM-4B-RL achieved an average Macro F1 score of 79.65 across all benchmarks, surpassing the previous state-of-the-art specialized model, Lynx-8B (77.25).
  • Rivaling Giants: In the full "Diagnosis" task, the 4B model achieved a detection accuracy of 92.28%, outperforming much larger models like o4-mini (89.12%) and GPT-4.1 (77.54%) when those models were used in a single-prompt diagnostic mode.
  • Efficiency: The model is significantly faster than larger counterparts, requiring roughly 2.2x less runtime than 32B-scale models while maintaining competitive accuracy.

Limitations and Future Work

  • Data Diversity: The current HDG pipeline relies primarily on Wikipedia. The authors acknowledge the need to expand to more specialized domains like scientific literature or legal documents to improve the model's "expert" knowledge.
  • Scaling Up: The study was limited to a 4B-parameter model due to computational constraints. Exploring whether these diagnostic capabilities scale linearly with model size is a key next step.
  • Benchmark Size: The newly created diagnosis benchmark is relatively small (285 samples); larger, more diverse human-verified benchmarks for the "diagnosis" task are needed for the field to progress.

Summary updated

Abstract

Hallucinations in Large Language Models (LLMs), defined as the generation of content inconsistent with facts or context, represent a core obstacle to their reliable deployment in critical domains. Current research primarily focuses on binary "detection" approaches that, while capable of identifying hallucinations, fail to provide interpretable and actionable feedback for model improvement, thus limiting practical utility. To address this limitation, a new research paradigm is proposed, shifting from "detection" to "diagnosis". The Hallucination Diagnosis Task is introduced, a task which requires models to not only detect hallucinations, but also perform error localization, causal explanation, and content correction. We develop the Hallucination Diagnosis Generator (HDG), an automated pipeline that systematically generates high-quality training samples with rich diagnostic metadata from raw corpora through multi-dimensional augmentation strategies including controlled fact fabrication and reasoning chain perturbation. Using HDG-generated data, we train HDM-4B-RL, a 4-billion-parameter hallucination diagnosis model, employing Group Relative Policy Optimization (GRPO) with a comprehensive reward function incorporating structural, accuracy, and localization signals. Experimental results demonstrate that our model surpasses previous state-of-the-art detection models on the HaluEval benchmark while achieving comparable performance to advanced general-purpose models. In comprehensive diagnosis tasks, HDM-4B-RL matches the capabilities of larger general models while maintaining a smaller size. This work validates the feasibility and value of hallucination diagnosis, providing an effective methodology for building more trustworthy and reliable generative AI systems.
BibTeX
@misc{liu2025from,
  author = {Yanyi Liu and Qingwen Yang and Tiezheng Guo and Feiyu Qu and Jun Liu and Yingyou Wen},
  title = {{From Detection to Diagnosis: Advancing Hallucination Analysis with Automated Data Synthesis}},
  year = {2025},
  eprint = {2601.09734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.09734},
}
Figure