HalluDetect: Detecting, Mitigating, and Benchmarking Hallucinations in Conversational Systems in the Legal Domain

Spandan Anaokar, Shrey Ganatra, Harshvivek Kashid, Swapnil Bhattacharyya, Shruthi Nair, Reshma Sekhar, Siddharth Manohar, Rahul Hemrajani, Pushpak BhattacharyyaEMNLP 2025

TL;DR

This paper introduces HalluDetect, a novel LLM-based system for detecting, mitigating, and benchmarking hallucinations in legal chatbots. It achieves high accuracy in identifying factual errors and demonstrates that specialized architectures like AgentBot can significantly reduce hallucinations, making AI-driven legal assistance more reliable.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly used in critical applications like customer support, healthcare, and legal services. However, they are prone to "hallucinations" – generating factually incorrect or unsupported information. In the legal domain, these hallucinations can mislead users, leading to significant legal and financial consequences. Existing hallucination detection methods often struggle with multi-turn conversations and can misclassify benign conversational elements as errors. This paper addresses the need for a robust, scalable system to detect and mitigate these hallucinations specifically within the complex domain of Indian consumer law chatbots.

Methods

The core contribution is HalluDetect, a multi-stage LLM pipeline designed for detecting hallucinations in multi-turn RAG-based conversations. HalluDetect addresses the "domain reliance paradox" by balancing retrieval-based grounding with the LLM's general knowledge. Its pipeline involves:

  1. Analysis: Identifies potential hallucinations by comparing responses against retrieved documents and a summarized conversation history.
  2. Filtering: Assigns a severity score to each potential hallucination and filters out low-impact ones.
  3. Aggregation: Compiles high-severity hallucinations with justifications and locations in the dialogue.

The paper also benchmarks five RAG-based chatbot architectures designed for hallucination mitigation: Vanilla, Prompt-engineered, EditorBot, FactChecker, and AgentBot. These architectures vary in their approach to integrating retrieval and verification mechanisms.

Datasets

The paper introduces the DetectorEval dataset, a benchmark consisting of 115 multi-turn dialogues (average 7 turns) between a Vanilla chatbot and legal experts. These conversations are annotated by human experts to identify hallucinations, providing a ground truth for evaluating detection systems.

Key Insights and Ideas

  • Hallucinations are subtle and critical in law: Unlike simple errors, legal hallucinations can have severe consequences, making accurate detection paramount.
  • Multi-turn context is key: Hallucinations often arise from long conversational histories, making single-turn detectors insufficient. HalluDetect's memory summarization is crucial here.
  • Agent-based architectures excel: The AgentBot, which mimics real-world legal processes with specialized roles (Receptionist, Paralegal, Lawyer, Drafter), significantly reduces hallucinations by breaking down complex tasks.
  • Balancing precision and recall: HalluDetect achieves a strong balance, detecting most hallucinations (high recall) while minimizing false positives (high precision) by filtering benign conversational elements.
  • Not all "errors" are hallucinations: HalluDetect's severity filtering correctly distinguishes between critical factual errors and polite, inferential, or conversational statements.

Key Results

  • HalluDetect Performance: Achieves an F1 score of 68.92% on the DetectorEval dataset, outperforming baseline detectors by 22.47%. It demonstrates an average precision of 54.56% and recall of 80.63%.
  • Mitigation Effectiveness: AgentBot is the most effective mitigation strategy, achieving the lowest Hallucinations per Turn (HPT) of 0.42 and the highest Token Accuracy (TokAcc) of 96.13%.
  • Chatbot Comparison: AgentBot leads in hallucination reduction, followed by FactChecker and EditorBot, while Vanilla and Prompt-engineered bots show significantly higher hallucination rates.
  • Efficiency: HalluDetect introduces a moderate computational overhead, approximately 2.52x that of the baseline chatbot, making it practical for real-world auditing.

Limitations and Future Work

  • Domain Specificity: While designed to be domain-agnostic, the current evaluation is limited to the legal consumer grievance domain. Its effectiveness in other domains needs empirical validation.
  • LLM Limitations: As HalluDetect itself is LLM-based, it may occasionally hallucinate or misclassify outputs, especially in ambiguous cases.
  • Future Work: The authors suggest extending the framework to other sensitive domains and exploring more advanced mitigation techniques. They also highlight the need for continued research into robust hallucination detection and mitigation strategies for LLMs.

Summary updated

BibTeX
@inproceedings{anaokar2025halludetect,
  author = {Spandan Anaokar and Shrey Ganatra and Harshvivek Kashid and Swapnil Bhattacharyya and Shruthi Nair and Reshma Sekhar and Siddharth Manohar and Rahul Hemrajani and Pushpak Bhattacharyya},
  title = {{HalluDetect: Detecting, Mitigating, and Benchmarking Hallucinations in Conversational Systems in the Legal Domain}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure