Abstract
Empowered by large language models (LLMs), intelligent agents have become a popular paradigm for interacting with open environments to facilitate AI deployment. However, hallucinations generated by LLMs—where outputs are inconsistent with facts—pose a significant challenge, undermining the credibility of intelligent agents. Only if hallucinations can be mitigated, the intelligent agents can be used in real-world without any catastrophic risk. Therefore, effective detection and mitigation of hallucinations are crucial to ensure the dependability of agents. Unfortunately, the related approaches either depend on white-box access to LLMs or fail to accurately identify hallucinations. To address the challenge posed by hallucinations of intelligent agents, we present HalMit, a novel black-box watchdog framework that models the generalization bound of LLM-empowered agents and thus detect hallucinations without requiring internal knowledge of the LLM's architecture. Specifically, a probabilistic fractal sampling technique is proposed to generate a sufficient number of queries to trigger the incredible responses in parallel, efficiently identifying the generalization bound of the target agent. Experimental evaluations demonstrate that HalMit significantly outperforms existing approaches in hallucination monitoring. Its black-box nature and superior performance make HalMit a promising solution for enhancing the dependability of LLM-powered systems.
Introduction
As LLM agents are increasingly deployed in high-stakes fields like medicine, law, and finance, the "hallucination" problem—where models confidently generate factually incorrect information—becomes a critical safety risk. Existing solutions generally fall into two camps: white-box methods that require internal model access, and external cross-checking which is often computationally expensive. HalMit addresses these gaps by treating hallucination detection as a boundary-monitoring task that works with any closed-source model.
The HalMit Framework
HalMit operates on the principle that hallucinations occur when a query pushes a model beyond its generalization bound—the limit of the semantic space where it can reliably perform. The framework functions in two distinct phases:
1. Generalization Bound Exploration
To map the "edge" of a model's knowledge, HalMit uses a Multi-Agent System (MAS). A Query Generation Agent (QGA) utilizes Probabilistic Fractal Sampling to generate increasingly complex queries. By using patterns of deduction, analogy, and induction, the system systematically "pokes" the target LLM to identify where its reliable knowledge ends and fabrication begins.
2. Watchdog Monitoring
Once the bound is mapped for a specific domain, the watchdog monitor evaluates live queries. It compares new inputs against the mapped boundary points using cosine similarity and measures semantic entropy (model uncertainty). If a query is likely to trigger a hallucination, it is flagged before the user relies on the output.
Experimental Results and Insights
The researchers evaluated HalMit across benchmarks like TruthfulQA, MedQuAD, and SQuAD using various backbone models including Llama 3.1 and Mistral-7b.
Domain-Specific Stability
A key insight from the study is that while hallucination characteristics vary between domains (e.g., medical vs. fiction), the patterns are remarkably stable within a single domain. This allows for the creation of reliable, domain-specific "knowledge maps" for specialized agents.
Performance Metrics
HalMit consistently outperformed baseline methods like SelfCheckGPT (SCG). In the "Inheritance" and "Modern History" domains, HalMit improved AUROC and AUC-PR metrics by up to 8%. On Llama 3.1, the framework achieved an accuracy of 0.89 in the Modern History domain, demonstrating its robustness for real-world deployment.
Conclusion
HalMit provides a promising black-box solution for enhancing the dependability of LLM-powered agents. By focusing on the input space and mapping the generalization bound, it offers a scalable and effective way to mitigate hallucinations without needing access to proprietary model internals.
Summary updated