Is There No Such Thing as a Bad Question? H4R: HalluciBot for Ratiocination, Rewriting, Ranking, and Routing

William Watson, Nicole Cho, Nishan SrishankarAAAI 2025 · DOI

Abstract

Large Language Models (LLMs) frequently generate confident but false information, a phenomenon known as hallucination. While most mitigation strategies are reactive and analyze outputs post-generation, this research introduces HalluciBot, a lightweight encoder-based model designed to predict hallucination risk before generation begins. By identifying "bad questions"—queries that are ambiguous or trigger model biases—HalluciBot enables the H4R framework (Ratiocinate, Rewrite, Rank, and Route) to optimize LLM pipelines. Experimental results demonstrate that this approach can save up to 46.6% in wasted computation while significantly improving response accuracy across diverse datasets.

Introduction

The prevailing approach to hallucination mitigation involves "post-hoc" analysis, where generated text is scrutinized for errors. This method is computationally expensive and inherently reactive. Researchers at J.P. Morgan AI Research argue that many hallucinations are rooted in the input query itself. A query's phrasing, ambiguity, or tendency to trigger memorization biases can serve as a leading indicator of failure. HalluciBot addresses this by acting as a pre-generation gatekeeper, assessing the stability of a query before any tokens are spent on generation.

Methodology: Training HalluciBot

HalluciBot is built upon a novel training methodology that utilizes Multi-Agent Monte Carlo Simulations. To establish a ground truth for hallucination risk, a query is subjected to semantic perturbations—rewriting it in multiple semantically identical but lexically different ways.

The HalluciBot training methodology uses Monte Carlo simulations and multi-agent consensus to train a fast encoder-based classifier.

As shown in the training methodology, six independent LLM agents answer these variations. If the agents disagree or fail consistently, the original query is labeled as high-risk. To avoid the high cost of running this simulation during inference, the results are used to train a fast, standalone RoBERTa-based encoder. This classifier learns to predict the "hallucination rate" and "consensus level" directly from the raw input string.

The H4R Framework

The predictive power of HalluciBot is operationalized through the H4R framework, which integrates the model into four distinct stages of the LLM pipeline:

  1. Ratiocinate: Assessing the query's quality score to determine initial risk.
  2. Rewrite: Providing feedback to an LLM to iteratively improve the query's phrasing.
  3. Rank: Using "Best-of-N" sampling to select the most stable version of a perturbed query.
  4. Route: Deciding whether a query requires a complex Retrieval-Augmented Generation (RAG) pipeline or can be answered directly.
Comparison of standard LLM pipelines versus the H4R framework, illustrating the integration of HalluciBot (HB) for Ratiocination, Rewriting, Ranking, and Routing.

Experimental Results

The study utilized 13 diverse datasets spanning extractive (e.g., SQuADv2), multiple-choice (e.g., MMLU), and abstractive (e.g., TriviaQA) scenarios, processing over 369,000 queries.

Predictive Performance and Efficiency

HalluciBot (RoBERTa-large) achieved a 76.0% test F1 score in identifying queries prone to hallucination. More importantly, the model enabled a 46.6% reduction in computation by intercepting high-risk queries before they reached the generation stage.

Framework Gains

  • Rewriting: Guided feedback resulted in a +30.2% transition from hallucinatory to non-hallucinatory states.
  • Ranking: Selecting the most stable perturbation led to a +50.6% improvement in accuracy.
  • Routing: Dynamically switching query types based on risk assessment produced a +60.0% positive transition in response quality.

Discussion and Conclusion

A key insight of this work is the concept of "Truthful Hallucination"—the inability of a model to consistently answer semantically identical queries. By using perturbation as a stress test, HalluciBot identifies these instabilities early.

While highly effective, the model's performance is currently scenario-dependent, performing best when the query type (e.g., multiple-choice) is known. Future work will focus on improving detection for "borderline" cases where risks are subtle and integrating HalluciBot into autonomous agentic workflows to serve as a self-correction module. Ultimately, H4R proves that identifying a "bad question" is the first step toward ensuring a reliable answer.

Summary updated

BibTeX
@inproceedings{watson2025is,
  author = {William Watson and Nicole Cho and Nishan Srishankar},
  title = {{Is There No Such Thing as a Bad Question? H4R: HalluciBot for Ratiocination, Rewriting, Ranking, and Routing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i24.34736},
}
Figure