Bounding Hallucinations: Information-Theoretic Guarantees for RAG Systems via Merlin-Arthur Protocols

Björn Deiseroth, Max Henning Höth, Kristian Kersting, Letitia ParcalabescuarXiv 2025 · DOI · arXiv

TL;DR

This research introduces a novel training framework for Retrieval-Augmented Generation (RAG) that reframes the pipeline as an interactive proof system based on the Merlin-Arthur (M/A) protocol. By training models against both helpful and adversarial provers, the authors provide information-theoretic guarantees that bound hallucinations and significantly improve both generator grounding and retriever performance.

The Challenge: Brittle RAG Systems

Retrieval-Augmented Generation (RAG) is the industry standard for grounding Large Language Models (LLMs) in external data. However, current RAG systems are notoriously brittle. They often treat retrieved documents as "hints" or heuristic cues rather than verifiable evidence. This leads to several failure modes:

  • Hallucinations: The model generates answers not supported by the retrieved text.
  • Over-reliance on Parametric Knowledge: The model ignores the context and answers based on its internal training data, which might be outdated or wrong.
  • Vulnerability to Noise: Misleading or irrelevant documents in the retrieved set can easily "trick" the model into giving incorrect answers.

The core challenge is the lack of a rigorous way to ensure—and measure—that an LLM's answer is actually derived from the provided context.

Methodology: The Merlin-Arthur Protocol

The authors adapt the Merlin-Arthur protocol, a concept from computational complexity theory, to the RAG setting. They define a game with three agents:

  1. Arthur (The Verifier): The generator LLM. Its goal is to be a discerning judge.
  2. Merlin (The Helpful Prover): Uses an explainability (XAI) method called ATMAN to identify and provide the most supportive context spans to help Arthur answer correctly.
  3. Morgana (The Adversarial Prover): Also uses ATMAN, but to identify and inject the most misleading context spans to trick Arthur into a wrong answer.
The Merlin-Arthur protocol framework for RAG, showing the interaction between Arthur (Generator), Merlin (Prover), and Morgana (Adversary).

Arthur is trained to satisfy two key properties:

  • Completeness: Answering correctly when Merlin provides valid evidence.
  • Soundness: Rejecting the query or resisting a misclassification when Morgana provides deceptive input.

Crucially, this framework allows the derivation of a new metric: Explained Information Fraction (EIF). EIF uses information theory to quantify how much of the model's predictive signal is provably captured by the context features, effectively "bounding" the potential for hallucinations.

System Architecture and Training Loop

The "hard" examples generated by Merlin and Morgana aren't just for the generator. They are fed back into the retriever's training, creating a feedback loop that teaches the retriever to better distinguish between useful evidence and high-overlap "confounders."

The complete RAG pipeline integration, illustrating the document pool, retriever training with contrastive loss, and the M/A training loop.

Key Innovations

  • Autonomous Rejection Training: The model learns to say "I don't know" (reject) without needing manually annotated "unanswerable" samples. Morgana’s adversarial examples act as a synthetic training signal for healthy skepticism.
  • Disentangling Error: The authors introduce "Conditional Evaluation" to separate cases where the model fails because it's "dumb" (predictive error) from cases where the explanation doesn't match the prediction (explanation failure). This makes the EIF metric a much more reliable measure of grounding.

Evaluation and Results

The researchers evaluated their approach across three diverse RAG benchmarks: SQuAD 2.0, HotpotQA, and TriviaQA. They tested these on several LLM families, including Llama-3.2 (1B and 3B) and Qwen3-4B.

Key Findings

  • Slashing Hallucinations: Under adversarial context, M/A training reduced incorrect answers by 25–60 percentage points compared to standard instruct models.
  • Better Retrieval: The M/A-enhanced retriever showed significant gains, with Recall@1 improving by 2–11 percentage points across datasets.
  • No "Utility Tax": Robustness gains didn't hurt general performance. M/A-trained models matched the accuracy of standard supervised finetuning on clean data.
  • Provable Grounding: The models achieved a conditional EIF of $\ge 0.3$ across all datasets, providing a measurable lower bound on how much the answer depends on the context.

Limitations and Future Outlook

While promising, real-world deployment faces hurdles:

  • Source Quality: The system is limited by its sources. If evidence documents contain misinformation, the model will treat them as "proof."
  • Over-conservatism: There is a risk the model could become too "shy," refusing to answer helpful queries because it cannot find a perfect, direct citation.
  • Computational Overhead: Generating Merlin and Morgana masks adds significant cost during training, though inference remains efficient.

Future work will explore scaling to larger models and more complex, multi-turn interactive proof protocols.

Summary updated

Abstract

Retrieval-augmented generation (RAG) relies on retrieved context to guide large language models (LLM), yet treats the retrieval as a heuristic rather than verifiable evidence -- leading to unsupported answers, hallucinations, and reliance on spurious context. We introduce a novel evaluation, data augmentation, and training framework that treats the RAG pipeline as an interactive proof system by adapting the Merlin-Arthur (M/A) protocol: Arthur (the generator LLM) receives context of unknown provenance and Merlin gives helpful evidence, while Morgana injects adversarial, misleading context. We implement them both with an XAI method to self-assess and modify evidence most influential to Arthur. Based on this framework we propose the Explained Information Fraction (EIF) score, that disentangles explanation fidelity from model predictive errors and imperfect benchmarks, and normalizes M/A mutual-information lower bounds to realistic empirical settings. When trained with those contexts, Arthur learns to answer when evidence supports the answer and abstain when evidence is insufficient. Across five QA benchmarks and four LLMs (1B to 32B parameters), M/A reduces incorrect answers under insufficient context by up to 35pp during training and by 18-20pp over vanilla finetuning. Abstention emerges even \emph{without any manually annotated unanswerable example or preference pair}. We improve EIF-cond by 0.1-0.4 during M/A training and by 0.33-0.38 over vanilla finetuning. Reusing Merlin/Morgana contexts as automatic hard positives and negatives also raises retriever Recall@1 by 2pp. While high accuracy does not guarantee entropy flow from context to answer, our EIF scores -- to our knowledge, the first context-to-answer information bound for RAG generators -- show that autonomous interactive-proof-style supervision enables RAG systems that treat retrieved documents as verifiable evidence.
BibTeX
@misc{deiseroth2025bounding,
  author = {Björn Deiseroth and Max Henning Höth and Kristian Kersting and Letitia Parcalabescu},
  title = {{Bounding Hallucinations: Information-Theoretic Guarantees for RAG Systems via Merlin-Arthur Protocols}},
  year = {2025},
  eprint = {2512.11614},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.11614},
}
Figure