Did the Model Read the Source? Cheap Context-Ablation Checks for Auditing Retrieval-Augmented Answers

When a language model answers from retrieved documents, which is called retrieval-augmented generation or RAG, the most common error is not contradicting the documents. It is quietly adding details the documents never said (RAGTruth). A lawyer, clinician or historian checking such an answer needs to know which sentences rest on the sources and which were filled in from the model's memory.

We propose a test of that dependence that is simple and causal. Score each generated sentence under the full sources, again with the sources removed, and again with only that sentence's apparent evidence removed. A sentence whose likelihood does not move when its evidence disappears was not produced from that evidence, however well it seems to match. The test needs a handful of single-pass scoring calls rather than many sampled answers, and no access to model internals. It produces a small piece of evidence a non-expert can verify: "this sentence came from these two source lines" or "this sentence survives with no sources at all."

The opportunity sits between two results. ReDeEP showed that separating "copied from context" from "injected from memory" detects RAG hallucinations well. It does this by reading attention heads and inner layers, so it only works on open-weight LLaMA models and needs model-specific profiling. Black-box detectors such as SelfCheckGPT work over APIs but sample many full answers. They also say nothing about why a sentence is suspect, and on RAG data they are weak. We want ReDeEP's question asked from the outside, at a price and in a form that a spreadsheet user can afford and read.

Core Claim

The proposal makes two claims. First, a perturbation-based "context reliance" score is a usable black-box stand-in for ReDeEP's internal measures. Second, and more important, it adds detection signal that entailment checks miss, specifically on RAGTruth's subtle baseless information category. Its explanations are faithful by construction because they are the perturbations that move the output, and we will measure that faithfulness rather than assert it.

Let $c$ be one generated sentence, scored by its average token log-probability. We restrict the average to content tokens (names, numbers, nouns, verbs) so that generic connective phrasing does not dilute the signal. We compute it under three inputs:

  • $\ell_{\text{full}}(c)$ under the full retrieved context;
  • $\ell_{\varnothing}(c)$ with all context removed;
  • $\ell_{-E}(c)$ with only the source sentences $E$ that best align with $c$ removed.

The two scores are

$$R(c)=\ell_{\text{full}}(c)-\ell_{\varnothing}(c), \qquad D(c)=\ell_{\text{full}}(c)-\ell_{-E}(c).$$

$R$ is overall reliance on context. $D$ is the drop when the claimed evidence is deleted; it is the generative analogue of the comprehensiveness measure in ERASER.

A grounded sentence should have high $R$ and high $D$. A memory-driven addition has low $R$. A sentence that leans on context but not on the evidence it appears to cite has high $R$ and low $D$; it is drawing on something other than that evidence, and is a candidate subtle conflict. The cost per response is two whole-response scoring passes plus one pass per sentence. Each pass is a single prefill, not a generation.

Attention-based provenance cannot give this signal. ReDeEP reports that in roughly 65–77% of hallucinations, attention heads did look at the correct context, but later layers overrode it. An attention heatmap would therefore reassure the user exactly when the output is wrong. Removing the context, by contrast, tests whether the output actually depended on it.

Research Program

Perturbation design. The method builds on SOCRAT, which explains black-box sequence-to-sequence models by perturbing inputs and regressing outputs on the perturbations. SOCRAT needs dozens of latent-space perturbations per output and was never applied to LLM hallucination. Our perturbations are fewer and targeted, because in RAG the hypothesis to test is known in advance: the output depends on the context. Evidence sets $E$ are chosen by a cheap alignment step, either lexical overlap or a small entailment model, with a cap of three source sentences per claim.

We borrow the bias toward sparse, few-link explanations from Rationalizing Text Matching. That work showed people work far better with about 6 alignments than with 600 or more. We do not take its optimal-transport training, which needs a trained matching model and is 3.3× slower to train.

Two access regimes. In the scored regime, open-weight models or APIs that return log-probabilities of given text, we compute $R$ and $D$ directly. In the pure-API regime, we regenerate greedily under each ablated context. We then use an entailment model to check whether $c$, or something equivalent to it, reappears. That gives binary versions of $R$ and $D$ at the cost of a few short generations.

Detector and calibration. The detector is a logistic regression on $R$, $D$, sentence position and an entailment score. Position is included because RAGTruth finds hallucinations concentrate late in long responses. We will report the regression with and without the entailment feature. The output is a per-sentence flag plus the evidence lines whose removal moved the score.

Interface. The front end follows AdapterHub Playground, which put model actions inside Google Sheets for non-programmers. It reports that 91% of 11 non-technical researchers found its prediction action easy. It covered only classification, so extending it to generative auditing is the open problem that paper names. Each response appears as one row per sentence, holding a reliance label, a "survives without sources" indicator and links to the evidence lines. The interface is the evaluation setting, not the contribution.

Baselines

Naive reference points. The do-nothing reference flags every sentence, or flags by the base hallucination rate. It tells us how much any detector earns above prevalence.

Likelihood alone. The ablation that isolates our key ingredient is $\ell_{\text{full}}$ by itself, which is plain model confidence. If $R$ and $D$ add nothing beyond confidence, the counterfactual idea fails. A second ablation drops $D$ and keeps only $R$, which tests whether the evidence-targeted perturbation matters.

White-box comparison. ReDeEP is the strongest method on the identical question. On LLaMA2-7B RAGTruth at chunk level it reports AUC 0.7458, against RAGAS at 0.7290 and SelfCheckGPT at 0.5844 (ReDeEP). We will run on the same LLaMA2-7B responses, the same splits and at chunk level. Matching ReDeEP within about 0.01 AUC without internals would count as success for the black-box claim, and exceeding 0.7458 would be a clear win. We also report SelfCheckGPT at a matched call budget, giving it as many generations as our method uses passes.

Supervised and prompted detectors. The strongest supervised detector is the fine-tuned Llama-2-13B from RAGTruth, at 78.7% response-level F1 and 52.7% span-level F1. Optimized GPT-4 prompting reaches 63.4% and 28.3% on the same paper's measures. We do not expect an unsupervised score to beat the fine-tuned detector outright. The test is whether adding $R$ and $D$ as features to that detector, retrained on the same RAGTruth training split, raises span-level F1 above 52.7%.

Entailment alignment. The plain sentence-level entailment baseline aligns each sentence to the sources and flags unsupported ones. No reported result was found on Lacuna for an entailment-only baseline on RAGTruth, so we will run it ourselves with the same entailment model our method uses.

Why not just…?

Why not just check whether the sources entail each sentence? This is the real threat. Hallucination in RAG is defined as unsupported content, and entailment measures support directly, so where entailment is reliable, causal reliance is redundant for detection. Our bet is narrower: entailment models over-credit subtle additions that look topically supported. Low $R$, meaning the sentence survives with no sources, is an independent warning for exactly those cases. The experiment that settles it is the per-category comparison on RAGTruth's four error types. If adding $R$ and $D$ to entailment does not improve "subtle baseless introduction" detection, we will report that the contribution reduces to cheaper explanations rather than better detection.

Why not just fine-tune a detector, as RAGTruth did? A fine-tuned detector is stronger than anything unsupervised (78.7% response F1). However, it needs labeled data in the user's domain, and it outputs a flag a domain expert cannot verify. We treat it as a complementary model, not a rival. The question is whether our features improve it and whether its flags plus our evidence lines help humans more than its flags alone.

Why not just prompt GPT-4 to find and explain the hallucinations? This is what a non-expert would actually do. RAGTruth reports it reaches only 28.3% span-level F1. Its explanations are also plausible-sounding text with no tie to what the generator did, which is the plausibility-versus-faithfulness gap ERASER documents. The user study below compares it head-to-head.

Why not just show attention heatmaps? Heatmaps are free, but ReDeEP's 65–77% finding means they look grounded in most of the cases that matter. We include an attention-based provenance condition on open models to show this directly.

Proposed Experiments

Detection. The main detection benchmark is RAGTruth, covering question answering, data-to-text and summarization, with responses from Llama-2, Mistral, GPT-3.5 and GPT-4. Open-model responses use the scored regime. GPT responses use the pure-API regime, which tests whether the binary variant holds up. We report AUC, span-level F1, F1 broken down by the four error categories, and cost in model calls and tokens per response. HalluLens LongWiki serves as a secondary check on long-form generation with retrieved Wikipedia passages (HalluLens). Its 15.4% automatic retrieval-failure rate means we will hand-check a sample before trusting labels.

Faithfulness of explanations. ERASER-style comprehensiveness and sufficiency apply to the evidence lines each method surfaces. For comprehensiveness, delete the lines and measure the score drop. For sufficiency, keep only those lines and measure the score retained. We compare our lines against entailment-chosen and attention-chosen lines.

Human audit. A within-subjects study with non-programmer domain researchers audits RAGTruth responses under four conditions: no aid, GPT-4 explanations, fine-tuned-detector flags, and our spreadsheet view. Measures are span-level detection F1, time per response, and over-trust, defined as the rate of accepting flagged-clean sentences that are actually hallucinated.

Decisive checks. Three results decide the proposal:

  • whether $R$ and $D$ beat likelihood alone and approach ReDeEP's 0.7458 AUC without internals;
  • whether they lift the fine-tuned detector's 52.7% span F1, especially on subtle baseless additions;
  • whether auditors using our evidence lines catch more hallucinated spans than those reading GPT-4 explanations at equal time.

Failing the first means the black-box proxy does not work. Failing the second while passing the third means the contribution is explanation, not detection, and we would present it that way.