Several hallucination detectors read a language model's hidden activations. Two examples are the spread of hidden states across sampled answers (INSIDE) and linear probes that flag ungrounded text (CH-Wang et al.). Both rest on one assumption: when the model is stable and confident inside, it is drawing on real knowledge. Benchmark contamination, meaning test questions and answers leaking into training data, produces exactly that stable, confident internal state through rote memory. If a model memorized a wrong answer, these detectors should certify the falsehood as grounded. Nobody has measured this. The detectors are evaluated on benchmarks such as TruthfulQA, SQuAD and Natural Questions, whose contamination status in the evaluated models is unknown.
We propose to borrow the core move of viral load testing. A quantitative PCR machine outputs a relative signal, the number of amplification cycles before fluorescence crosses a threshold. Labs turn that signal into an absolute count of viral copies by running a dilution series of known concentrations and fitting a standard curve (Lavezzo et al.; Liao et al.). We will do the same for language models. We inject fresh, provably unseen question–answer items into continued pretraining at known copy counts (0, 1, 2, 4, …, 32). We record internal and output-level signals for each item and fit a per-model curve from signal to exposure count. We then use the curve for two jobs. The first is to read off an estimated "exposure load," in copies, for items of public benchmarks. The second is to measure how much of a hallucination detector's apparent accuracy is explained by that load.
Core Claim
Our first claim is that internal-state hallucination detectors are confounded by exposure. As the copy count of a memorized false answer rises, EigenScore and grounding probes will increasingly score that answer as truthful. Their measured accuracy on public benchmarks is therefore partly a memorization effect.
Our second claim is that a dose-calibrated readout from hidden states estimates per-item exposure in interpretable units. We further predict that it stays informative after clean reinforcement learning, the setting where output-probability detectors collapse (Fragility of Contamination Detection).
The new element is not detecting contamination, and not probing hidden states. It is the controlled dilution series used as a measuring instrument for the confound between memorization and "knowing." No prior work in the file supplies this. Schaeffer et al. inject MATH replicas into pretraining of 34M and 93M models, but they measure only output accuracy and loss. They say explicitly that they lack a model of the dose relationship, and they do not look inside the network.
Research Program
The calibrant pool. The standards must be items no model has seen. We draw them from the non-member split of WIKIMIA, which covers Wikipedia events created after 2023. We convert each event into a short question–answer pair. For a second, "counterfactual" arm we pair TruthfulQA-style questions with the imitative falsehood rather than the truth. We also pair fresh WIKIMIA questions with fabricated answers. This lets us separate "memorized" from "true."
Injection. We use continued pretraining of Llama-3-8B, Mistral-7B and Qwen2.5-7B on a FineWeb-Edu-style background stream. The calibrant items are spiked in at doses $n \in \{0,1,2,4,8,16,32\}$. Each item is assigned to exactly one dose, and dose assignments are randomized across items.
Signals per item. At the output level we record Min-K% probability, sequence loss, and the paired original-versus-paraphrase confidence gap from PaCoST. Internally we record three things. The first is EigenScore over $K$ samples. The second is the output of a linear probe at each layer, trained to separate dose-0 from dose-≥1 items. The third is a geometric signal: the layer-wise intrinsic-dimensionality profile (Joshi et al.). The profile is included because memorized items may compress earlier in depth than items the model has to work out.
The curve. By analogy to the log-linear relation between cycle threshold and copy number, which follows from per-cycle doubling, we fit
$$s_i = \alpha + \beta \log_2(n_i + 1) + \varepsilon_i,$$
where $s_i$ is a signal for item $i$, $n_i$ its injected copy count, $\beta$ the "efficiency" of that signal (how much it moves per doubling of exposure), and $\alpha$ the per-model intercept. We fit the curve on held-out items, invert it to estimate $\hat n$ for any audited item, and report $\hat n$ with prediction intervals. A signal with $\beta$ near zero, or with curvature at high dose, is reported as such rather than forced to fit.
Analysis of the confound. For every hallucination detector we compute AUROC for true-versus-false answers within each dose stratum. We also compute it after regressing out $\hat n$. The question is how much detector skill survives once exposure is held fixed.
There is an important limitation. Exposure by continued pretraining at 7B scale is a proxy for exposure in original pretraining. We test whether the proxy transfers by checking the curve's predictions on WIKIMIA member items, which are naturally pre-2017 and therefore seen in original pretraining. Those items should read as $\hat n > 0$, and dose-0 items should not.
Baselines
The naive reference point is the hallucination detectors evaluated with no regard for exposure, using the original protocols. The strongest internal-state detector in the file is EigenScore. On SQuAD with LLaMA-7B it reports 81.5% AUROC, a 5.1-point gain over lexical similarity (INSIDE). The strongest probe-based detector is the ensemble probe of CH-Wang et al., which reports 0.75 F1 on CNN/DailyMail. We rerun both on our injected checkpoints with their published layer choices, sample counts ($K$) and feature-clipping settings. We will not retune them on our data beyond what their papers describe. This test does not aim to beat these detectors. It aims to show that their skill is worse when evaluated against memorized falsehoods, and to quantify by how much. The detectors "fail" on our terms if their AUROC on counterfactual items falls monotonically with dose.
For estimating exposure, the strongest existing output-level methods are LiRA and loss-based detection. The fragility paper reports LiRA at 89.1% AUROC on SFT-contaminated Qwen2.5-7B-Instruct, falling by about 14% after one epoch of clean GRPO, while reference-free methods fall to roughly 50% (Fragility). We also run Min-K% probability, which reports a 7.4% AUC gain on WIKIMIA over the prior best (Shi et al.), and PaCoST. All of these receive the same dose-series data and are fitted to the same standard curve, so calibration is not an advantage held only by our method. Each method gets the same budget for choosing layer and hyperparameters, selected on a validation split of calibrants.
The hidden-state readout beats them under two conditions. On pre-RL checkpoints, it must give lower error in $\log_2$ copies at matched dose. On post-GRPO checkpoints, it must keep dose-1 membership AUROC clearly above the reported collapse to near 50%. "Clearly" means we pre-register a margin of at least 10 AUROC points over the best output-level signal.
Two ablations isolate the key ingredients. In the first, we train the same probe as a binary seen/unseen classifier with no dose structure. This tests whether the graded series adds anything. In the second, we fit the curve on output signals only. This tests whether hidden states add anything.
Why not just…?
Why not just calibrate Min-K% or loss against the dose series and skip the internals? This is a real threat. On models that only received supervised fine-tuning, a calibrated loss curve may be as good as anything internal, and we will report that if it happens. The case for internals rests on the post-RL setting. There, clipping in the reinforcement learning update erases the output log-probability gap (Fragility). The output-only ablation run after GRPO decides the question. If it matches the hidden-state readout, the internal component is unnecessary and the contribution reduces to the confound analysis.
Why not just evaluate on post-cutoff data and ignore contamination? Temporal splits avoid the problem for new benchmarks. They cannot audit the existing benchmarks on which hallucination detectors and steering methods such as TruthFlow report their gains. TruthFlow reports Llama-3-8B TruthfulQA "True" rising from 52.32% to 64.79%, on a benchmark whose exposure in Llama-3 is unknown. Temporal splits also cannot tell whether a detector's skill depends on exposure. That requires varying exposure while holding everything else fixed, which is what the dilution series does.
Why not just use PaCoST-style paraphrase testing? PaCoST gives a dataset-level yes/no at $p<0.05$. It is not a per-item count, and it relies on the same output confidence that reinforcement learning compresses. We include it as a calibrated baseline. If its paraphrase gap fits the standard curve well and survives GRPO, that is a cheaper answer and we will say so.
Proposed Experiments
Decisive check one: the counterfactual dose test. On items injected with false answers, we plot EigenScore and probe "grounded" scores against dose. The confound claim is confirmed if detector AUROC for true-versus-false falls steadily as dose rises. It is refuted if detectors stay flat, which would mean their internal signals separate memorization from knowledge after all.
Decisive check two: transfer to natural exposure. The curve fitted on injected calibrants should assign $\hat n>0$ to WIKIMIA members and near-zero to non-members. If it does not, continued-pretraining exposure is not a valid proxy and the audit claims are withdrawn.
Decisive check three: survival after reinforcement learning. We follow the fragility protocol: SFT contamination on MATH and GSM8K items, then one epoch of clean GRPO. We then compare dose-1 AUROC and log-copy error across all signals.
Application. Finally we read out $\hat n$ for TruthfulQA, SQuAD and Natural Questions items in the three base models. We then report hallucination detector AUROC stratified by estimated load, and TruthFlow's gain restricted to items read as low-exposure. The DCR adjustment (DCR) is computed on the same models as a black-box comparison for the benchmark-level correction.
References
- INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
- Do Androids Know They're Only Dreaming of Electric Sheep?
- Suppression of a SARS-CoV-2 outbreak in the Italian municipality of Vo’
- The effect of water temperature on the pathogenicity of decapod iridescent virus 1 (DIV1) in Litopenaeus vannamei
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Causally Quantifying the Effect of Test Set Contamination on Generative Benchmarks
- Detecting Pretraining Data from Large Language Models
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- Geometry of Decision Making in Language Models
- TruthFlow: Truthful LLM Generation via Representation Flow Correction
- DCR: Quantifying Data Contamination in LLMs Evaluation