When researchers test whether a language model can be tricked into stating falsehoods, they usually plant a false claim in the question ("As we know, antibiotics cure viral infections, so…"). They then ask another model to judge whether the target went along with it, and report the fraction of successful tricks as an attack success rate (ASR). That single number is now used to rank models and to draw scientific conclusions. One recent example is the claim that some heavily aligned models are more easily fooled by hesitant phrasing ("I guess…") than by confident phrasing. But the number mixes four different things. It mixes a model that does not know the fact with a model that knows the fact and caves anyway. It mixes either of those with a model whose safety filter returned an empty or evasive answer. And it mixes all three with a judge that misread the response. We propose to separate these components and estimate each with known error bars. We then re-test the headline findings of existing adversarial factuality benchmarks to see which survive.
The core idea fits in two sentences. First, measure each model's knowledge of each fact without any attack, and correct the automated judge's verdicts using a small human-labeled calibration set. Second, split the corrected attack success rate into "capitulation" (the model knew the truth but adopted the false premise) and "knowledge gap" (the model never knew it), so that the number we call hallucination ASR actually measures what it claims to.
Core Claim
Current adversarial factuality ASR is not a valid measure of hallucination, for three separable reasons. Each can be estimated and corrected with modest human labeling.
The first reason is judge error. The judges disagree with humans at rates that are unknown, and these rates probably differ by model and by prompt tier.
The second is that refusals and evasions are counted inconsistently.
The third is that failures on facts the model does not know are pooled with failures on facts it does know. Only the latter are what the hallucination literature would call a model contradicting its own knowledge, and what the safety literature would call a successful attack.
We predict that once judge error is corrected, part of the reported confidence-tier inversion shrinks. We also predict that capitulation on known facts and knowledge gaps respond to prompt framing in qualitatively different ways. We state this as a falsifiable prediction, not a foregone result.
Why this gap is real
The motivating evidence comes from two benchmarks. Battling Misinformation injects false statements at three confidence levels into the 209-item Adversarial Factuality dataset and scores eight open models with GPT-4o as judge. It reports that Phi 3 rises from 52.63% ASR under confident framing to 93.78% under hesitant framing, and LLaMA 3.1 from 4.78% to 10.05%, while Qwen 2.5 falls from 34.45% to 7.66%. The paper does not estimate the judge's error rates. AdversaRiskQA builds 400 health, finance and law items with confident false premises and scores them with GPT-5-mini plus a search agent. It notes that safety filters produce blank or repetitive outputs, which it handles by post-hoc filtering: raw accuracy of 90.2% for GPT-5 and filtered accuracy of 94.7% for Qwen3-Next-80B. Neither benchmark asks whether the model knew the fact before being attacked.
Our method comes from four papers that each supply one tool or one warning.
Comparison requires valid measurement argues that ASR comparisons need a defined estimand. Taking the best of K samples and scoring a single sample measure different quantities, and judges with model-dependent true- and false-positive rates make cross-model comparisons invalid. We adopt its estimand discipline and apply it to factuality attacks, which it does not study.
LoFT shows that a compliant-sounding preamble followed by harmless evasion inflates ASR under string matching. The factuality analogue is a polite "That's an interesting point…" that never actually endorses the falsehood.
Are Factuality Checkers Reliable? shows that classifier-based factuality checkers lose more than 50 points of accuracy under antonym swaps. That warns against cheap lexical scoring, though it tested small checkers rather than modern LLM judges.
Position: Adversarial ML for LLMs Is Not Making Any Progress criticizes LLM-as-judge evaluation as circular, which is why we anchor everything to human labels.
The conceptual split between "doesn't know" and "knows but caves" follows HalluLens. HalluLens separates factuality errors from hallucinations that contradict the model's own training knowledge, but it does not apply the distinction to adversarial premises.
Do Language Models Know When They're Hallucinating References? supplies the probing technique. It shows that asking indirect questions across independent contexts and measuring how consistent the answers are exposes what a model actually knows. We reuse this to classify facts as known or unknown before any attack.
Research Program
For every item, we first measure the model's clean-context knowledge. We ask it the neutral version of the question, with no false premise, several times in fresh contexts, together with indirect paraphrases in the style of Agrawal et al. An item counts as known if the model answers it correctly and consistently, unknown if it answers wrongly or inconsistently, and ambiguous otherwise. The ambiguous items are reported separately rather than forced into a bin.
We then run the attack at each confidence tier. Each response is labeled into one of four categories: correct rebuttal, adoption of the false premise, evasion or refusal, and empty or degenerate output. Splitting out the last two categories handles AdversaRiskQA's filter problem directly instead of discarding those responses.
To calibrate the judge, humans label a stratified sample of responses, stratified by model, tier and judge verdict. From these labels we estimate the judge's true-positive rate $\mathrm{TPR}_{m,t}$ and false-positive rate $\mathrm{FPR}_{m,t}$ for model $m$ and tier $t$. We do this separately per cell because differential error is exactly the failure that Chouldechova et al. warn about. The corrected rate uses the standard misclassification correction:
$$\widehat{\mathrm{ASR}}_{m,t} = \frac{\hat p_{m,t} - \mathrm{FPR}_{m,t}}{\mathrm{TPR}_{m,t} - \mathrm{FPR}_{m,t}},$$
where $\hat p_{m,t}$ is the judge's raw rate of declaring attack success. We obtain confidence intervals by bootstrapping over both the attacked items and the calibration labels. The corrected rate then decomposes as
$$\widehat{\mathrm{ASR}} = \pi_{\text{known}}\,\mathrm{ASR}_{\text{known}} + \pi_{\text{unknown}}\,\mathrm{ASR}_{\text{unknown}},$$
where $\pi$ is the fraction of items in each knowledge class. We call $\mathrm{ASR}_{\text{known}}$ the capitulation rate. It is the quantity that actually measures an attack overriding the model's own knowledge.
Following Chouldechova et al., all headline numbers are single-sample rates at a fixed decoding temperature. Best-of-K rates are reported separately and labeled as a different estimand.
The correction formula itself is textbook statistics, and we do not claim it as the contribution. The contribution is the protocol as a whole, the decomposition, and the re-test of published findings under it.
Baselines
The naive reference point is the raw judge verdict exactly as the source papers compute it. That means GPT-4o judging with the prompts of Battling Misinformation, and GPT-5-mini (or an open substitute if it is unavailable) for AdversaRiskQA. We reproduce it on the same models and items, and check that our reproduction approaches the reported figures, for instance Phi 3 at 52.63% versus 93.78% across tiers. The proposal beats this reference if the corrected estimates differ from the raw ones by more than their bootstrap interval in at least some cells, especially if the difference is differential across tiers. If correction never moves the numbers, the existing benchmarks are vindicated, and we would report that.
The strongest existing practice is AdversaRiskQA's filtered accuracy: drop degenerate outputs, then score the rest with an LLM judge plus web-search verification. We run it on the same items, models and decoding settings. It beats us if its filtered score agrees with human labels as well as our corrected estimate does, on a held-out set of human-labeled responses not used for calibration.
A second strong competitor is an ensemble of judges from different model families with majority vote. We give it the same item set and the same budget of prompt engineering as our single judge, with no human labels.
Two ablations isolate our ingredients. One applies judge correction without the knowledge split. The other applies the knowledge split with raw judge verdicts. Together they show whether the change in conclusions comes from fixing measurement error or from conditioning on knowledge.
Why not just…?
Why not just use a stronger judge, or an ensemble of judges? This is the most serious threat. If an ensemble from different model families has low error that does not vary by model or tier, then simply recommending ensembles would be enough. The concern is that judges share failure modes when they read a hedged answer. Agreeing with the user's tone but not their claim is exactly the kind of case where LLM judges are criticized as circular in Position: Adversarial ML for LLMs Is Not Making Any Progress. Correlated errors are not removed by voting. Our held-out human set measures the ensemble's per-cell error directly, so the experiment settles the question either way. Even if the ensemble wins on judging, the knowledge decomposition still stands on its own.
Why not just have humans label every response? For the 209-item dataset across a handful of models and three tiers, full human labeling is affordable. We will do it for two models, to validate the correction against ground truth. The correction matters when the benchmark is extended to new models, tiers or domains, where relabeling everything each time does not scale. Full labeling also does not answer the scientific question: it tells you how often the model adopted the premise, not whether it knew better.
Why not just string-match for the injected falsehood, or use multiple choice? LoFT documents how polite preambles fool surface matching. Are Factuality Checkers Reliable? shows surface-level checkers collapse under simple edits such as negation. A response that repeats the false claim in order to rebut it would match the string and be scored as a success. We include string matching as an extra baseline against the human labels to quantify this, but we do not expect it to compete.
Why not just resample at high temperature to find hallucinations? That changes the estimand rather than the measurement. Comparison requires valid measurement shows that temperature mainly changes output randomness and inflates best-of-K rates. It does not reveal how framing changes the model's behavior. We report best-of-K separately so readers can see the gap, but it does not address judge validity or the knowledge split.
Proposed Experiments
The test data are the Adversarial Factuality dataset at all three confidence tiers, used as in Battling Misinformation, and AdversaRiskQA, extended to the hesitant tiers it omits. The target models are the open models in the first study that exhibited the inversion or its opposite: LLaMA 3.1-8B, Phi 3-3.8B, Qwen 2.5-7B and Gemma2-9B. We use TruthfulQA items as a contrast set where errors are mostly imitative misconceptions, so knowledge gaps should dominate.
Three checks decide the proposal's value.
The first is whether Phi 3's increase from 52.63% to 93.78% survives judge correction and the separation of evasions. If the judge's false-positive rate is markedly higher on hesitant-tier responses, the inversion is at least partly a measurement artifact.
The second is whether, among known items, capitulation rises with confident framing while knowledge-gap errors stay flat across tiers. That pattern would confirm the two components are different phenomena that the single ASR number hides.
The third is whether the corrected protocol agrees with full human labels better than raw judging, filtered judging, or a judge ensemble, on the two fully labeled models.
A clean negative result on all three would also be informative: it would certify the existing benchmarks as valid. A positive result on any of them changes how adversarial factuality numbers should be reported.
References
- Battling Misinformation: An Empirical Study on Adversarial Factuality in Open-Source Large Language Models
- AdversaRiskQA: An Adversarial Factuality Benchmark for High-Risk Domains
- Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming
- LoFT: Local Proxy Fine-tuning Improves Transferability to Large Language Model Attacks
- Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization
- Position: Adversarial ML for LLMs Is Not Making Any Progress
- HalluLens: LLM Hallucination Benchmark
- Do Language Models Know When They're Hallucinating References?
- TruthfulQA: Measuring How Models Mimic Human Falsehoods