Language-model benchmarks can score a model's candidate answers in two ways. The model can judge each candidate on its own, or it can see all candidates side by side and pick one. On reasoning benchmarks, showing the options together raises scores a great deal. Llama 3.1 70B goes from 64% to 93% on ARC Challenge, a jump that makes the benchmark look solved (ARC 'Challenge' Is Not That Challenging). Hallucination research points the other way. Giving a model a correct answer and a fabricated one together makes it worse at spotting the fabrication (HaluEval). Side-by-side judging also pulls models toward confident, verbose answers and away from correct ones (The Comparative Trap). We argue that these results are consistent. Seeing the options lets a model discriminate: it can rank a true answer above false ones. Hallucination is a failure of assertion: the model states something false when no comparison set is there to catch it. When a factuality benchmark is scored jointly, it can climb toward its ceiling, a condition called saturation, while measuring very little about how often the model would hallucinate in normal use.
The proposal measures this gap item by item and model by model. We call the gap contrast dependence: how much of a model's apparent factual competence exists only when the right answer is put in front of it next to the wrong ones. The concrete test is predictive. Across a pool of open-weight models, we ask which scoring protocol for factuality multiple-choice items best predicts each model's hallucination rate in open-ended generation. If joint scoring saturates and predicts poorly, while independent truth verification keeps its spread and predicts well, then the field's cheap factuality leaderboards are measuring the wrong quantity, and there is a direct fix.
Core Claim
For factuality, the natural independent protocol is not cloze likelihood, which scores each answer string by its probability. That protocol suffers from surface-form competition: different phrasings of the same answer split the probability between them (Surface Form Competition). Nor is it multiple-choice prompting, where the model sees lettered options and emits a letter (Leveraging Large Language Models for Multiple Choice Question Answering). The right independent protocol is per-candidate truth verification with abstention. Each candidate is shown alone as a claim, and the model labels it true, false, or unsure. An item counts as correct only if the model accepts the true answer and accepts no falsehood.
We define contrast dependence for a model on a benchmark as
$$C = A_{\text{joint}} - A_{\text{verify}},$$
where $A_{\text{joint}}$ is accuracy when all options are shown together and $A_{\text{verify}}$ is item-level accuracy under independent truth verification. We make three predictions. First, $C$ is much larger on factuality benchmarks whose distractors are misconceptions or fabrications than on science reasoning benchmarks, where Borchmann found that much of the joint gain is legitimate because 31% of ARC items are "hardly answerable in separation." Second, joint-format scores on factuality benchmarks show high saturation while verification scores do not. Third, $A_{\text{verify}}$ predicts open-generation hallucination rates better than $A_{\text{joint}}$ does.
The closest prior work runs the format comparison on reasoning benchmarks (ARC 'Challenge' Is Not That Challenging), studies shortcuts that work without the question (Artifacts or Abduction), or fixes judge bias (The Comparative Trap). None of these asks whether a format's score predicts hallucination outside the benchmark. That validity question is what separates this proposal from a format-sensitivity replication.
Research Program
Five scoring protocols. We take existing factuality multiple-choice items and score each one under five protocols:
- Separation: cloze likelihood of each answer string, as in the original TruthfulQA multiple-choice scoring and Borchmann's Separation setting.
- Options: multiple-choice prompting with permuted option orders. We report plurality agreement across permutations, so that the positional bias and symbol-binding failures documented by Robinson and Wingate stay visible rather than hidden in the average.
- Choices-only: the options with the question removed, following Artifacts or Abduction. This measures how far a model can get from patterns among the options alone.
- Independent verification: each candidate shown alone as a claim, labeled true, false, or unsure.
- Missing-answer: the correct option is replaced by a fresh, plausible falsehood, and the prompt states that "none of these" is permitted.
The missing-answer protocol is the joint-format analogue of the HalluLens NonExistentRefusal task. A model that truly knows the answer should abstain. A model that is only discriminating will choose the least-bad option, which is a hallucination produced on demand.
Benchmarks. The items come from three sources:
- TruthfulQA's 817 questions, whose distractors are imitative falsehoods.
- HaluEval's QA split, derived from HotpotQA. We convert each correct/hallucinated pair into a two-option item.
- A dynamically generated PreciseWikiQA-style set built with the HalluLens procedure, so that contamination of the static sets can be ruled out.
ARC Challenge and OpenBookQA serve as reasoning controls, where we expect $C$ to be large but legitimate.
Open-generation reference. For the same models we measure open-ended hallucination with three tasks: HalluLens PreciseWikiQA (hallucination rate and false-refusal rate), NonExistentRefusal (false-acceptance rate), and FActScore biographies (FActScore). For biographies we decompose claims with the Russellian/neo-Davidsonian decomposition, which A Closer Look at Claim Decomposition found to score highest on its decomposition-quality measure. This reduces the risk that decomposer artifacts dominate the reference.
Model pool. The pool consists of open-weight families at several sizes: Llama 3.1 8B, 70B and 405B, Qwen 2.5 7B through 72B, and Mistral 7B through Large. These are the families used in the Borchmann and HalluLens studies.
Saturation measure. For each protocol we compute the saturation index of When AI Benchmarks Plateau using its released code (evaleval/benchmark-saturation). The index compares the spread of scores among the top five models with their standard errors, and it approaches 1 when the differences between models fall below measurement noise. The index was designed for real leaderboards. Applying it to our fixed pool is a controlled use, and we will say so when reporting.
Mechanism analysis. The analysis tests why joint presentation hurts. On the HaluEval pairs we vary the surface features of the hallucinated option independently of its truth: its length, its hedging versus confident tone, and its lexical overlap with the question. If our account is right, joint-format accuracy should track these features, and verification accuracy should not.
Baselines
Naive reference. The do-nothing baseline is the protocol the benchmarks already ship with: TruthfulQA's isolated likelihood scoring and HaluEval's single-response detection prompt. The original papers report the relevant numbers. On TruthfulQA, the best model reached 58% truthfulness against a 94% human baseline (TruthfulQA). On HaluEval QA detection, ChatGPT reached 62.59%, and open models scored near or below 50% (HaluEval). Everything we report is measured against these protocols, re-run on our pool.
Strongest existing scoring method. This is multiple-choice prompting as formalized by Robinson and Wingate, the format behind both the 9.7-point average gain across 24 datasets and Borchmann's 64% to 93% jump. It competes on exactly our question, because it is what a practitioner would adopt today to "fix" isolated scoring. We give it the same items, the same models and the same prompt-tuning budget as verification: an equal number of prompt templates tried on a held-out tuning split, and averaging over several option orders. I found no reported result on Lacuna for multiple-choice prompting on TruthfulQA or the HaluEval QA split, so we establish that number ourselves. Verification "beats" this baseline if two things hold. Its scores must correlate more strongly with open-generation hallucination across the model pool, which we test with a bootstrap over models and items. And its saturation index must be lower at comparable mean accuracy. Beating it on raw accuracy is not the goal, because we expect verification accuracy to be lower.
Strongest existing method for the contrast failure. This is PREPAIR, which has the model reason about each candidate on its own before making a comparative decision (The Comparative Trap). It lifted Llama-3.1-8B on LLMBar-Adversarial from 48.59% to 70.84%, a 24.48-point average gain across six models. PREPAIR is a hybrid that still ends in a forced choice. Comparing it to pure verification therefore isolates the ingredient we care about: whether abstention, and the absence of a forced comparison at the final step, matters for predicting hallucination. We run PREPAIR on the HaluEval pairs and on the missing-answer items with the same budget.
Ablations. Two ablations isolate the key ingredient. The first is verification without the "unsure" label, which tests whether abstention or mere isolation drives predictive validity. The second is joint scoring with "none of these" permitted, which tests whether the problem is forced choice rather than co-presentation of the options.
Why not just…?
Why not just subtract the choices-only score from the joint score? The choices-only test of Artifacts or Abduction measures shortcuts that need no question. Contrast dependence also includes shortcuts that need the question but work only relative to the other options, such as ruling out the option that sounds like a misconception. Choices-only subtraction cannot see the second kind. We include choices-only as a protocol, and the regression of open-generation hallucination on all protocols will show whether it accounts for the joint–verification gap. If it does, the simpler correction wins and we will say so.
Why not just abandon multiple choice and evaluate open generation, as HalluLens does? This is the real threat. The dynamic open-generation benchmark is closer to deployment, and it already avoids leakage of static test sets. Its costs are noise and expense. LongWiki's automated verification fails at retrieval roughly 15.4% of the time (HalluLens), and decomposition choices shift factuality scores substantially (A Closer Look at Claim Decomposition). Our experiments settle this by measuring split-half reliability across models for both open generation and verification. If verification ranks models as reliably as open generation at a fraction of the judging cost, it earns its place as a cheap proxy. If open generation is both more reliable and better aligned, then the contribution narrows to a diagnosis of why joint multiple-choice factuality scores mislead.
Why not just use PREPAIR-style pointwise-then-pairwise judging? PREPAIR fixes judge bias, but it keeps the forced choice. Under our account, the missing-answer condition is exactly where a forced choice fails. If PREPAIR matches verification's predictive validity and its rate of correct abstention on missing-answer items, then our abstention ingredient adds nothing. The ablation that drops the "unsure" label tests this directly.
Decisive Checks
Four results will decide whether the idea holds.
- Size of the gap. Contrast dependence must be substantially larger on TruthfulQA and HaluEval-QA than on ARC Challenge after subtracting choices-only accuracy. If the factuality and reasoning gaps are similar, the "relative versus absolute" distinction is wrong.
- Predictive validity. Across the model pool, the rank correlation between verification accuracy and the open-generation measures (PreciseWikiQA hallucination rate, NonExistentRefusal false acceptance, FActScore) must exceed that of joint accuracy, and the bootstrap interval for the difference must exclude zero.
- Missing-answer behavior. Models with high joint accuracy should still pick a falsehood on many missing-answer items. That pick rate should track their NonExistentRefusal false-acceptance rate.
- Inverse scaling. We test whether TruthfulQA's inverse scaling, the finding that larger models are less truthful, survives under each protocol. The original finding comes from isolated likelihood scoring. It may vanish under joint scoring and reappear under verification, which would show that the format determines the trend. Alternatively, it may vanish under every protocol for modern instruction-tuned models.
We will treat the last outcome as a finding, not a failure, and report it alongside the saturation indices under each protocol.
References
- ARC 'Challenge' Is Not That Challenging
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluators
- Surface Form Competition: Why the Highest Probability Answer Isn't Always Right
- Leveraging Large Language Models for Multiple Choice Question Answering
- Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- HalluLens: LLM Hallucination Benchmark
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- A Closer Look at Claim Decomposition
- When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation