ChainCheck: Near-Perfect Evidence-Sufficiency Scores Can Conceal Sensitivity to Construction Artifacts -- Paired Conditional Contrasts that Expose an Evaluation Confound

Hyojung Han, Ryangkyung Kang, Youngeun Kim, Seung-Hun JeonarXiv 2026 · arXiv

TL;DR

High scores on "evidence-sufficiency" benchmarks for RAG systems are often a mirage. This paper introduces ChainCheck, an audit protocol showing that models frequently achieve near-perfect scores by detecting "edit artifacts" (traces left by benchmark construction) rather than performing actual multi-hop reasoning.

What Problem Does This Paper Address?

In Retrieval-Augmented Generation (RAG), a model must decide if a set of retrieved passages is sufficient to answer a query. To test this, researchers build benchmarks with "negative" examples where the reasoning chain is intentionally broken (e.g., by swapping a "bridge entity" that connects two facts).

The problem is shortcut learning. Every method of breaking a reasoning chain leaves a trace. If you delete a passage, query relevance drops. If you swap an entity, the text might become slightly less natural or fluent. A model might reach a "ceiling" score simply by becoming a world-class "edit detector" rather than a logical reasoner. The authors argue that current evaluation metrics (like pair AUC) mix these two behaviors, creating an evaluation confound that hides whether a model actually understands the evidence chain.

Methods

The authors propose the ChainCheck protocol, which moves beyond simple "Positive vs. Negative" testing by introducing a third "control" cell to isolate the effect of the edit itself:

  • Cell A (Original): Chain is alive; no edits.
  • Cell B (Consistent Swap): Chain is alive, but the bridge entity is swapped consistently across all passages. This introduces an edit trace but keeps the logic intact.
  • Cell D (Substitution): Chain is broken; the bridge entity is swapped in only one part of the chain. This has both an edit trace and a broken chain.

By comparing these cells, they derive new metrics:

  • Chain Effect (CE): Sensitivity to the broken chain (B vs. D).
  • Edit Effect (EE): Sensitivity to the edit trace (A vs. B).
  • Chain Selectivity ($\Sigma$): The difference (CE - |EE|). A positive value suggests the model is more sensitive to logic than to artifacts.

They also introduce artifact-only baselines: scorers that cannot see the reasoning chain (e.g., a model that only looks at one passage at a time) to see if the benchmark can be "solved" without multi-hop reasoning.

The ChainCheck design uses three cells to separate reasoning from edit detection.

Datasets

The study evaluates systems across three major multi-hop QA datasets:

  1. MuSiQue: Used for initial exploration and a "sealed" confirmation.
  2. 2WikiMultiHopQA: Used for external replication.
  3. HotpotQA: Used for an independent confirmation set with document-disjoint splits to prevent leakage.

Key Insights and Ideas

  • The "Nominal Ceiling" is Deceptive: Models often sit near 0.99 AUC, leaving no room to distinguish between them. ChainCheck reveals that underneath these identical high scores, models have wildly different levels of "edit dominance."
  • Baselines can "Cheat" Perfectly: An artifact-only language model that never sees the full chain reached a nominal AUC of 0.997 on HotpotQA. Even a simple string-match baseline (checking if a bridge string matches a title) reached the theoretical maximum on some sensitivity statistics.
  • Operational vs. Semantic Reasoning: The authors distinguish between "bridge-string consistency" (detecting if names match) and "semantic multi-hop reasoning." Most models are doing the former, not the latter.
  • Fragile Generalization: Patterns found on one dataset (like MuSiQue) often failed to replicate on others (like HotpotQA). This suggests that different benchmarks have different "tells" that models exploit.
  • Batching Artifacts: In a surprising find, the authors noted that even the way records are packed into batches for GPU processing can change a model's score enough to fail a reproducibility gate.

Key Results

The paper's most striking result is the comparison of evaluated systems against "brainless" baselines. As shown in Figure 1, systems that are supposedly doing complex reasoning are often outperformed by baselines that don't even read the chain.

Figure 1: Artifact-only baselines against the evaluated systems on the crossed HotpotQA arm E (2,214 pairs, exploratory). (a) Nominal pair AUC of the original contrast (A over D) for the 11 systems; the dashed line is AB2, a per-passage language-model log-probability scorer that never sees the chain (0.997). (b) CS4, the crossed chain-sensitivity statistic of Section 11.2, with Bonferroni-adjusted document-group bootstrap intervals; the dashed line is AB3, which checks whether one bridge string equals one title and reaches the maximum 0.500. The panels use different metrics on separate axes and are not compared with each other. Values are in Table 6.
  • Edit Dominance: On MuSiQue, every trained system was "edit-dominated," meaning they were more sensitive to the fact that text was edited than to whether the reasoning chain was broken.
  • Selectivity: The only system that was consistently "chain-selective" (more sensitive to logic than edits) was the zero-shot "judge" model, though it often had lower overall nominal scores than the trained probes.

Limitations and Future Work

  • Scope: The study primarily used one backbone (Qwen3.8-27B) and focused on English-language datasets.
  • The "Missing Cell": It is currently impossible to construct a "broken chain without an edit" (Cell C) because any intervention to break a chain is an edit. This means we can only measure chain effects in the presence of edits.
  • Reproduction: Because some fine-tuned checkpoints and specific HotpotQA records remain private, external researchers can only partially reproduce the results.
  • Future Direction: The authors call for benchmark builders to ship an "edit-only" control cell (like Cell B) as standard practice to ensure that "near-perfect" scores actually mean something.

Summary updated

Abstract

Evidence-sufficiency benchmarks ask whether a model can tell a complete passage set from one whose reasoning chain is broken. When broken sets are made by editing text, near-perfect sufficiency scores--and even positive crossed-control scores--need not demonstrate multi-hop reasoning. We present ChainCheck, an evaluation-validity audit that pairs validator-certified counterfactual cells with artifact-only baselines. In an exploratory crossed control on 2,214 HotpotQA pairs, a language-model scorer that never reads the chain reaches a nominal AUC of 0.997, and a one-string match reaches the maximum of the crossed chain-sensitivity statistic CS4 (0.500), while the 11 evaluated systems score from 0.308 to 0.484; the two statistics are different metrics. We therefore read positive CS4 as operational bridge-consistency sensitivity, not as semantic multi-hop reasoning. Every system is also sensitive to one extra edit, and the edits are detectable according to model-based naturalness ratings. On an independent HotpotQA set of 2,276 pairs, patterns pre-specified on MuSiQue and 2WikiMultiHopQA did not replicate: 5 of 6 trained readouts were edit-dominated, and the reference judge's selectivity interval [-0.007, +0.042] includes zero. Judge scores also depended on batch layout. The MuSiQue and 2Wiki records are public; the HotpotQA records and the fine-tuned checkpoints are not, so external reproduction is partial.
BibTeX
@misc{han2026chaincheck,
  author = {Hyojung Han and Ryangkyung Kang and Youngeun Kim and Seung-Hun Jeon},
  title = {{ChainCheck: Near-Perfect Evidence-Sufficiency Scores Can Conceal Sensitivity to Construction Artifacts -- Paired Conditional Contrasts that Expose an Evaluation Confound}},
  year = {2026},
  eprint = {2611.11965},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11965},
}
Figure