Severed or Absent: Telling Chunking Damage from True Unanswerability in Multi-Hop Retrieval

Retrieval-augmented language models answer questions by searching a document collection, but they almost never search whole documents. Documents are first cut into fixed-size pieces ("chunks"), and the model only sees the pieces it retrieves. Questions that need several linked facts make this worse. Take "In what city was the director of Inception born?" Here the model must find the director, then the birthplace. If the cut falls between "Christopher Nolan" and "He was born in London," the second fact exists in the collection but is unreadable in any single retrieved piece. The model then faces a choice it cannot make well. It can refuse ("the corpus doesn't say"), which is a false refusal. Or it can guess the missing link, which is a hallucination. We argue that current abstention benchmarks conflate these cases, and that some of the well-documented reluctance of search agents to refuse may be a sensible response to this ambiguity, not just a training defect.

We propose to build matched test sets that separate severed bridges, where the evidence is present but split by a chunk boundary, from absent bridges, where the evidence is truly missing. We then test a cheap, targeted fix. Whenever the agent is about to refuse, or a retrieved piece of evidence ends in an unresolved reference, it fetches the neighboring chunks from the same source document and re-checks the link. Only a link that is still missing after this "contiguity audit" yields a refusal, and the agent receives an explicit certificate of that fact. The same audit extends to cross-lingual collections, where a chunk translated sentence by sentence loses the antecedent of its pronouns. Here the repair is to re-translate the chunk together with its neighbors.

Core Claim

Abstention errors in chunked multi-hop retrieval come in two kinds that need opposite remedies. Boundary-induced pseudo-unanswerability causes false refusals. Momentum from a valid partial chain causes false answers. A refusal-time contiguity audit that checks adjacent chunks before committing should do two things at once. It should cut false refusals on severed-but-answerable questions. It should also raise correct refusals on truly unanswerable ones, because the agent now has a verified reason to stop. It should do both at a fraction of the cost of uniform chunk overlap. The second half of this claim is the risky and interesting one.

This builds directly on two findings. HopRefusalBench evaluates search agents on 889 multi-hop questions with deliberately broken links. It shows that enabling search lowers correct halting for 9 of 10 models, with the best search-augmented model at 42.9% Target-Aware Correct Halting Rate (TCHR). When models do refuse, they name the right reason 84.7–98.4% of the time. The authors read this as a commitment problem. We add a hypothesis: commitment is hard partly because, over chunked text, "I didn't find the link" is weak evidence that the link is absent. CRUMQs builds hard-to-shortcut multi-hop queries that are fully or partially unanswerable, over NeuCLIR and TREC RAG 2025. It finds GPT-5 gives an acceptable response only 28% of the time, with partial information hardest. Both benchmarks treat chunks as fixed and unanswerability as a property of the query–corpus pair. Neither asks whether segmentation itself manufactures unanswerability. That question is what we add.

Research Program

The first component is a diagnostic benchmark, which we call SeveredHop. We start from answerable chains in MuSiQue and 2WikiMultiHopQA, whose multi-hop structure resists single-hop shortcuts. We re-chunk each chain's source documents with fixed-token windows, and place the offsets so that the bridge fact's entity mention and its predicate land in different chunks. Splitting a pronoun from its antecedent is the canonical case. For each question this yields an intact version and a severed version that differ only in boundary placement. We pair these with absent versions, drawn from HopRefusalBench's Root, Middle and Terminal defects and from CRUMQs' partially unanswerable queries. We also apply the same severing to the answerable portions of CRUMQs so the three conditions share a corpus.

The main measurement is the pseudo-unanswerability gap: the false refusal rate on severed questions minus the false refusal rate on intact ones. If this gap is near zero for strong agents, the premise of the proposal fails and we report that.

The second component is the contiguity audit. It is triggered in two situations: when the agent emits a refusal, or when a cited evidence span begins or ends in an unresolved reference. Unresolved references include a pronoun, a definite description, or a truncated named entity. The second trigger is detected with a coreference model plus a simple boundary-truncation check. Once triggered, the audit fetches the preceding and following chunks from the same source document. It asks the model to re-state the bridge with a word-for-word quote, borrowing the Chain-of-Quote format from Making Long-Context Language Models Better Multi-Hop Reasoners, where requiring quotes for each step made a 7B model robust to distractors. Unlike AttrLoRA, which assumes every question is answerable and that paragraphs are pre-segmented, we use quoting as a verification gate. If no quote supports the bridge after the audit, the agent receives a structured "bridge absent after neighbor check" message, and we measure whether this certificate makes it commit to refusing. The window is deliberately small, one chunk on each side. This choice is motivated by the finding in Multi-Hop Transformer for Document-Level Machine Translation that the benefit of surrounding context peaks around three preceding sentences and then turns into noise.

The third component is a cross-lingual arm, which treats translation as a stress condition rather than as the core method. On CRUMQs over NeuCLIR's non-English collections, retrieved chunks are translated for the reasoning model. We compare three ways of doing this: sentence-isolated translation, chunk-level translation, and audit-time re-translation of the chunk together with its neighbors. The concatenation approach to context-aware translation, refined in the Flat-Transformer of A Simple and Effective Unified Encoder for Document-Level Machine Translation, shows that letting attention see neighboring sentences fixes cross-sentence pronouns. We test whether the same repair, applied only at audit time, fixes severed bridges that are invisible in translation. We keep the question and instructions in their original form rather than pre-translating them. This follows the warnings in Cross-lingual Extractive Question Answering with Unanswerable Questions, where translate-first prompting hurt abstention, and in How and Where to Translate?, where forcing low-resource-language prompts cost over 30%.

Baselines

The naive reference point is an interleaved retrieve-and-reason agent in the style of IRCoT. IRCoT uses each intermediate reasoning sentence as the next search query and reported 11–21 point recall gains on HotpotQA and 2WikiMultiHopQA. We run it over fixed, non-overlapping chunks with a plain "say unanswerable if the evidence is missing" instruction. That instruction alone raised GPT-4o-mini's no-answer F1 from 4.97 to 60.23 on miXQuAD in the cross-lingual extractive QA paper. Every other system is compared against this agent.

The strongest existing reference on refusal is the best search-augmented agent in HopRefusalBench, Qwen3.5-397B at 42.9% TCHR, under its 8-call search budget. We will rerun an open model from that evaluation under the same 8-call budget, both on HopRefusalBench as released and on our severed and absent splits. The SeveredHop splits are new, so no reported result exists on Lacuna for them. On answer accuracy, the strongest reported reference is AttrLoRA at 38.1 Exact Match (EM) on MuSiQue, against 37.0 for ChatGPT. We will run a quote-required version of the agent with no audit to separate the effect of quoting from the effect of neighbor fetching.

The strongest segmentation competitor is uniform overlap. We use 50% overlapping windows, the setting InstructCoder found to balance boundary preservation against cost. We also use persistent document headers prepended to every chunk, following Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations, where a 128-word header let a 1024-token model reach 0.3331 ROUGE-1 F1 and beat a 4096-token long-context model.

All systems share one backbone model, one retriever, one index built from the same documents, the same search-call budget, and the same prompt-tuning budget, with prompts tuned on a held-out dev split. We report total tokens processed per question so that overlap's roughly doubled index and context are visible.

Three ablations isolate the key ingredient. In the first, the audit fetches an equal number of random chunks from the same document instead of adjacent ones, which controls for "more context." In the second, the audit fetches neighbors but issues no absence certificate, which isolates the commitment effect. In the third, the audit fires on every hop rather than only at refusal time, which tests the cost claim. An oracle that sees unchunked source documents gives the upper bound.

The audit counts as beating the baselines only if three conditions hold. It must cut the severed-question false refusal rate by at least half relative to the naive agent. It must not lower answer EM on intact questions. It must raise TCHR on absent questions by a margin that clears seed-to-seed variance. And it must match uniform overlap on the first two criteria while using fewer tokens.

Why not just…?

Why not just use 50% overlap or larger chunks? This is the most serious threat. Overlap may simply remove most severed bridges, and if it matches the audit on false refusals at similar cost, the segmentation half of our story collapses into "use overlap." We think it will fall short on the other side of the ledger. Overlap doubles the indexed text and fills the context with near-duplicate distractors on every hop, not just disputed ones. HopRefusalBench shows that more exposed evidence lowers halting and wastes tokens: Kimi-K2.5 averaged over 10,000 wasted tokens per evidence-exposed trajectory. Overlap also gives no explicit signal that a link was checked and found missing. The compute-matched comparison on absent questions settles this. If overlap's TCHR equals the audit's, we drop the commitment claim.

Why not just put whole documents in a long context window? This works only when the candidate documents are known in advance, which is not the case in open-collection search over NeuCLIR or TREC RAG. Long contexts also bring their own failure. AttrLoRA's paper documents steep drops for long-context models as distractors increase, and the clinical summarization result shows chunking with a header beating a native long-context model. We include unchunked documents as the oracle. The audit's job is to approach that oracle at chunked cost, not to beat it.

Why not just train or prompt the model to abstain better? Stronger abstention prompting or fine-tuning moves the single answer-or-refuse threshold. Severed and absent questions look identical from inside a chunk, so a better threshold cannot fix both. Pushing toward refusal raises false refusals on severed questions, and pushing toward answering raises hallucination on absent ones. The cross-lingual QA paper shows exactly this trade-off: mDeBERTa refused accurately but failed answerable extraction, while large models answered well but hallucinated without hints. AbstentionBench further finds that reasoning-trained models abstain 24% worse, so extra reasoning is not a substitute either. The paired severed and absent splits test this directly. A threshold-only method should trace a trade-off curve, and the audit should sit above it.

Why not just translate the whole collection to English first? Translating once with document-level MT would avoid split antecedents, but it costs a full pass over the collection. The cross-lingual QA paper also found translate-first pipelines carry errors into abstention decisions. Our cross-lingual arm includes full-document pre-translation as a comparison system. If it matches audit-time re-translation on both false refusals and correct halting, the translation component reduces to a preprocessing choice, and we will say so.

Decisive Checks

The proposal stands or falls on three results. First, a non-trivial pseudo-unanswerability gap must exist for strong agents on SeveredHop. Without it, there is nothing to repair. Second, the audit must lower false refusals on severed questions while also raising TCHR on absent ones, and the random-neighbor and no-certificate ablations must show this comes from adjacency and the explicit certificate rather than added text. Third, at matched token budgets, the audit must equal or beat 50% overlap on both splits. A secondary check uses the HopRefusalBench breakdown by defect position: if the commitment story is right, the largest TCHR gains should appear on Middle defects, where a valid prefix creates the most momentum. In the cross-lingual arm, the analogous check is whether neighbor-aware re-translation closes the gap between translated and source-language severed questions.