When a question needs several lookups chained together ("where was the director of Inception born?"), retrieval-augmented language models search, read, and search again. When one link in that chain does not exist, for example because the premise is false or the corpus simply lacks the fact, these systems rarely stop. They keep searching and then produce a confident answer anyway. Recent diagnostics suggest this is not because models fail to notice the broken link. When search agents do refuse, they name the correct reason 84.7–98.4% of the time, yet the best of them halts correctly on only 42.9% of defective chains (HopRefusalBench). The failure is one of commitment. The model that is generating the answer is also the one deciding whether to stop, and the pressure to finish the answer overrides what it already knows.
We propose to take that decision away from the generator. The system keeps a shared, typed record, a "ledger," with one entry per hop. Each entry must bind an intermediate entity to a specific retrieved passage. A small, separate checker labels every entry as bound, contradicted, absent, or ambiguous. The final answer writer may read only bound entries, and it is never called unless every hop in the chain is bound. Abstention then stops being a choice the language model makes at the end. It becomes a structural property of the pipeline, and it comes with a pointer to the hop that failed.
Core Claim
Multi-hop abstention failures are concentrated where a valid prefix of the chain builds momentum, which is why Root and Middle defects are hardest in HopRefusalBench. They are also concentrated where partial evidence is present, which is why models detect unanswerability worse with partial context than with none in CRUMQs. Our claim is that a per-hop check outside the generator, combined with hiding unverified passages from the answer writer, recovers most of the gap between "the model recognizes the flaw" and "the system halts." It should do so without materially hurting accuracy on answerable chains, and it should halt earlier and spend fewer search calls than agentic baselines.
The new ingredient is not "add a verifier." It has three parts. Verification happens per hop, before the next search is issued. The halting decision is made by a non-generative module over a typed record. Ambiguity is carried forward rather than resolved early, so the system refuses only when every candidate path has failed.
How This Relates to Prior Work
Three benchmarks motivate the proposal by locating the gap. HopRefusalBench places defects at Root, Middle, or Terminal positions of 2–5 step chains and finds that enabling search lowers correct halting for 9 of 10 models. Its authors call for "coordinated premise verification" but do not build it, and building it is our goal. CRUMQs generates unanswerable multi-hop queries over NeuCLIR and TREC RAG 2025, reports a 28% acceptable-response ratio for GPT-5, and shows refusal declining linearly with hop count. MultiHoax shows that models often say "I don't know" without being able to name the false premise: Claude 3.5 Sonnet scores 46% on detection and 23% on justification. All three are purely diagnostic. We use them as held-out test beds.
Two results constrain the design. AbstentionBench finds that reasoning-tuned models abstain 24% worse than instruction-tuned ones and that extra thinking tokens reduce abstention. More generator deliberation is therefore the wrong lever, which argues for a separate, non-generative check. Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL shows that abstention detectors exploit surface artifacts of unanswerable templates. For that reason our checker never sees the original question, only a single sub-question, a candidate binding, and a passage.
Two modular multi-hop systems are the closest prior work. ComposeRAG splits multi-hop retrieval into decomposition, retrieval gating, reranking, answer generation, and a verification module that triggers self-reflection when a sub-answer is unsupported. It differs from our design in three ways. Its plan is fixed up front, so a bad decomposition cannot be recovered. Its verifier responds to a failure by searching for an alternative path rather than by halting, which is exactly the behavior HopRefusalBench links to forced answers. It was never evaluated on abstention benchmarks. Think Straight, Stop Smart removes the stop decision from the language model, but its trigger is cosine similarity between consecutive sub-queries, a loop detector. After stopping it still synthesizes an answer from whatever was retrieved. We keep its insight that termination should be taken out of the generator, but we condition termination on evidence rather than on repetition. IRCoT supplies the incremental retrieve-reason loop we build on. It has no sufficiency check and accumulates every retrieved passage into context.
From classical modular NLP we borrow two mechanisms. The first is the typed shared annotation object of Stanford CoreNLP, in which downstream modules read only keys that upstream modules have filled. The ledger is that object, and the answer writer reads only bound keys. The second is the practice in MACAON of passing weighted alternative hypotheses forward instead of committing early. Ambiguous hops keep up to $k$ candidate bindings, which protects against over-refusal on underspecified-but-answerable chains.
Research Program
The system has four modules that communicate through the ledger.
The planner is the backbone language model. It proposes the next sub-question using only the bound entries so far, so decomposition is incremental, as in IRCoT, rather than fixed up front as in ComposeRAG.
The retriever fetches passages for that sub-question.
The checker takes each (sub-question, candidate binding, passage) triple and assigns one of four states. The binding is bound if the passage supports it. It is contradicted if the passage refutes a presupposition of the sub-question, which signals a false premise. It is absent if no retrieved passage supports any candidate after a small retry budget, which signals an unknown answer. It is ambiguous if several candidates are each supported, which signals underspecified context. These states map directly onto HopRefusalBench's three failure causes (Answer Unknown, False Premise, Underspecified Context).
The answer writer runs only when the final hop is bound. It sees only the ledger's bound passages. Otherwise the system abstains and reports the failing hop and its state.
We will train the checker on single-hop supervision derived from MuSiQue. MuSiQue questions are composed from gold single-hop questions, and MuSiQue-Full provides contexts in which a link has been severed (MuSiQue). These give us bound and absent labels. Contradicted labels come from perturbing gold bindings, and ambiguous labels from sampling distractor entities that share a type with the gold entity. We will compare a fine-tuned small entailment-style classifier against a prompted mid-size model used as the checker. Neither variant is trained on any HopRefusalBench, CRUMQs, or MultiHoax items. The entire backbone stays frozen, and no reinforcement learning is involved.
Baselines
The naive reference point is a standard dense-retrieval pipeline with a fixed number of hops, where the generator is told it may answer "I don't know." It shows how much the task is solved by permission alone.
The strongest existing agentic reference is the search-augmented setting of HopRefusalBench, whose best reported Target-Aware Correct Halting Rate (TCHR) is 42.9%, for Qwen3.5-397B under an 8-search-call cap over the 2018 KILT Wikipedia snapshot. We will report our system under that same cap and corpus. To keep the comparison fair, we will also re-run an agentic loop of the same style with our own backbone, so that model size is not confounded with architecture. Beating this baseline means exceeding 42.9% TCHR with a backbone no larger than the one reported, and also beating our same-backbone agent by a margin outside bootstrap confidence intervals.
The strongest modular competitor is ComposeRAG, reimplemented with the same backbone, retriever, and search budget. The file contains no per-dataset ComposeRAG number on any benchmark, and it was never run on abstention benchmarks, so we will report our own reproduction against its aggregate claim of up to 5% over Search-o1.
IRCoT serves as the no-verification iterative baseline. On HotpotQA, TSSS reports 28.0 exact match for IRCoT and 34.1 for itself. TSSS is also our efficiency baseline, run with the same backbone and evaluated on both halting and latency.
Three ablations isolate the key ingredient. The first is a post-hoc judge: the same checker model reads the full IRCoT trace and decides whether to abstain, which keeps verification but drops per-hop gating. The second removes context isolation: the answer writer sees all retrieved passages, not only bound ones. The third removes carry-forward: ambiguous hops trigger immediate refusal.
All systems share the backbone, retriever, corpus, search cap, and a matched prompt-tuning budget of equal development-set queries drawn from MuSiQue only.
Why not just…?
Why not just prompt the agent to check each premise and allow refusal? HopRefusalBench already evaluates frontier agents that are permitted to refuse, and their failures are commitment failures, not ignorance. AbstentionBench shows that more generator reasoning lowers abstention. Prompting still asks the generator to stop itself. We include this as the naive baseline and expect it to improve Terminal defects but not Root or Middle ones.
Why not just run a separate verifier on the finished trace? This is the real threat. AbstentionBench reports that reasoning traces often mention missing information before the model answers anyway, so a post-hoc reader might catch most defects. We test it directly with the post-hoc judge ablation, which uses the identical checker model. Our per-hop design should win on two fronts. The first is Middle failures, where the trace contains a hallucinated bridge that reads as fluent support, the partial-context regime CRUMQs identifies as hardest. The second is search calls and tokens, because post-hoc judging cannot stop wasted search (HopRefusalBench reports over 10,000 wasted tokens per evidence-exposed trajectory for one model). If the post-hoc judge matches us on halting accuracy, we will report that the simpler design suffices for accuracy, and our remaining contribution reduces to cost.
Why not just fine-tune or reward the model for abstaining? Reward schemes that score only correct answers penalize abstention, and repairing that requires carefully calibrated abstention rewards plus labeled unanswerable chains at scale. A rewarded model also still makes the stop decision inside the generator and offers no hop-level justification. We do not run a reinforcement-learning baseline, which is a stated limitation. Our system leaves the backbone frozen, so the two approaches are complementary rather than competing.
Why not just threshold the generator's answer confidence? A single end-of-chain confidence score cannot tell which hop failed, so it cannot produce MultiHoax-style justifications. Scale does not fix overconfidence on unanswerable inputs (AbstentionBench). We include a confidence-threshold variant swept to match our abstention rate on answerable data.
Proposed Experiments
The first test, and the most decisive, is HopRefusalBench broken down by topology and cause. We predict the largest gains on Middle and Root defects and on Underspecified Context, the categories where current agents are weakest. We also predict that enabling search will no longer lower halting for our system, reversing the 9-of-10 degradation. In addition, we will score whether the ledger's reported failing hop matches the gold defect position.
The second test is CRUMQs, scored by acceptable-response ratio (GPT-5 reference: 28%). We will split results into partial-context and zero-context queries and into hop counts 2 through 7. Our system should flatten the linear decline with hop count and close the partial-versus-zero gap. A post-hoc judge, which sees partial support, should not do either.
The third test is MultiHoax Task 1 and Task 2 (Claude 3.5 Sonnet reference: 46% and 23%), run with retrieval. Task 2 explanations are generated from the contradicted ledger entry. We expect the gap between detection and justification to narrow, because each refusal is tied to a specific passage.
The fourth test guards against over-refusal. On MuSiQue-Full we report pair accuracy on answerable/unanswerable contrast pairs, which penalizes a system that simply refuses everything. On MuSiQue-Ans and HotpotQA (TSSS reference: 34.1 exact match), we report answer F1 and exact match. The no-carry-forward ablation should show that carrying ambiguity forward is what protects answerable accuracy.
Across all benchmarks we report search calls, tokens, and latency per query. The proposal fails if the post-hoc judge matches per-hop gating on both halting and cost, or if per-hop gating costs more than a few points of answerable F1 relative to IRCoT. Either outcome would itself be informative, because it would show that the commitment gap can be closed with an end-of-chain reader alone.
References
- HopRefusalBench: Diagnosing Refusal Failures in Search-Augmented Agents for Multi-Hop Reasoning
- Evaluating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries
- MultiHoax: A Dataset of Multi-hop False-premise questions
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
- Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL
- ComposeRAG: A Modular and Composable RAG for Corpus-Grounded Multi-Hop Question Answering
- Think Straight, Stop Smart: Structured Reasoning for Efficient Multi-Hop RAG
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- The Stanford CoreNLP Natural Language Processing Toolkit
- MACAON An NLP Tool Suite for Processing Word Lattices
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition