Abstaining at the Broken Link: Hop-Level Selective Prediction for Multi-Hop Search Agents

Search agents answer questions like "In what city was the director of Inception born?" by chaining lookups: first find the director, then find where he was born. When one link in that chain cannot be supported, the right behavior is to stop and say so. Current agents mostly do not. Instead they keep searching and then produce a fluent guess. We propose to move the abstention decision from the end of the chain to each individual hop. At every step, the agent's sub-answer receives a calibrated probability that it is actually supported by retrieved evidence. The agent abstains as soon as the running product of these probabilities falls below a threshold set to meet a target error rate. When it abstains, it also reports which hop failed.

The opportunity is concrete. HopRefusalBench runs ten frontier models on 889 unanswerable multi-hop questions. The best correct-halting rate with search enabled is 42.9%, and enabling search lowers halting for nine of the ten models. Failures cluster where the defect sits in the middle of the chain: a valid prefix builds momentum that carries the agent past a broken bridge. A single confidence score computed at the final answer is badly placed for this failure. By the time the answer is written, the doubt from hop two has been overwritten by the fluency of hop three. Our bet is that the uncertainty signal exists locally, at the hop where the link breaks, and that it is cheap to read there.

Core Claim

Selective prediction is the practice of letting a model decline to answer when it is likely to be wrong, trading coverage (the fraction of questions answered) against risk (the error rate on answered questions). Our claim is that, for multi-hop search agents, a per-hop selection rule beats final-output selection on the risk–coverage trade-off. The rule combines retriever confidence for the sub-question, whether the agent actually read a document, and whether the read text supports the sub-answer, each calibrated per hop. We expect the advantage to come specifically from Root and Middle defects. We also claim the rule saves search and token budget, because it stops early instead of exhausting the budget.

The idea is deliberately simple at the aggregation step. Let $s_t \in [0,1]$ be the calibrated probability that hop $t$'s sub-answer is supported. The agent abstains at the first hop $t$ where

$$\prod_{j \le t} s_j < \tau,$$

with $\tau$ chosen on held-out data so that selective risk stays below a target $\alpha$. The product form follows the weakest-link logic that Performance Prediction for Multi-hop Questions uses for bridge questions, where it multiplies pre-retrieval entity probabilities along the path. We do not claim a learned aggregator is needed. If a plain product or minimum of hop scores captures most of the gain, the contribution is the localization of the signal, and we will report it that way.

How This Differs From the Closest Work

Four papers come close, and each stops one step short.

HopRefusalBench motivates the work. It codes unanswerability along three axes: cause (answer unknown, false premise, underspecified context), defect position (root, middle, terminal), and chain depth. It documents the middle-defect blind spot, but it is purely diagnostic and calls for, without building, "coordinated premise verification."

Calibration of Machine Reading Systems at Scale supplies the method. It shows that calibrating the reader alone is catastrophically overconfident, and that jointly calibrating retriever and reader confidence brings expected calibration error from 55.1% to 3.56% on NQ and FEVER. That work covers a single retrieve-then-read step. We repeat its joint calibration at every hop of an interleaved loop.

MASH trains abstention as penalized search with reinforcement learning. It can only abstain by withholding the search tool entirely, so it cannot stop partway through a search when a link breaks. It also costs about 1,800 H100 hours.

Before Reasoning Can Fail constrains our design and supplies one feature. It splits wrong answers into four categories: never reading a document, reasoning errors after reading the gold passage, retrieval misses, and ambiguous cases. Its READ-GATE blocks answers given with zero reads, which recovers 14.9 to 19.9 points on trajectories that skipped reading. The authors note that READ-GATE has no way to abstain when the evidence it forces the agent to read is useless. Our selector supplies exactly that missing exit.

The multi-dimensional coding direction plays two roles here. It is the evaluation lens: every trajectory is coded by cause, position, and procedural compliance, so gains can be attributed. It also provides the feature design: each hop feature corresponds to one coded axis. It is not the main mechanism.

Research Program

Agent. The base agent is an interleaved search-and-read loop in the style of IRCoT. IRCoT alternates one reasoning sentence with one retrieval call and improves recall by 11–21 points over one-shot retrieval. We modify it to emit an explicit sub-question and sub-answer at each hop. The agent has separate search and read tools, following the design in Before Reasoning Can Fail, and a budget of eight search calls to match HopRefusalBench. We run it with two open-weight models: one mid-sized model from the Qwen family evaluated in HopRefusalBench, and one smaller model.

Hop features. For each hop we compute four signals:

  • the retriever's score margin for the sub-question;
  • a read-discipline indicator (did the agent read anything at this hop);
  • an entailment score from a small natural-language-inference model, checking whether the read passage supports the sub-question–sub-answer pair;
  • agreement of the sub-answer across a handful of resamples.

These are mapped to $s_t$ in two ways, so the value of learning a calibrator is measured rather than assumed: temperature scaling, and a gradient-boosted forecaster as in the machine-reading calibration paper.

Training labels. Supervision comes almost for free from MuSiQue-Full (MuSiQue). Its unanswerable questions are contrast pairs built by deleting one specific supporting fact, and the question is composed from known single-hop sub-questions. That tells us which hop is broken. Because answerable and unanswerable versions share surface form, the selector cannot pass by spotting question style. Thresholds are set by the error-controlled procedure in Act or Defer, which chooses decision thresholds under an explicit error cap.

Baselines

Unmodified agent. The naive reference is the agent answering everything, or abstaining only when it chooses to on its own. This anchors coverage at roughly 100% and shows the raw halting behavior that HopRefusalBench measured.

Final-output selection. This is the strongest cheap competitor. It thresholds the final answer's token entropy, as EHRSQL does with percentile-based thresholds that reach answerability F1 above 90% on single-turn text-to-SQL, together with agreement across the same number of full-trajectory resamples. It sees every hop's evidence in context, so it is a fair test of whether localization matters. It gets the same models, the same sampling budget in total tokens, and thresholds tuned on the same held-out split.

Final-step joint calibration. This applies the machine-reading calibration paper's retriever-plus-reader forecaster once, using last-hop retrieval scores and the final answer. It isolates whether the gain comes from per-hop placement or merely from adding retriever confidence.

READ-GATE plus final-output selection. This tests whether forcing reads and thresholding at the end already recovers our gains.

Prompted per-hop verification. The same agent is instructed to check each link and allowed to abstain. This tests whether a calibrated external score is needed at all.

MASH. We do not have its compute budget, so we compare against its reported number: 59.9% precision on answered HotpotQA questions, versus 53.1% for honesty-trained preference optimization. To make this fair, we match our abstention rate to MASH's on HotpotQA. No reported result was found on Lacuna for MASH on HopRefusalBench or MuSiQue-Full.

Bar on HopRefusalBench. The strongest reported correct-halting rate is 42.9% (Qwen3.5-397B). Beating it with a smaller model, without lowering accuracy on answerable HotpotQA and 2WikiMultiHopQA questions by more than a pre-registered margin, counts as a clear win. No reported risk–coverage result was found on Lacuna for unanswerable detection on MuSiQue-Full. There we must beat our own run of the final-output baseline.

Ablations. We remove each hop feature in turn and replace the calibrated $s_t$ with raw scores. We also compare product, minimum, and learned aggregation, and compare stopping early against computing hop scores but deciding only at the end. That last comparison separates better information from earlier action.

Why not just…?

Why not just threshold confidence on the final answer? This is the most serious threat. The machine-reading calibration paper shows reader-only confidence is badly overconfident in retrieval pipelines. HopRefusalBench shows middle defects yield fluent final answers. But that evidence comes from single-step reading and from models that were not explicitly thresholded. A self-consistency threshold over full trajectories might still separate answerable from unanswerable questions well. Our position-stratified risk–coverage comparison settles it. If final-output selection matches us on Middle defects, the core claim is false.

Why not just give the model more thinking tokens or tell it to verify each hop? Before Reasoning Can Fail found that larger hidden-thinking budgets increased zero-read failures. HopRefusalBench found that models identify the correct refusal reason in 84.7–98.4% of the cases where they do refuse, yet they rarely choose to refuse. The models can recognize the problem but do not act on it. The prompted-verification baseline measures this directly on our data.

Why not just learn abstention end-to-end with reinforcement learning, as MASH does? It is expensive and binary. It can abstain only when the search tool is removed, so it cannot stop at a broken middle link. Its abstention boundaries also depend on retriever quality. If an end-to-end policy turns out better, the HotpotQA comparison at matched abstention will show it. Our method still costs orders of magnitude less to train.

Why not just use the minimum of raw hop scores with no calibration? This would be a good outcome, not a threat to the main claim. We run it as an ablation and will report it as the recommended method if it matches the calibrated version.

Proposed Experiments

Training and transfer. We train calibrators on the MuSiQue-Full training split. We evaluate in-distribution on the MuSiQue-Full development set. We then transfer, without retuning, to HopRefusalBench, which uses a 2018 KILT-seeded Wikipedia snapshot and a different construction process. Answerable-question accuracy and false abstention are checked on HotpotQA full-wiki and 2WikiMultiHopQA.

Metrics. We report:

  • area under the risk–coverage curve;
  • selective risk at 80% coverage;
  • HopRefusalBench target-aware correct halting rate, broken down by cause, position, and depth;
  • hop-localization accuracy against the deleted fact in MuSiQue-Full;
  • search calls and tokens per trajectory.

Every error and every abstention is also coded along the Before Reasoning Can Fail categories, so we can see which failure type each method removes.

Decisive checks. First, the gap between hop-level and final-output selection should be largest for Middle defects, smaller for Root, and near zero for Terminal. A flat gap across positions would mean the gain is not about localization. Second, stopping early should cut wasted search budget relative to the baselines. HopRefusalBench reports over 10,000 wasted tokens per trajectory for Kimi-K2.5 after the evidence of unanswerability was already exposed. Third, the selector trained on MuSiQue-Full must keep its advantage on HopRefusalBench. If it does not, it has learned dataset artifacts rather than a reusable signal that a link has broken.