Telling Ambiguity from Distraction: Context-Resampled Clarification for Retrieval-Augmented QA

When a question-answering system that reads retrieved documents is unsure, it can mean three different things. The question may be truly ambiguous, as in "who played young Tom Riddle?", which has several correct answers depending on which film is meant. The retrieved documents may contain irrelevant text that pulls the model toward wrong answers. Or the model may simply not know. Current systems use a single uncertainty signal to decide whether to ask the user a clarifying question. That signal rises in all three cases, so a system exposed to noisy retrieval asks clarifying questions nobody needs, offers options that are invented, and still gives the same answer whichever option the user picks.

The proposal rests on one observation. Uncertainty that comes from genuine ambiguity is stable and attributable: each competing answer keeps reappearing whenever a particular supporting passage is present. Uncertainty that comes from distraction is unstable: answers change depending on which irrelevant passages happen to be in the prompt. So we propose to resample the context, not only the model's output. The model answers repeatedly under random subsets of the retrieved passages. We then cluster the answers and measure which passages each cluster depends on. The system clarifies only when two or more answer clusters are each anchored to distinct evidence. It builds the clarification options from those anchored clusters alone. After the user replies, it answers from the anchoring passages alone.

Core Claim

Resampling the context separates query ambiguity from context-induced distraction, and output resampling alone cannot do this. A clarification policy built on this separation should do three things better than existing methods on ambiguous open-domain QA with injected retrieval noise. It should trigger fewer false clarifications on unambiguous-but-noisy questions. It should propose fewer hallucinated clarification options. It should produce answers after clarification that actually change with the option the user chose. This is the "collapse" failure, where the reader returns the same entity on every branch, reported by Asking Clarification Questions to Handle Ambiguity in Open-Domain QA.

Let the retrieved passages be $D=\{d_1,\dots,d_k\}$. We draw $N$ random subsets $S_1,\dots,S_N \subset D$, generate one answer $a_j$ per subset, and group the answers into clusters $c$ by answer equivalence. We define the support of passage $d$ for cluster $c$ as

$$s(c,d) = P(a_j \in c \mid d \in S_j) - P(a_j \in c \mid d \notin S_j),$$

estimated by counting across the $N$ draws. A cluster is grounded if $\max_d s(c,d) > \tau$, meaning some passage reliably causes that answer. The decision rule follows from this. If there is exactly one grounded cluster, the system answers. If there are two or more grounded clusters with different anchoring passages, it clarifies. If no cluster is grounded, it abstains or retrieves more. Clusters that keep appearing without any anchoring passage are treated as distraction or guessing, and they never become options. Merging clusters that share an answer follows the point from Peter van Beek et al. that clarification is warranted only when interpretations lead to functionally different outcomes.

Relation to Prior Work

Three papers motivate the gap. Selectively Answering Ambiguous Questions separates denotational uncertainty (the question admits several readings) from epistemic uncertainty (the model lacks the fact). It shows that sampling repetition, the fraction of sampled outputs that match the greedy answer, calibrates far better than likelihood. Its evidence is closed-book and single-turn only, and the system only abstains, never clarifies. Breaking Focus: Contextual Distraction Curse in Large Language Models shows that coherent but irrelevant text cuts accuracy by about 45% on average. It also shows the knowledge is still there, since models recover once the distractor is removed. That is exactly the signature our context resampling is designed to detect. Aligning Predictive Uncertainty with Clarification Questions in Grounded Dialog finds that uncertainty rises with dialogue length regardless of how clear the current utterance is. This is a second confounder of the same kind, and we treat it as an extension test.

Two lines of work supply pieces of the method. The noise-robust readers in DAS and Ext2Gen show that weighting answers by passage relevance, or extracting evidence before answering, reduces hallucination under noise. Both assume each question has a single gold answer, so they would treat a valid second interpretation as noise to be suppressed. Our approach reuses their intuition of grounding answers in specific passages, but it keeps several grounded answers when they exist.

The closest interactive systems constrain the design. MIXALIGN reduces hallucination by up to 22.2% through clarification, but it depends on table schemas to find distinguishing attributes. We substitute passage attribution for schema matching, which carries the idea to unstructured text. CLARINET distils expected information gain into a question generator conditioned on retrieved passages. It is judged only on retrieval rank, so it never checks whether its questions mention attributes that exist. CAMBIGNQ reports that 71.8% of GPT-generated clarification questions needed human revision, and that post-clarification answer quality is low. Both are evaluated on clean inputs. Our proposal predicts that both problems get worse under noise, and that both are reduced by restricting options and final reading to anchored evidence.

Research Program

We will build three components on an open instruction-tuned reader. The first is a context resampler, with subset size and $N$ fixed by the compute budget. The second is an answer clusterer using normalized exact match plus a model-based equivalence check. The third is a support estimator implementing $s(c,d)$. The clarification question is a templated multiple-choice question in the CAMBIGNQ format, whose options are the grounded clusters, each labelled with a distinguishing phrase extracted from its anchoring passage. After the user selects an option, the reader answers with only that cluster's anchoring passages in the prompt. This directly tests the "re-entrainment" concern: that the model is pulled back toward stale distractors still sitting in context.

The main analysis is a 2×2 factorial design: ambiguous versus unambiguous questions, crossed with clean versus noisy context. Unambiguous items come from the single-answer portion of AmbigNQ, and ambiguous items from AmbigQA/CAMBIGNQ. Noise comes in two forms. Natural noise means scaling the number of passages retrieved by Dense Passage Retrieval from 5 to 100, as in DAS. Adversarial noise means distractors produced by the error-guided tree search of Breaking Focus. Users are simulated by a language model given the gold interpretation, following the simulated-user protocol of CLARINET. For the dialogue-length extension, we prepend resolved earlier turns and check whether the trigger rate on unambiguous questions stays flat.

Baselines

The naive reference point is a reader that always answers greedily from all retrieved passages and never clarifies. It sets the floor for hallucination and for false clarification, which is zero by construction.

The strongest existing trigger is sampling repetition from Selectively Answering Ambiguous Questions, applied to the full noisy context. It is the strongest competitor because it is the best-calibrated signal reported for denotational uncertainty. The paper reports C@80 of 26.2% on AmbigQA versus 17.4% for likelihood, closed-book. We found no reported result on Lacuna for sampling repetition under retrieval noise. Producing that number is part of the contribution. We give it the same $N$ forward passes, the same reader, and the same threshold-tuning split. When it triggers, it produces options from its own sample clusters. It beats us if it matches our false-clarification rate on the unambiguous-noisy cell at equal recall on ambiguous questions.

The strongest denoising competitor is DAS from Improving Zero-shot Reader by Reducing Distractions, which reports a 64% average exact-match gain over naive zero-shot readers. It is paired with sampling repetition on its query-likelihood-reranked passages. Ext2Gen-style extract-then-answer from Ext2Gen, which reports 0.557 versus 0.473 for default Llama-3.1-8B-Instruct, is run on the same base model where feasible.

For the full pipeline, the reference is the CAMBIGNQ system, which reports detection F1 61.3, option F1 25.1, and clarification-based QA F1 40.5. We compare on its clean test set before adding noise.

Two ablations isolate the key ingredient. The first keeps $N$ fixed but samples outputs at temperature over the full context instead of over context subsets. This tests whether context resampling, rather than extra samples, does the work. The second uses our trigger but builds options and the final answer from the full context. This isolates the contribution of anchored evidence to option validity and to avoiding collapse.

We count it a win if all of the following hold. Area under the ROC curve for separating ambiguous from unambiguous questions in the noisy cells is higher by a margin outside bootstrap confidence intervals. The hallucinated-option rate, meaning options supported by no passage as judged against gold interpretations, is lower. Branch-divergence is higher, meaning the fraction of ambiguous items where different options yield different final answers. Post-clarification F1 is higher.

Why not just…?

Why not just use sampling repetition on the full context? It is the most serious threat. If distractors rarely flip the model's answers in practice, output sampling already captures most of the signal. Our first ablation tests exactly this. The reason to expect a gap is the Breaking Focus finding that distractors push models confidently toward a single wrong answer. Output sampling over a fixed distracting context would read that as low uncertainty. Removing the distractor changes the answer, and context resampling is designed to catch that change. If the ablation shows no gap, we will report sampling repetition as sufficient, and the remaining contribution reduces to evidence-anchored option construction.

Why not just denoise first and then measure uncertainty? If a filter such as DAS or an Ext2Gen-style extractor removes distractors, the dispersion that remains might reflect pure ambiguity. The risk is that these methods are tuned to one gold answer. They have no principled reason to keep the passage supporting a second valid reading, so they may suppress the ambiguity we want to detect. The denoise-then-sample baseline measures this directly. Its recall on ambiguous questions, compared with ours, settles the matter.

Why not just prompt the model to say whether the question is ambiguous? Breaking Focus found that instructions to ignore irrelevant information fail against coherent distractors. CAMBIGNQ found most prompted clarification questions needed human revision, and Selectively Answering Ambiguous Questions found that self-verification is poorly calibrated. We still include disambiguate-then-answer prompting as a baseline in the noisy cells, because its failure under noise has not been measured.

Why not just give one long answer covering every reading? ASQA shows models reach a DR score (its combined measure of fluency and coverage of the distinct readings) of only 32.1 against 61.8 for humans, and 52.5 even with oracle passages. CAMBIGNQ found that 59% of users preferred clarification. We nevertheless report ASQA scores for a variant that writes one paragraph per grounded cluster. That variant shows whether anchoring helps static synthesis too, which would be useful independently of interaction.

Decisive Checks

The central experiment is the unambiguous-noisy cell. If our method's false-clarification rate rises with the number of passages as steeply as sampling repetition's does, the core claim fails. The second check is branch divergence on CAMBIGNQ with 25 to 100 noisy passages. The prediction is that full-context readers collapse onto one answer and anchored reading does not. The third check is cost. We report accuracy against forward passes from $N=5$ to $N=30$, because the 10-plus passes that Selectively Answering Ambiguous Questions already requires are a known latency concern. A method that wins only at large $N$ would point toward distilling the support estimator, in the manner of CLARINET, as the next step.