Asking from the Evidence: Clarification Decisions for Retrieval-Augmented Models, Tested on Uncontaminated Corpora

When a user asks a question that has more than one reasonable reading, a good assistant should ask which one they mean. Current language models rarely do. In retrieval-augmented generation, the model first fetches passages from a document collection and then writes an answer from them. These systems tend to pick one reading silently and answer it. The usual fix is to ask the model how uncertain it is, but that signal turns out to be unreliable in exactly this case. Instruction-tuned models report near-zero uncertainty on ambiguous questions, and a recent analysis argues that a model's output distribution alone cannot separate "the question is ambiguous" from "the model doesn't know." This proposal takes a different route. A retrieval system already holds an extra source of information that a closed-book model lacks: the retrieved passages themselves. If the top passages split into groups that each support a different answer to a different reading of the question, the question is ambiguous with respect to the collection. If the passages are only off-topic, the problem is noise and there is nothing to clarify.

The second half of the proposal concerns measurement. Almost every testbed for clarification and for robustness to irrelevant context is built from Wikipedia-derived QA sets such as Natural Questions, TriviaQA and AmbigQA, which models have very likely seen during training. A model that has memorized the "default" reading of an ambiguous Wikipedia question will look confident and will not ask. That is a contamination artifact, not a judgment. We therefore build the evaluation on procedurally generated fictional knowledge bases. In these, the number of valid readings, the number of distractor passages, and the user's true intent are all known by construction, and none of it can have been memorized.

Core Claim

Whether to ask a clarifying question in a retrieval system should be decided from the structure of the retrieved evidence: how the relevant passages cluster into mutually exclusive readings, weighted by how strongly each passage actually matches the query. It should not be decided from the spread of the model's sampled answers. We claim three things. First, this evidence-based signal stays calibrated as the number of distractor passages grows, while answer-sampling signals both collapse on real ambiguity and inflate spuriously on noise. Second, standard noise filters, which are trained to keep only what predicts a single gold answer, will tend to discard the passages that carry minority readings. Filtering before deciding to clarify is therefore actively harmful. Third, clarification rates measured on contaminated Wikipedia benchmarks understate the gap between methods, and this can be shown directly by renaming entities.

The motivation comes from four papers. The Illusion of Certainty shows that semantic entropy, internal probes and ensembles all fall from concordance scores of 0.7–0.9 on unambiguous TriviaQA to 0.5–0.6 under ambiguity. It also argues that output distributions alone cannot disentangle the two kinds of uncertainty. That impossibility is about the predictive distribution, so it does not cover a signal computed from retrieved documents, and this is the opening we use. Unanswerability Evaluation for Retrieval Augmented Generation finds that underspecified queries are the hardest unanswerable category for retrieval systems, which guess rather than ask. Its queries come from static QA sets and it scores single-turn intent, not real clarification loops. CORAL shows that accumulated dialogue history acts as noise by turns 4–6, which means any multi-turn clarifier must survive its own history. Search-Time Data Contamination shows that a search agent which finds a leaked answer key overrides its own reasoning and copies the label. On ambiguous queries, we expect a leaked key to fix one reading and silence clarification.

Research Program

The testbed. The testbed is a generator of fictional knowledge bases. Following the triple-distortion idea of INVITE, which builds fresh invalid questions by perturbing DBpedia triples, we sample fictional entities and relations and render them into natural-language passages. We then deliberately create collisions, for example two fictional composers with the same surname or one festival held in two cities in different years. From each collision we generate an underspecified query whose valid readings are known.

The generator has independent controls in the spirit of DyVal, which varies graph depth and adds distractor nodes for symbolic reasoning. One control sets the number of valid readings, from one to four. A second sets the number of distractor passages, from 5 to 100, matching the range over which DAS documents reader collapse. Distractors come in two kinds: plainly irrelevant ones, and "look-alike" ones that share surface terms with a reading but do not answer it. A third control sets the position of the minority-reading passage in the context, since Yun Xiong et al. show that mid-context evidence is under-weighted. A fourth sets the depth of prior dialogue history. A fifth adds unanswerable and out-of-database queries, which are needed to separate "ask" from "abstain."

Because the user's intended reading is sampled before the query is written, a simulated user can answer clarifying questions faithfully for as many turns as needed. This lets us go beyond the single follow-up that most testbeds evaluate. As a bridge to existing data, we also build an entity-renamed counterpart of AmbigQA. Each question and its gold passages are kept, with consistent fictional substitutions for the named entities.

The method. For each retrieved passage, a small extraction model writes the answer that passage gives to the query, if any. This step reuses the extract-then-generate idea of Aligning Extraction and Generation for Robust RAG. Each passage is also given a relevance weight from the query likelihood $P(q \mid d)$, the probability a small open scorer assigns to the query given the passage, as in DAS. Extracted answers are grouped into interpretation clusters using a natural-language-inference model to test mutual entailment, the same clustering tool that INTENT-SIM applies to simulated user replies. The ambiguity score is the entropy of relevance-weighted cluster mass:

$$A(q) = -\sum_{c} w_c \log w_c, \qquad w_c = \frac{\sum_{d \in c} P(q \mid d)}{\sum_{d'} P(q \mid d')},$$

where $c$ ranges over interpretation clusters and $d$ over retrieved passages that yield an answer. Passages that yield no answer contribute to an abstention mass instead.

The system asks when $A(q)$ exceeds a threshold. It abstains when abstention mass dominates, and it answers otherwise. The clarifying question is generated to tell the two heaviest clusters apart. This works like the expected-information-gain questions that CLARINET distills into a model, except that it runs over explicit clusters and needs no training. The user's reply is used to re-weight clusters, not to re-retrieve from scratch. This keeps the growing history out of the generator's context, which addresses the problem CORAL found.

Two properties of the generator matter here. It needs only text outputs from the generator, so it works with closed models. White-box filters like IB-RAG cannot do this. Its cost is one extraction pass over $k$ passages, not the many sampled dialogues that INTENT-SIM requires.

Baselines

The naive reference point is a retrieval pipeline that always answers from the top passages and never asks. Next to it we run an always-ask system, which bounds the achievable accuracy gain and the maximum burden placed on the user.

The strongest prompting competitor is the restrictive-rejection prompt from Unanswerability Evaluation for Retrieval Augmented Generation. We extend it with an explicit instruction to ask a question when the passages support different readings. That paper reports that restrictive prompts improve unanswerable-query performance by nearly 80%, and that Claude 3.5 Sonnet reaches a 10.4% higher Acceptable Ratio than GPT-4o. We run this prompt on the same generator models and the same retrieved passages. We tune prompt wording on a validation split with the same number of trials we allow for our threshold.

The strongest existing decision method is INTENT-SIM from Clarify When Necessary. It samples clarifying questions and simulated replies, clusters them with NLI, and thresholds intent entropy. It reports the highest AUROC in 4 of 6 settings, and at a 20% clarification budget it captures 29% of the total gain available from clarifying everything. The file gives no absolute AUROC value for it, so for that metric there is no reported result found on Lacuna for INTENT-SIM. We run it with the retrieved passages in context, the same generator, and a sample count chosen so that its compute is reported alongside ours. We also run a matched-compute variant.

As the sampling-uncertainty baseline, we compute semantic entropy over answers sampled with the same context. The Illusion of Certainty predicts this baseline will degrade toward 0.5–0.6 concordance.

For noise handling we include the IB-RAG filter from An Information Bottleneck Perspective for Effective Noise Filtering placed in front of semantic entropy. On Natural Questions, that filter reaches 21.5 exact match against 14.7 for raw top-5 retrieval and 16.2 with no retrieval. We also include DAS, which reports a 64% average exact-match gain over zero-shot readers.

For question formation on the original domain we report against CLARINET's reported 0.659 MRR and 58.5% top-1 on Goodreads tip-of-the-tongue search. We also run our cluster-based questions on that dataset.

Two ablations isolate the key ingredient. The first removes relevance weighting, so all clustered passages count equally. This isolates noise robustness. The second replaces passage-derived clusters with clusters of sampled generator answers under the identical pipeline. This isolates "evidence versus output distribution."

Beating the strongest baseline means three things. Our method must achieve higher ask/no-ask AUROC than INTENT-SIM and the extended rejection prompt at every distractor count of 20 or more. Its final task accuracy at a fixed clarification budget must be at least as high. It must use fewer generator calls.

Why not just…?

Why not just prompt a strong model to ask when the passages disagree? This is the most serious threat. Frontier models may notice conflicting passages on their own, and UAEval4RAG's large gains from prompting alone show how much prompting can do. Two things work against it. The same paper finds underspecified queries remain the hardest category even for strong models. And prompted judgments still run through the model's own confidence, which collapses under ambiguity. The extended-prompt baseline settles this directly. If it matches our method within the confidence interval across distractor counts on the fictional corpus, the method contributes nothing beyond cost and closed-model compatibility, and we would report the study as a benchmark result.

Why not just use INTENT-SIM or semantic entropy with the passages in context? Conditioning on passages might in principle rescue sampling-based uncertainty. The concern is that noise adds spread unrelated to ambiguity: look-alike distractors produce varied but wrong answers. Meanwhile memorized priors and entropy collapse suppress spread on genuine ambiguity. The output-cluster ablation answers this, because it holds everything fixed except where the clusters come from.

Why not just filter noise first with IB-RAG and then measure uncertainty? We predict this backfires. The IB objective keeps what predicts a single target $Y$, so passages supporting a minority reading look like noise and can trigger the discard token. We test this by measuring how often the minority-reading passage survives filtering, as a function of distractor count.

Why not just evaluate on AmbigQA instead of building fictional corpora? If contamination does not affect clarification behavior, the existing benchmark is cheaper and more natural. The renamed-AmbigQA comparison tests exactly this. If clarification rates and method rankings do not change after renaming, we would recommend AmbigQA and treat the generator only as a tool for controlled noise.

Proposed Experiments and Decisive Checks

The central experiment varies the distractor count from 5 to 100 and the number of readings from 1 to 4 on the fictional corpus. At each setting we report ask/no-ask AUROC, the unnecessary-question rate, and final accuracy after interactions of up to five turns, all against generator-call cost. The decisive pattern is a crossover. Sampling-based signals should lose AUROC as distractors increase while our score holds, and the no-weighting ablation should degrade in between. If our score degrades as fast as semantic entropy, the core claim is false.

Second, on the original AmbigQA and its renamed counterpart, we measure each method's clarification rate and its F1 over all answers after clarification. For reference, AmbigQA reports 42.3 F1 for its disambiguation model, and more than half of sampled NQ-OPEN questions are ambiguous. The prediction is that renaming raises the clarification rate of sampling-based methods, because memorized defaults vanish, and changes ours little.

Third, we plant an answer-key passage for one reading, formatted like a dataset mirror in the style of the Search-Time Data Contamination setting, and measure how often each system answers that reading without asking. This tests whether a leaked key suppresses clarification and whether evidence-based clustering, which counts the key as one passage among several, resists it.

Fourth, we extend dialogues to CORAL-style depths and check whether reweighting clusters from user replies avoids the late-turn degradation that history accumulation causes.

Finally, a small human study on a subset of fictional queries, with participants who are given a hidden intent, checks that simulated-user results transfer to real people. CLARINET left this gap open.