Who Can Fix This Claim? Routing Unsupported Facts to the User, the Retriever, or a Hedge

When a retrieval-augmented chatbot writes an answer, some of its individual statements will not be backed by the documents it fetched. Retrieval-augmented generation, or RAG, means the model first looks up passages in a document collection and then writes from them. Today's systems treat every unbacked statement the same way: they either keep it, which means hallucinating, delete it, or search again. These failures have different causes. Sometimes the statement is unbacked because the user's question was ambiguous, for example "the Washington treaty" when there are three, and only the user knows which one they meant. Sometimes the right passage exists but the search query missed it. Sometimes no source in the collection can settle the point. Each cause needs a different fix: ask the user, search again, or hedge. This note proposes a system that splits a draft answer into single-fact statements and checks each one. For each unsupported statement it decides who can resolve it, and when the user is the right party, it turns that specific statement into a short clarifying question.

The idea, in two sentences: break the draft answer into atomic facts and find the ones the evidence does not support. Then test whether each one's truth depends on how the question is read, which means asking the user, or on what was retrieved, which means searching again, and hedge on whatever neither can fix. The opportunity is that answer-level clarification and post-hoc fact editing have each been studied in isolation. Neither uses the other's signal, and so both waste effort in predictable ways. A system that asks the user about missing database coverage annoys them. A system that re-searches an ambiguous question retrieves confidently about the wrong treaty.

Core Claim

The central claim is that the reason a claim is unsupported can be diagnosed per fact, cheaply, from two tests. The first test checks whether the fact's support flips across alternative readings of the user's question. The second checks whether a query targeted at that fact finds support. Routing on that diagnosis gives higher factual precision and better coverage of what the user actually wanted than any single policy of always asking, always re-searching, or always dropping. It does this while asking fewer questions, and the questions it does ask are ones the user can answer. The new element is that clarification is driven by facts discovered while writing the answer, not by ambiguity scored on the query before generation begins. Our hypothesis is that much ambiguity becomes visible only once the system tries to write and verify concrete facts.

Relation to Prior Work

Three systems supply the parts we reuse.

FActScore defined atomic-fact decomposition for scoring long-form text against Wikipedia, and we reuse its decomposition and precision metric. Atomic Fact Decomposition Helps Attributed Question Answering (ARE) goes further: it issues a search query for each atomic fact, verifies the fact with an entailment model, and edits unsupported facts. ARE is our retrieval branch, and its 23.1% verification error rate is the main constraint on our design. ARE has no way to involve the user, so when a fact is unsupported because the question is ambiguous, it re-searches or rewrites toward whatever reading the retriever happened to favor.

CLAIM and Nous supply the clarification method. CLAIM scores a question's ambiguity by how much an ensemble of models disagree about what it means, and chooses clarifying questions that reduce uncertainty about intent. Nous trains a questioner with a reward for reducing uncertainty over the attributes of the user's intent. Both work at the query level, without retrieval, and before any answer exists. We apply the same information-gain criterion, but over readings that have been confirmed to change which facts are supported.

Two papers motivate the work by documenting the gap. VISTA already separates unverified dialogue claims into Contradicted, Lacking Evidence, Out-of-Scope, and Abstention. It does this only as an offline evaluation, and it does not separate "lacking evidence because the question is ambiguous" from "lacking evidence because retrieval missed." DialFact measures the cost of retrieval failure: verification accuracy falls from 69.2% with oracle evidence to 51.6% with system-retrieved evidence. That gap is exactly the share of cases our retrieval branch must catch without bothering the user.

MedRAGChecker and RECLAIM represent strong single-turn, non-interactive claim grounding. MedRAGChecker verifies claims post hoc against literature and a knowledge graph, and RECLAIM interleaves verbatim citations with generated sentences. Neither can return a question to the user. Effective Social Chatbot Strategies for Increasing User Initiative adds a constraint: systems that question users too often fall into an "interrogation trap" that suppresses the user's own initiative. For that reason we treat the number of questions asked as a cost to minimize, not a free action.

Research Program

The system has four stages, and all of them run on an open 8B-class model so that the comparisons stay controlled.

The first stage drafts an answer with standard RAG over the top-k retrieved passages. The second stage decomposes the draft into atomic facts, resolving pronouns and other references against the dialogue history as VISTA does. It then labels each fact as supported, contradicted, or unsupported using an entailment model against the retrieved passages.

The third stage runs two tests on each fact that is not supported. The interpretation test samples $m$ alternative readings of the user's request, using the ensemble-disagreement procedure from CLAIM, and retrieves passages for each reading. It then re-checks the fact under each reading. A fact is interpretation-sensitive if it is supported under some readings and unsupported or contradicted under others. The retrieval test issues an ARE-style query targeted at the fact against the original reading and checks whether support appears. Facts that pass the retrieval test are repaired silently. Facts that are interpretation-sensitive go to the user. Facts that fail both tests are rewritten as explicit hedges or dropped. A fact that passes both tests is the hard case, and we send it to the user only if the newly retrieved support itself differs across readings.

The fourth stage composes at most one clarifying question per turn. It pools the readings implicated by every interpretation-sensitive fact and picks the question $q$ that maximizes

$$\text{score}(q) = \sum_{a \in A_u} \big[ H(R_a) - \mathbb{E}_{y \sim p(y \mid q)} H(R_a \mid y) \big],$$

where $A_u$ is the set of user-routed facts, $R_a$ is the distribution over readings under which fact $a$'s support differs, $y$ is a possible user reply, and $H$ is entropy. This is the information-gain objective of CLAIM and Nous, restricted to readings that actually change which facts survive. A question is asked only if the score exceeds a threshold tuned on development data. Otherwise the system answers with hedges.

Across turns, the system keeps an inventory of facts it has verified, a live version of VISTA's background store. A user's answer or correction then updates specific facts without forcing the whole response to be regenerated. The inventory matters in multi-turn evaluation because it is where hallucination snowballing would otherwise happen (MM-Snowball), but it is secondary to the routing claim.

The main build effort is an evaluation set whose ground-truth routing labels are known by construction. The base is ASQA, the long-form ambiguous-question set used by RECLAIM, whose questions come annotated with multiple disambiguated readings and their answers. For each question, a simulated user privately holds one reading and can answer questions about it but knows nothing else. We create three conditions.

In the user-resolvable condition, the corpus contains gold passages for all readings, so a fact's support depends only on which reading the user intends. In the retrieval-resolvable condition, the question is made unambiguous by fixing the reading in the prompt, and the gold passage is demoted below top-k but left in the collection. In the unresolvable condition, the gold passage is deleted from the collection.

We add facet-ambiguous queries from ClariQ and CLAMBER for breadth, and FaithDial dialogues for multi-turn grounding. A smaller study with real users, perhaps a few hundred dialogues, checks that the simulator is not flattering the method.

Baselines

The naive reference point is plain RAG. It answers from the top-k passages with a "use only the text and cite" instruction, with no verification and no clarification. RECLAIM reports what this level looks like on ExpertQA: ChatGPT reaches a Correct Attribution Score of 66.7% and Llama-3-8B reaches 51.4%. Any method that fails to beat our re-run of this baseline on factual precision is not worth its added cost.

The strongest non-interactive competitor is ARE-style repair: decompose, re-search per fact, and edit or drop unsupported facts. On Mintaka with GPT-3.5, ARE reports a 72.4% improvement in its combined attribution metric ($Ave\text{-}F1_{PP}$) over sub-question editors such as RARR. It is the strongest competitor because it already attacks unsupported facts one at a time and needs nothing from the user. Our system contains ARE as its retrieval branch, so it uses the same decomposer, entailment model, retriever, and per-fact query budget. Beating it means higher recall of the facts for the user's intended reading at equal or better precision, specifically on the user-resolvable condition. We expect the two to tie on the retrieval-resolvable condition, and a tie there is the correct outcome.

The strongest interactive competitor is query-level clarification in the style of CLAIM. Ambiguity is scored before generation and a question is asked if needed, after which the system runs the same RAG and ARE repair. CLAIM reports 81.85% clarification-judgment accuracy on ClariLM-test. For ASQA in our protocol, no reported result was found on Lacuna. It gets the same ensemble, the same question budget, and a threshold tuned on the same development split. Beating it means fewer questions per dialogue at equal recall, or higher recall at equal question count.

The key ablation removes the interpretation test and routes every unsupported fact to the user. This isolates whether the per-fact diagnosis matters, as opposed to simply asking more questions. A second ablation removes the retrieval test and routes everything unresolved by the user to hedging. We also run a single prompted model that is told to verify its draft and ask the user if anything is unclear, which measures how much an LLM judge already does without the structured routing.

Why not just…?

Why not just drop or hedge every unsupported fact? This is the most serious threat, because hedging alone could match our factual precision. VISTA's abstention category and FaithDial's "admit ignorance" persona show the approach is easy to deploy. What hedging gives up is coverage. On the user-resolvable condition, dropped facts are exactly the answers the user wanted, so recall against the user's ASQA reading should fall, and repeated hedging may itself read as unhelpful. If our system does not recover a clear share of that recall at a modest question cost, the hedge baseline wins and we will report that.

Why not just clarify the query up front, as CLAIM does? If ambiguity were always visible from the query, up-front clarification would be cheaper and would make our approach redundant. Our bet is that much ambiguity surfaces only when concrete facts conflict across readings. The experiment that settles this is to split ASQA questions by whether CLAIM's query-level score flags them, and then measure how often our fact-level test finds interpretation-sensitive facts in the questions CLAIM did not flag. If that rate is near zero, the up-front approach suffices and the contribution shrinks to cheaper question selection.

Why not just search harder, with more retrieval rounds or larger k? Searching harder fixes the retrieval-resolvable condition, which is the gap DialFact measures. It cannot fix the user-resolvable condition, because more passages about every possible reading make the answer more confidently mixed, not correct. Our three-condition design measures this directly by giving the re-search baseline a matched or larger retrieval budget.

Why not just ask a strong LLM to judge its draft and ask questions when unsure? BinEval (Ask, Don't Judge) found that holistic judges correlate worse with human judgments than decomposed binary checks, 0.514 against 0.563 Spearman on SummEval. VISTA reports that claim-level verification beats holistic judges on dialogue factuality. Nous shows that frontier models tend to guess missing constraints rather than ask about them. We include the prompted judge as a baseline, and we expect it to over-ask, under-ask, or ask about the wrong aspect, which we measure as the share of its questions the simulated user can answer.

Proposed Experiments

The primary metrics are four. Factual precision is FActScore-style: the fraction of final atomic facts supported by gold passages for the user's reading. Intent recall is the fraction of the gold short answers for the user's ASQA reading that appear in the final answer. Questions per dialogue measures user burden. Question answerability is the fraction of questions whose answer the simulated user actually holds, which directly measures the interrogation trap. Routing accuracy against the constructed labels is the diagnostic that explains the other numbers.

Because the routing tests rely on entailment, ARE's 23.1% verification error is a direct risk. We will therefore report routing accuracy separately with oracle entailment and with model entailment, to show how much of the gain survives noisy verification.

The decisive check is a crossover pattern across the three conditions. On user-resolvable questions, our system should beat ARE repair on intent recall. On retrieval-resolvable questions, it should match ARE while asking almost no questions, and the ask-everything ablation should do clearly worse there on burden and answerability. On unresolvable questions, it should match the hedge baseline in precision without asking. No single fixed policy can win all three conditions, so if routing is real, the router should be within a small margin of the best fixed policy in each condition. If instead the ask-everything ablation matches the full system on burden, or the CLAIM-style baseline matches it on recall, the per-fact diagnosis is not earning its cost and the claim is falsified.

A secondary multi-turn check on FaithDial-style dialogues tests the persistent fact inventory. It injects a user correction to one previously stated fact and measures whether later turns stay consistent, compared with full regeneration. The small human study then checks whether real users rate the fact-derived questions as more answerable and less tedious than query-level ones.