Systems that answer multi-step questions over a document collection, such as "in what city was the director of Inception born?", work in a loop. They retrieve some passages, reason a little, and decide whether to retrieve again. That decision is usually driven by one number: how unsure the language model is. But "unsure" covers two different situations that call for opposite responses. In the first, the needed fact has not been found yet, and another search can fix that. In the second, the retrieved passages disagree or are ambiguous: two people share a name, or two sources give different dates. Searching again then mostly adds distractors, and the model does worse. This proposal builds a small controller that reads the evidence gathered so far as a graph and reports these two kinds of uncertainty separately. The controller then retrieves when evidence is missing, re-reads and disambiguates when evidence conflicts, and stops or abstains when neither action will help.
The cost of conflating the two situations is documented. On FanOutQA, a benchmark where answering a question means consulting five or more long Wikipedia articles, models given open-book retrieval often score worse than with no retrieval at all. The authors attribute this to the model losing track of the question as retrieved text piles up. Dynamic retrieval systems such as MIND, which triggers search when the model's per-word uncertainty (entropy) crosses a threshold, cannot tell whether high entropy comes from a gap or from a contradiction. A contradiction is exactly the case where more retrieval feeds the pile-up.
Core Claim
The claim is that the stopping and next-step policy of an iterative retrieval system should be driven by two signals instead of one. The first is vacuity: no evidence supports any candidate answer to the current sub-question. The second is dissonance: evidence supports several incompatible candidates. Each signal maps to a different action. High vacuity means expand retrieval from the current frontier. High dissonance means re-rank or disambiguate among what is already retrieved, without adding passages. Both low means answer. If, after a capped number of expansions, the system is still vacuous, it abstains.
The prediction is that this split matters most where today's systems fail worst. One such regime is wide "fan-out" questions, where over-retrieval causes forgetting. Another is questions with no answer in the corpus, where single-signal controllers keep searching. On both, the split should cut retrieval calls and raise accuracy compared with a controller that thresholds a single uncertainty score, even when that controller is trained on identical data.
The vacuity/dissonance vocabulary comes from EFGNN. EFGNN is a graph neural network, a model that updates each node's representation by passing messages along graph edges. It maps per-hop class evidence to Dirichlet distributions so that "no evidence" and "conflicting evidence" can be read off separately. With this separation it stays at 83.0% accuracy at 32 layers on Amazon Computers, where a standard graph network falls to 17.0%. EFGNN only ever classifies nodes in fixed graphs. We move its decomposition from passive calibration to active control of retrieval in question answering. That is the new step, and the experiments include a test of whether the Dirichlet machinery itself is needed or only the two-signal split.
Research Program
The evidence store follows PropRAG. The corpus is decomposed offline into propositions, which are self-contained atomic sentences. These are linked to the entities they mention, giving a proposition–entity graph. PropRAG then runs an LLM-free beam search plus Personalized PageRank over this graph and reaches 53.9% F1 on MuSiQue. We keep that retriever unchanged as the "expand" action. The new piece is what decides when to call it and what to do instead.
At each step, the propositions retrieved so far form a small evidence subgraph. Each proposition is encoded with a frozen sentence encoder, conditioned on the current sub-question. A two-layer graph network propagates information along shared-entity edges. This is the hybrid semantic-structural element: the text encoder supplies meaning, and the edges let a proposition about "Nolan" inform one about "the director of Inception". The network outputs non-negative evidence for each candidate answer entity in the subgraph, plus a reserved "not present" slot. From the resulting Dirichlet distribution we compute vacuity as total-evidence deficit and dissonance as the balance of evidence across competing candidates, following EFGNN's formulation.
Graph structure is not optional here, and the reason is concrete. Conflict between two propositions only becomes visible when they attach to the same sub-question slot through different entity bridges. A model reading a flat concatenation of passages has to rediscover those bridges from the text alone.
Training labels come from annotations the benchmarks already provide. MuSiQue composes each question from single-hop sub-questions with known supporting paragraphs. For any partial retrieval state we can therefore compute whether the gold support for the current hop is present. Absent support gives a "missing" state. Present support with no gold competitor gives a "sufficient" state. MuSiQue-Full adds unanswerable contrast questions, which supply states that stay missing however much is retrieved.
Natural conflict labels are scarce, and we construct them in two ways. The first injects propositions about a same-named different entity, mined from the corpus via the entity graph. The second swaps the answer entity in a copy of the gold proposition. This is the weakest part of the design, because synthetic conflict may not look like real conflict. The experiments therefore report results separately on questions where conflict arises naturally. To find those, we flag 2WikiMultiHopQA questions whose bridge entity name maps to several Wikidata items.
Uncertainty-based active learning enters in a narrower but useful role: transferring the controller to a new benchmark with few labels. FanOutQA ships 7,305 human-written sub-question decompositions. Treating these as a labeling budget, we choose which questions to label using median-uncertainty selection in the style of Peng et al., whose federated trajectory work showed that picking median rather than maximum uncertainty avoids unlearnable outliers. We compare this against random selection and maximum-entropy selection.
Baselines
The naive reference point is one-shot retrieve-then-read: dense retrieval of the top passages followed by a single LLM call. A second naive point is PropRAG with a fixed step budget and no controller. PropRAG is the strongest published retriever in the file. It reports 53.9% F1 on MuSiQue and a 64.9% zero-shot average across five benchmarks, against 38.5% for GraphRAG on MuSiQue. Any proposal that adds a controller on top of PropRAG must first match this number in our reproduction with the same reader LLM. Beating it means a statistically significant F1 gain at equal or fewer retrieval calls.
The strongest competitor on the specific question, when to retrieve, is MIND. MIND gates retrieval on token entropy plus attention influence and filters spurious entities with confidence and chain-of-thought checks. It reports +3.0 EM and +3.5 F1 on 2WikiMultihopQA over DRAGIN and SEAKER, with 10–15% fewer retrieval calls. The file gives no absolute MIND numbers and none for IIRC or StrategyQA. We will therefore re-run MIND's trigger over the same PropRAG retriever and reader, rather than compare against its paper. Its threshold gets the same validation tuning budget as ours. Beating it means higher F1 on 2WikiMultiHopQA and MuSiQue at matched average retrieval calls, together with clearly better abstention on MuSiQue-Full. For MuSiQue-Full, no reported result was found on Lacuna for any system's abstention behavior, so we establish all numbers there ourselves.
IRCoT retrieves after every reasoning sentence. It reports gains of up to 15 F1 and 11–21 recall points over one-shot retrieval. It is the "always retrieve" reference and the latency ceiling, and it runs with the same reader and corpus.
HippoRAG is a single-step graph retriever, reported up to 20% above ColBERTv2 on MuSiQue and 2WikiMultiHopQA. It represents the view that no loop is needed at all.
The ablations isolate the key ingredient by holding the graph network, training data, and retriever fixed while changing only the output. The single-head variant predicts one "retrieve again?" probability and thresholds it. The three-way softmax variant directly classifies the state as missing, conflicting, or sufficient, with no Dirichlet. The flat variant removes graph edges and encodes concatenated evidence. If the three-way softmax matches the Dirichlet head, the contribution is the action split and its labels, not subjective logic, and we will report it that way.
Why not just…?
Why not just tune a single uncertainty threshold, as MIND does? This is the main threat, and the single-head ablation tests it directly with identical training data. Our prediction is specific. The single-signal controller should do about as well on questions with no conflict, and do worse on the conflict slice and on unanswerable questions. On the conflict slice it should retrieve more and lose accuracy, because high entropy from a name clash triggers more search. On unanswerable questions it should loop until the budget runs out. If the gap does not show up on those slices, the core claim is wrong.
Why not just ask the LLM to verify whether it has enough evidence? The four-axis survey of multi-hop retrieval–reasoning processes lists verifier models as a stopping criterion, and a prompted verifier can, in principle, say "these passages conflict". The objections are cost, since a verifier needs a full LLM call per step, and the fragile behavior of in-context judgments: Probing the Decision Boundaries of In-context Learning shows that in-context predictions form jagged islands that do not smooth out with scale. Neither objection is decisive on its own, so we include a prompted three-way verifier as a baseline with the same reader model. If it matches our controller's accuracy, we compare on calls and latency. If it matches on those too, the small graph controller has no reason to exist.
Why not skip iteration and use a strong single-step graph retriever plus a long-context reader? HippoRAG and PropRAG already get much of the multi-hop benefit without loops. FanOutQA reports that when the evidence is simply handed over, accuracy tracks context size (r² = 0.782), with Claude 2.1 reaching 0.653 loose accuracy. Against this, open-book FanOutQA performance falls below closed-book, which means that in realistic retrieval the reader does not get clean evidence. That is the regime where choosing between more and fewer passages matters. The PropRAG fixed-budget and HippoRAG baselines settle this directly. If they match our controller on FanOutQA open-book and MuSiQue, the loop adds nothing.
Proposed Experiments
The main evaluation covers MuSiQue-Ans and MuSiQue-Full, 2WikiMultiHopQA, HotpotQA, and FanOutQA open-book. All systems share one open-weight reader LLM and the PropRAG index. We report answer F1 and EM, or loose accuracy for FanOutQA, alongside retrieval calls and wall-clock latency per question. Comparisons are made at matched average call budgets by sweeping each controller's threshold and comparing accuracy–cost curves, not single operating points. On MuSiQue-Full we report how well the system abstains on unanswerable questions without sacrificing answerable ones. The human references are 78.0 answer F1 on MuSiQue-Ans and 68.5% open-book loose accuracy on FanOutQA. These bound how much headroom remains.
The decisive check is an intervention on conflict states. For test states the controller labels high-dissonance, we force each of the three actions and measure final accuracy: retrieve more, disambiguate, or answer immediately. We then do the same for high-vacuity states. The claim holds only if the best action differs by state type and matches what the controller chose. An oracle policy that picks the best action per state, computed from these forced runs, gives the ceiling and shows how much of it each controller captures.
A second check is FanOutQA accuracy plotted against the number of retrieved documents. Our controller should flatten the decline that the FanOutQA authors observed in open-book mode, while the single-signal controller should not.
The transfer experiment trains the controller on MuSiQue and adapts it to FanOutQA using 5%, 10%, and 25% of the human decompositions. Labels are chosen by median-uncertainty, maximum-entropy, or random selection. It tests whether median selection reaches full-label accuracy with fewer labels and whether maximum-entropy selection is pulled toward ambiguous, unlearnable questions as Peng et al. found in trajectory forecasting. Finally, the DEEPSYNTH tasks, where o3-deep-research reaches only 8.97 F1, serve as a qualitative stress test. We do not expect large absolute gains there. The aim is to see whether the controller's abstentions line up with the tasks experts mark as needing sources the system cannot reach.
References
- FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models
- Memory-Aware and Uncertainty-Guided Retrieval for Multi-Hop Question Answering
- Uncertainty-Aware Graph Neural Networks: A Multi-Hop Evidence Fusion Approach
- PropRAG: Guiding Retrieval with Beam Search over Proposition Paths
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Privacy-Preserving and Uncertainty-Aware Federated Trajectory Prediction for Connected Autonomous Vehicles
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
- Retrieval--Reasoning Processes for Multi-hop Question Answering: A Four-Axis Design Framework and Empirical Trends
- Probing the Decision Boundaries of In-context Learning in Large Language Models Download PDF
- A Benchmark for Deep Information Synthesis