Grounded Expectations for Conversational Retrieval

People in conversation are terse. After an assistant explains a city's history, the user types "and the population?" or just "why?", and trusts the listener to know what those fragments refer to. Classic dialogue systems handled this by holding expectations: before the user spoke, the system already knew which slots were open, so a fragment like "Three" could be dropped straight into the pending time slot. Modern retrieval-augmented generation (RAG), where a language model answers from passages fetched from an external corpus, has mostly lost this idea. Current systems wait for the fragment, rewrite it into a full question with another model, retrieve, and then generate. Each step is a new chance to invent an entity.

We propose to bring expectations back and to build them from retrieved evidence rather than from imagined text. After each assistant turn, the system computes an expectation frontier. This is the set of knowledge-graph nodes adjacent to what was just discussed, meaning the entities and passages the user could plausibly ask about next. Four decisions then run against this one object. The next user fragment is resolved by matching it into the frontier. Evidence for the frontier is fetched before the user finishes typing. A fragment that matches nothing in the frontier is flagged as a likely misunderstanding or topic shift, so the system can clarify or search globally instead of guessing. The answer is scored for whether the entities it asserts come from the frontier and the retrieved subgraph. Put in two sentences: predict the user's next move as a region of a knowledge graph, not as a simulated sentence, and use that region to interpret follow-ups, prefetch evidence, and catch hallucinations. Because the expectation is a set of real retrieved nodes, the prediction step cannot itself invent facts.

Core Claim

In multi-turn information-seeking dialogue, expectations drawn from graph structure should resolve elliptical follow-ups more accurately and more faithfully than two alternatives. The first is generative query rewriting. The second is expectations formed by simulating the user's next utterance in text. The gain should concentrate on topic-continuing turns. A calibrated "outside the frontier" signal should also recover most of the loss on topic-shift turns. If the frontier helps only because it retrieves more context, a baseline that retrieves the same number of passages from the full history will match it. That is the main thing the experiments must rule out.

Concretely, let $S_t$ be the focus set after assistant turn $t$: the graph nodes cited in that turn. The frontier $F_t$ is the $k$-hop neighborhood of $S_t$ in a text-attributed graph $G$, where each node carries text and edges are links between passages or entities. Each candidate node $v \in F_t$ receives a score $s(v \mid h_t)$ given the dialogue history $h_t$. When the user fragment $u_{t+1}$ arrives, we compute

$$m_{t+1} = \max_{v \in F_t} \mathrm{sim}(u_{t+1}, v \mid h_t),$$

where $\mathrm{sim}$ is a learned matching score. If $m_{t+1} \ge \tau$, the top frontier nodes become the retrieval result and the resolved referent of the fragment. If $m_{t+1} < \tau$, the turn is treated as out-of-frontier, and the system either runs global retrieval or asks a clarifying question. The threshold $\tau$ is calibrated on held-out data.

Relation to Prior Work

The expectation idea comes from symbolic dialogue work. Carbonell mapped fragments into open caseframe slots. Carberry tracked anticipated goals on a stack. Holzapfel et al. used the system's own speech act to re-weight the recognizer's grammar for the expected reply. These works supply the mechanism we reuse: constrain interpretation top-down before the input arrives. Their expectations came from hand-built domain schemas, whereas ours come from whatever subgraph retrieval returned.

The neural descendants simulate text. Predict-Then-Decide generates plausible future user and agent turns and feeds them to a wait-or-answer classifier. On Alibaba After-Sales this lifts accuracy from 66.02% to 71.14%, but nothing checks the simulated turns. The RIFTS study trains a forecaster that predicts when a prompt needs clarification. It motivates our out-of-frontier signal: early grounding failures triple the chance that the next turn fails, and its forecaster raises Llama 3.1 8B from 24.22% to 54.48%. However, that forecaster reads only the conversation and never the knowledge source. Our proposal differs from both. The expectation is a set of real retrieved nodes, and "the user said something we did not expect" becomes a measurable graph event rather than a text classifier's guess.

On the retrieval side, Dialogue-RAG is the closest competitor. It repairs elliptical queries by linking spans from the history into the current utterance, which raises TOPIOCQA retrieval accuracy from 22.43% to 48.65%, and it runs more than three times faster than rewriting with GPT-4o-mini. It is reactive and copies strings. It does not use corpus structure or anticipate the next turn. DialFact shows the same pressure from the evidence side: context-aware retrieval reaches 75.0% recall against 60.8% for claim-only retrieval. Predictive Prefetching anticipates retrieval 8–16 tokens ahead from hidden-state entropy and cuts end-to-end latency by 43.5%. That anticipation is short-range, token-level, and requires white-box access to the model. Frontier prefetch is the discourse-level counterpart: it anticipates the next turn, not the next few tokens, and needs only retrieval outputs.

For graph structure, G-Retriever supplies the method template. It extracts a connected subgraph with a prize-collecting Steiner tree and encodes it with a graph neural network as soft prompt tokens. Removing the graph encoder costs 22.5%, and the method reduces hallucination by 54% relative to global-embedding baselines, but it answers single static questions. GraphFormers and GLEM show that fusing text encoders with neighborhood aggregation beats cascaded pipelines on static citation and product graphs. We use this family only to score frontier nodes. We treat its benefit as an open question to be tested, not an assumption. Disco-RAG adds discourse structure to RAG, gaining 12.74 LLM Score points on Loong. It parses the rhetoric of retrieved documents, not the trajectory of the dialogue, which is the axis we work on.

For hallucination, HalluGraph scores answers by the fraction of their entities and relations supported by the context graph. It reaches AUC 0.94 on legal contract QA against 0.60 for BERTScore. It also constrains our design. It is post-hoc, and it drops to chance on short contexts such as TruthfulQA, which is exactly the terse, elliptical regime of dialogue. We expect the frontier to help because it supplies the missing context: the accumulated conversational subgraph instead of the single short turn. HaluEval shows that ChatGPT detects dialogue hallucinations with only 72.40% accuracy. That result argues against relying on self-checks.

Research Program

The substrate is TOPIOCQA's Wikipedia passage collection. We turn it into a text-attributed graph by adding hyperlink edges between passages and edges between passages that share an entity. TOPIOCQA suits the question because its conversations mix follow-ups on the current topic with deliberate topic shifts. Under our hypothesis, the frontier should help on the first kind and must detect the second.

The system has four parts. The first is a focus tracker that records which nodes each assistant turn drew on. The second is a frontier builder that expands the focus by one or two hops, pruned by a prize-collecting Steiner tree in the style of G-Retriever. This keeps $F_t$ connected and small; we will target a budget of about 50 nodes. The third is a scorer $s(v \mid h_t)$ that ranks frontier nodes as likely next targets. We build three versions of equal parameter count: a SentenceBERT similarity scorer, a scorer that encodes history turns and frontier nodes as one graph and passes messages across it (GraphFormers-style), and an LLM that reads the serialized frontier. The fourth is the matcher and threshold. We train it on TOPIOCQA's gold passages, labeling a turn as in-frontier when its gold passage lies inside $F_t$.

Generation is conditioned on the top in-frontier nodes. We score it with a HalluGraph-style entity-grounding fraction computed against the accumulated conversation subgraph. The same score can gate the answer: if it falls below a threshold, the system regenerates once.

Three analyses complement the headline numbers. First, we measure how often a gold passage for turn $t+1$ lies inside $F_t$ as a function of hop depth. This coverage ceiling bounds everything else. Second, we break every metric down by continuation versus shift turns. Third, we measure user-turn latency when frontier retrieval runs during the gap between the assistant's turn and the user's reply, versus when retrieval waits for the user.

Baselines

The do-nothing reference passes the raw user turn to a dense retriever. Dialogue-RAG reports 22.43% retrieval accuracy for this on TOPIOCQA. A second naive reference concatenates the full dialogue history into the query. It must be run with the same top-$k$ as our method so that "more context" is controlled.

The strongest existing method is Dialogue-RAG's node-linking rewriter, for which the paper reports 48.65% retrieval accuracy on TOPIOCQA. We will rerun it with the same retriever, corpus, and top-$k$, and with the authors' tuning where released. We give our threshold and hop depth the same development-set search budget as its hyperparameters. Beating it means higher accuracy on the held-out split, significant under a paired bootstrap, with no drop greater than 2 points on topic-shift turns. A win on continuations bought by losses on shifts does not count.

The generative rewriter baseline is Llama-3.1-8B or GPT-4o-mini prompted to rewrite the query, run with identical retrieval. It matters because it is what practitioners actually deploy.

The simulated-expectation baseline is PTD-style: the same 8B model generates a predicted next user question after each assistant turn, and we retrieve on that prediction before the real fragment arrives. This isolates the core claim, that expectations built from graph nodes beat expectations written as text, because it keeps anticipation and removes structure.

For the clarification decision, the baseline is the RIFTS Grounding Forecaster recipe, retrained on our data with the same base model.

For answer faithfulness, the reference is FiD-RAG from Shuster et al., which reduced human-rated hallucination on Wizard of Wikipedia from 68.2% to 7.9%. We run it over the same retrieved passages.

Ablations isolate each ingredient. The first swaps the graph-fusion scorer for SentenceBERT and for the prompted LLM. The second removes the out-of-frontier fallback. The third replaces graph neighbors with the top-$k$ passages most similar to the last assistant turn. That last ablation keeps the "anticipate a local region" behavior but drops the edges.

Why not just…?

Why not just rewrite the query with an LLM? The existing evidence cuts both ways. Dialogue-RAG reports that rewriting with GPT-4o-mini was more than three times slower than span linking and introduced entities absent from the history. But a strong 2025-era rewriter might still match frontier retrieval on accuracy. The generative rewriter baseline will settle this directly. If it ties us on accuracy, the remaining argument is only latency and entity grounding, and we will report it that way.

Why not just simulate the user's next question and retrieve on it, as Predict-Then-Decide does? This is the cleanest competitor to the core claim. Simulated turns are unverified text. Once one names a wrong entity, retrieval follows it, and the hallucination literature shows such errors compound. The simulated-expectation baseline tests exactly this, and the per-turn diagnosis will count how often its prediction contains an entity outside the corpus graph. If it matches the frontier, our structural claim fails.

Why not just put the whole conversation into a long-context dense retriever? DialFact's 75.0% versus 60.8% recall shows that history helps. It does not show that structure helps beyond history. The full-history baseline at equal top-$k$, together with the neighbors-without-edges ablation, separates "more context" from "graph neighborhood." We expect full history to be competitive on shift turns and weaker on terse continuations such as "why?"

Why not skip the graph neural network and serialize the frontier into the prompt? G-Retriever's 22.5% drop without its graph encoder argues for learned structure. That result is on static graph QA, though, and serialization is a real threat when the frontier holds only about 50 nodes. For this reason the core claim rests on the frontier itself, and the scorer comparison is an ablation. If the prompted scorer matches graph fusion, we will drop the graph network from the recommended system.

Proposed Experiments and Decisive Checks

The first check is retrieval. On TOPIOCQA we report retrieval accuracy, Recall, and MRR@5, split by continuation versus shift turns and compared against Dialogue-RAG's 48.65%. This check fails outright if frontier coverage at two hops is so low that the ceiling sits below 48.65%. We will measure coverage first, before building anything else.

The second check is the out-of-frontier signal. We evaluate it as a detector of topic shifts on TOPIOCQA using AUROC. As a trigger for clarification, we test it on the MultiWOZ portion of RIFTS, where a database supplies the graph, against the Grounding Forecaster's 54.48%. We will not claim results on RIFTS tasks that have no knowledge source.

The third check is faithfulness. We generate on TOPIOCQA and on knowledge-grounded dialogue from FaithDial, and score outputs with its FaithCritic discriminator, the entity-grounding fraction, and a 300-turn human hallucination audit against FiD-RAG. The prediction is that frontier-scoped generation lowers entity-level hallucination most on elliptical turns, the short-context case where HalluGraph fails. If the grounding score computed over the conversation subgraph still sits near chance on these turns, the mechanism behind our hallucination claim is wrong.

The fourth check is latency. We report user-turn time-to-first-token with frontier prefetch against synchronous retrieval. As context, Predictive Prefetching reports a 62.4% improvement in time-to-first-token, but on single-turn tasks with token-level anticipation.