Gated Commits and Dependency Rollback for Conversational RAG Memory

Retrieval-augmented generation (RAG) systems answer questions by first fetching documents and then writing a response from them. In a multi-turn conversation they also keep a memory of earlier turns: which sub-questions were asked, which passages answered them, and what the system concluded. Once an early conclusion is wrong, every later turn builds on it. Prior work has measured this compounding. Isolated-turn accuracy looks high, but live multi-turn performance collapses (Is Your Goal-Oriented Dialog Model Performing Really Well?). Models keep repeating an earlier mistake even when they later see contradicting evidence (Investigating and Mitigating the Multimodal Hallucination Snowballing). And an error written into context becomes a fixed premise for all later reasoning (LLM-as-RNN).

This proposal treats conversational memory the way a build system treats compiled files. Each intermediate result is a node in a dependency graph, and an automated judge decides whether the node may be committed to persistent memory. When later evidence contradicts a committed node, that node and everything derived from it are invalidated and recomputed, while the rest of the history is left alone. The judge is a language model used as an evaluator, or a smaller verifier. Its acceptance thresholds are calibrated so the rate of wrong commits is statistically bounded. That bound is spent most strictly on the nodes many later turns will depend on.

Core Claim

In multi-turn, multi-hop RAG, the decision that matters most is not whether the final answer is good. It is whether an intermediate result should become a permanent premise. A wrong final answer costs one turn. A wrong committed sub-answer can corrupt every turn that later reuses it.

We claim that calibrated per-module commit gates, combined with targeted invalidation along recorded dependency edges, reduce downstream error propagation more than two alternatives at matched compute. The first is end-of-turn checking with full retry. The second is heuristic history pruning. We further claim that allocating the error budget by expected reuse beats a uniform per-node threshold.

The falsifiable signature is specific. After an injected or natural error at turn $k$, accuracy on turns after $k$ should recover toward the no-error level under our system, and it should stay depressed under the baselines.

Relation to Prior Work

The closest pieces exist separately, and none combines a persistent dependency-tracked memory with calibrated online gating and targeted rollback.

Structured memory without verification. CMT-RAG stores conversational memory as a graph of sub-questions, search keywords, passage IDs and dependency edges, and reaches 41.73 Exact Match (EM) on the long split of MuMu-QA. Its authors note that flawed early decompositions populate the graph and compound downstream, and it has no verifier before writing to memory. We reuse its graph format as the substrate and add the missing commit gate and invalidation.

Offline attribution. AgentEval shows that judging each step of an execution graph against its immediate upstream inputs works. It reaches 0.89 failure-detection recall against 0.41 for end-to-end judging, Cohen's κ of 0.84 with humans, and 72% root-cause accuracy. It also finds that 63% of step failures propagate from upstream faults. But it runs only after execution, for debugging. This motivates the proposal: the attribution signal exists, and nobody acts on it while the conversation is running.

Uncalibrated online repair. Two systems repair errors during execution, but neither has a statistical guarantee or a persistent cross-turn memory. ComposeRAG verifies intermediate sub-answers and backtracks the decomposition when they are unsupported, but its prompted verifier is uncalibrated and its scope is single questions. ACR rewrites or prunes dialogue history with a learned router that has no uncertainty estimate. Its "Path Pruning" operator rolls back to a clean prior state rather than invalidating only the affected descendants.

Calibration methods we borrow. CiteGuard caps the false discovery rate of generated claims with conformal p-values from a cross-encoder verifier. Its key observation for us is that errors are correlated across sentences, so independence fails. Judge, Retrieve, or Abstain calibrates a judge's thresholds with Clopper–Pearson bounds and routes uncertain cases to retrieval or abstention. Both are single-turn. We borrow their calibration machinery and confront the problem CiteGuard identifies, correlated errors, now across turns rather than sentences.

Research Program

The system is a modular pipeline built on the module decomposition of Modular RAG. It has four modules: decompose the user turn into sub-questions, retrieve, bind passages to sub-questions, and draft sub-answers. A final module composes the response.

Every module output becomes a node $v$ with explicit parent edges. Each module type $m$ has its own judge. The judge returns a nonconformity score $s(v)$, where higher means more likely wrong, and it checks the node against its parents only, as AgentEval does. A node is used provisionally within the current turn but written to persistent memory only if it passes the gate.

The commit rule is

$$\text{commit}(v) \iff s(v) \le \tau_m(\alpha_v), \qquad \alpha_v = \alpha \cdot \frac{w_v}{\sum_{u \in \text{turn}} w_u},$$

where $\tau_m(\alpha_v)$ is the threshold for module $m$. It is chosen on a held-out calibration set so that the Clopper–Pearson upper bound on the wrong-commit rate is at most $\alpha_v$. Here $\alpha$ is the per-turn error budget. The weight $w_v$ is set to $1/(1+\hat{r}_v)$, where $\hat{r}_v$ is the predicted number of future turns that will reuse $v$. This prediction comes from a small regressor trained on dependency graphs from training conversations.

The allocation is Bonferroni-style, which is valid under arbitrary dependence among nodes. This is why we use it rather than independence-based corrections, given CiteGuard's correlation finding. Nodes predicted to be heavily reused therefore face stricter thresholds.

Failing nodes are re-executed once with an alternative operator, such as a different query rewrite or a retrieval fallback in the style of Judge, Retrieve, or Abstain. If they fail again, they stay provisional, and the response marks the dependent claim as uncertain or abstains.

Rollback is triggered later in the conversation. When a newly retrieved passage refutes a committed node, according to the same cross-encoder or judge, the node is retracted. Its descendants are marked stale and are recomputed lazily, only when a future turn needs them. Untouched branches of memory are kept verbatim, which avoids the discourse fragmentation that naive pruning causes in CiteGuard.

Judges are tiered, following the asymmetric-capacity argument in the modular RAG literature. A DeBERTa cross-encoder serves the bind and sub-answer checks, and an 8B open model serves the decomposition checks. Using open models gives log-probability access, which the calibration needs. A frontier judge is reserved for adjudicating retractions.

The guarantee assumes the calibration and test conversations are exchangeable. MTRAG's shift in the unanswerable-question rate breaks this assumption, and we measure that directly rather than assuming it away.

Baselines

Do nothing. The reference point is the pipeline with every node committed unconditionally, which is essentially CMT-RAG-style memory. On MuMu-QA the strongest reported number is CMT-RAG's 41.73 EM on the long split, a 6.07-point gain over direct RAG (CMT-RAG). We rerun it with our generator and retriever so the only difference is gating and rollback. Beating it means a statistically significant EM gain on the long split and, more importantly, higher accuracy on turns after an error.

Strongest heuristic repair. We compare against ACR's operator router (ACR), retrained with its published recipe of about 3,800 examples, and against ComposeRAG's verify-and-backtrack loop (ComposeRAG) extended to keep cross-turn memory. These are the most direct competitors: both repair errors online, and both lack calibration and targeted invalidation. ACR reports 35.1 EM on HotpotQA. No reported multi-turn number for either was found on Lacuna for MuMu-QA or MTRAG.

Multi-turn RAG with a voting gate. On MTRAG we compare against the AILS-NTUA pipeline (AILS-NTUA) with its three-judge answerability gate. It reports 0.7698 harmonic mean on Task B and 0.5776 nDCG@5 on Task A. No reported Task C number was found on Lacuna.

Fairness controls. All systems share the same retriever, generator and corpus. They are compared at matched total judge-plus-generation tokens per conversation, with equal threshold-tuning data.

Ablations isolating the key ingredients. Three ablations each remove one piece. The first uses uniform $\alpha_v = \alpha/|\text{turn}|$ in place of reuse-weighted budgets. The second keeps gating but removes rollback. The third keeps rollback but replaces calibrated thresholds with the judge's raw yes/no.

Why not just…?

Why not just judge the final answer and regenerate the whole turn? AgentEval reports that end-to-end judging catches only 0.41 of failures against 0.89 for step-level judging, and most failures originate upstream. More fundamentally, a turn-level retry does not touch errors already committed to memory in earlier turns, which is where compounding lives. This is still a real threat on short dialogues. We run it at matched tokens and report results stratified by conversation length. If it ties on long MuMu-QA dialogues, the memory-gating claim is wrong.

Why not just discard memory and re-decompose from the raw history each turn? This removes persistent errors by construction, but long-context models still repeat early mistakes (ACR). History also displaces retrieved evidence after 4–6 turns (AILS-NTUA). We include this stateless baseline explicitly. If it matches our system, persistent memory is not worth its complexity, and we would report that.

Why not just let the model self-reflect? CiteGuard shows that uncalibrated self-correction methods such as Self-RAG and CoVe miss their error targets because errors are correlated. Snowballing studies show that models re-checking their own output favor consistency over correction. The third ablation, raw judge verdicts without calibration, tests this directly inside our pipeline.

Why not just apply CiteGuard or Judge, Retrieve, or Abstain uniformly to every node? This is the closest simple alternative, and the uniform-budget ablation is exactly that. Our hypothesis is that reuse weighting spends the error budget where downstream damage is largest. If uniform budgets perform as well, the contribution reduces to multi-turn deployment of existing calibration, and we would say so.

Proposed Experiments

Error injection (the decisive test). On MuMu-QA and MTRAG we inject controlled errors at turn $k$: a wrong sub-question, a wrong passage binding, or a wrong sub-answer. We then measure accuracy on later turns as a function of distance from $k$. Our prediction is that rollback restores post-error accuracy to within a few points of the uninjected run, while the do-nothing and ACR baselines stay depressed. We also record "contamination," the fraction of committed nodes that are wrong, and "turns-to-recovery."

End-task accuracy. Natural, uninjected performance is measured as MuMu-QA long-split EM and MTRAG Task B and Task C scores. HotpotQA and MuSiQue serve as single-turn sanity checks, where gating should cost little accuracy.

Calibration validity. We check the empirical wrong-commit rate against $\alpha$ on in-distribution splits. We then check it under MTRAG's development-to-test shift in unanswerable questions (6.5% to 19.1%). If the bound breaks under the shift, we test recalibration on a small labeled test-domain sample.

Cost. We report tokens and latency per turn against ACR's 278.5 tokens per turn and CMT-RAG's 0.83 s per turn. We also report the share of judge calls handled by the cheap tier.