Picking the right reply in a long conversation often means joining two facts said far apart. A user mentions in turn 4 that their sister just moved to Lyon, and in turn 50 asks whether they should visit her in spring. The good reply mentions France. The tempting wrong reply mentions only "your sister." Retrieval-based chatbots score a pool of candidate replies against the conversation instead of writing a reply word by word. They usually summarize the history into one vector, or they only look at the last ten or so turns. Neither design can make this kind of join once the relevant turns are far back. This note proposes a small memory for the conversation that stays the same size no matter how long the conversation gets. The memory stores facts as key–value associations. At reply time it is read several times in sequence, and each read's output becomes the next read's query. This lets the system follow a chain of facts without storing, or re-reading, the raw history.
The larger goal is to settle a question the literature leaves open. Can continuous recurrent memory, meaning a hidden state updated turn by turn, preserve the "bridge" links that multi-hop reasoning needs? Or is it necessary to store and retrieve explicit text? Response selection is the right place to test this. Its metric is deterministic: did the true reply rank first among the candidates. We can also plant the distractor that exposes a failure to reason: a candidate that is correct for the first hop only. That gives a cleaner signal than grading free-form generated text.
Core Claim
Our main prediction is that a fixed-size associative memory read only once will fail on replies that require two or more hops. This should happen even when the memory has room to store every fact, because one read returns the bridge fact, not the composed fact. We expect that chaining two or three reads through the same memory, with a learned projection between reads, will recover most of the gap to explicit iterative text retrieval. It should do so at constant memory cost and lower latency. Accuracy should degrade gracefully as the number of stored facts approaches the memory's capacity.
The memory update comes from Gated Delta Networks. That paper improves Mamba2, a state-space model (a recurrent layer with a linear-time update), by adding an erase-and-replace rule for key–value pairs. In our setting, each turn $t$ writes to a matrix memory $S_t \in \mathbb{R}^{d \times d}$:
$$S_t = S_{t-1}\,\alpha_t\left(I - \beta_t k_t k_t^\top\right) + \beta_t v_t k_t^\top$$
Here $k_t, v_t \in \mathbb{R}^d$ are a key and value computed from the turn's encoding. The scalar $\alpha_t \in (0,1)$ is a learned, data-dependent decay. The scalar $\beta_t \in (0,1)$ is a learned write strength, so the second term overwrites whatever value was stored under $k_t$.
The new part is at read time. Let $u$ be the encoding of the current utterance. We set $q^{(0)} = W_0 u$ and $q^{(j)} = \mathrm{norm}(W_j S_T q^{(j-1)})$ for $j = 1, \dots, J$, where each $W_j$ maps retrieved values back into key space. The context vector is $h = \sum_{j} \gamma_j q^{(j)}$, with learned mixing weights $\gamma_j$. A candidate reply $r_i$ is scored with the dual-encoder dot product $h \cdot r_i$ used by ConveRT. The whole system is trained with the standard next-utterance classification objective: rank the true reply above $k-1$ distractors.
Relation to Prior Work
Several papers motivate the gap. Multi-hop Selector Network (MSN) repeatedly attends from the latest utterance over previous turns to filter out distractor turns. Its "hops" are relevance filtering inside a fixed window, not the composition of facts, and it keeps the raw turns rather than a compressed memory. ConveRT encodes up to 10 past turns into a small, fast dual encoder and never sees anything further back. CMT-RAG comes closest in spirit. It pairs a Mamba-2 recurrent state, used for local discourse continuity, with an explicit graph of reasoning traces for long-range facts, and it answers through a generative reader. That design effectively concedes that the recurrent state cannot carry multi-hop links, which is the assumption we test directly. Our memory must carry them itself, and we score candidates rather than generate.
Other papers constrain the design. MemoryPrompt passes an LSTM-generated soft prompt across segments to a frozen language model. It shows that a tiny model with memory (80.69%) can beat a full-context OPT-2.7B (17.91%) on long fact updating. It also shows that fully fine-tuned recurrent Transformers forget up to 97% of pretrained knowledge. For that reason we freeze the utterance encoder and train only the memory, projection, and scoring heads. Gated DeltaNet reports that Mamba2's needle-in-a-haystack recall collapses after 2K tokens, while the delta rule sustains it to 8K. This is why we use the delta rule rather than plain decay.
On the multi-hop side, IRCoT and Baleen show that re-querying after each hop is what makes multi-hop retrieval work. Chained reads are the in-memory analogue of that re-querying. Direct Evaluation of Chain-of-Thought finds a 15.76% to 16.44% gap between correct final answers and faithful intermediate steps. That result motivates avoiding generated intermediate text. Memory Augmented Sequential Paragraph Retrieval uses a gated, rewritable vector buffer for paragraph retrieval over documents. We borrow its concern about "memory poisoning," which is what the $\beta_t$ gate addresses, but we apply it to dialogue.
Research Program
We will first build the test bed. MuMu-QA already contains dialogues of up to 67 turns in which sub-questions depend on earlier turns. We will turn each answer-bearing turn into a response-selection item with four kinds of candidates:
- the gold reply;
- a "first-hop" distractor that correctly uses the bridge entity but not the final fact;
- distractors that reuse entities from unrelated turns, the lexical trap MSN documents;
- random replies.
We will also build a controlled synthetic set by composing single-hop pairs, following the recipe in MuSiQue. This set varies three factors independently: hop count (1–4), the distance in turns between linked facts, and the number of unrelated facts stored in between. Following MuSiQue's shortcut filtering, we delete the bridge turn from the history and keep only items where accuracy then drops to near chance. This ensures items cannot be solved by surface cues.
We then train the chained-read selector at $J \in \{1,2,3\}$. We sweep memory width $d$ against the number of stored facts to map capacity, and we compare the delta-rule update with a plain-decay update.
Baselines
The naive reference is ConveRT's multi-context dual encoder with its 10-turn window. ConveRT reports 71.2% R100@1 on DSTC7-Ubuntu, where R100@1 is how often the true reply ranks first among 100 candidates. We will retrain it on our data with its published setup. Any gain beyond the window must come from memory. A most-recent-turn BM25 ranker gives a floor.
The strongest in-window response selector is MSN, which reports 80.0% R10@1 on Ubuntu and 60.6% on E-commerce (versus 50.1% for DUA). We will run it on as much trailing history as fits the same GPU memory budget as our model. Our model must match MSN within about one point on standard Ubuntu and E-commerce, which are mostly one-hop and short. It must beat MSN on multi-hop items whose linked facts lie outside MSN's window.
The strongest competitor on the actual question is explicit iterative retrieval over stored turn text. This baseline keeps every turn and retrieves with a dual encoder. It re-queries with the retrieved turn in the style of IRCoT, but without generating text. It then reranks the gathered turns plus the current utterance against candidates with a cross-encoder. It receives the same encoder backbone, the same number of hops, and the same hyperparameter search budget. Beating it means one of two things: at least 90% of its R@1 on two- and three-hop items with lower latency and constant memory, or higher R@1 once session length exceeds its retrieval pool.
We also include a frozen language model with a MemoryPrompt soft-prompt wrapper, used as a candidate scorer, to represent the soft-prompt style of recurrent memory. CMT-RAG reports 41.73 EM on the long-dialogue split of MuMu-QA. That number measures exact-match answers from a generative reader and is not directly comparable to R@1. We will report it as context only. There is no reported result on Lacuna for any response-selection metric on MuMu-QA.
The key ablation is $J=1$ against $J=2,3$ with parameters and FLOPs matched: the single-read model gets a deeper projection so that it has equal capacity. A second ablation replaces the delta rule with plain decay.
Why not just…?
Why not just feed the whole history to a long-context cross-encoder? At 67 turns this may well match us on accuracy, and we will run it. MemoryPrompt's fact-updating result (80.69% versus 17.91%) suggests that having a long context does not by itself mean facts are tracked correctly. The cost also grows with every turn, whereas the recurrent memory stays constant. CMT-RAG found per-turn latency of 1.61s with full-context Transformer encoding versus 0.83s with a recurrent backbone. We will report R@1 against per-turn latency and memory as sessions grow. If the long-context model wins on both at every length we test, then our contribution shrinks to a claim about memory footprint.
Why not just store the raw turns and retrieve iteratively? This is the real threat, and the proposal's central experiment is built around it. It is exactly the "explicit text is mandatory" side of the open question. If this baseline beats chained reads at every hop count under equal latency, we will report that continuous memory cannot carry bridge links at this scale. That is a useful negative result for designs like CMT-RAG's recurrent half. The capacity sweep will show whether the failure comes from memory width or from the reading mechanism.
Why not just widen the dual encoder's window, as ConveRT's multi-context model does? Widening the window helps only when the linked facts fall inside it. The distance-stratified results will show a drop at the window edge for this baseline and a flatter curve for the memory model. If the curves turn out to be flat for both, then our items are not really long-range, and the shortcut filter has failed.
Why not just generate a reasoning chain with a language model and then match candidates? Generated intermediate steps are unfaithful about 16% of the time, even when the final answer is right. An error in the first hop derails everything after it, as the IRCoT paper notes. We will include this baseline and measure how often it chooses the first-hop distractor.
Proposed Experiments and Decisive Checks
The decisive plot is R@1 (and mean reciprocal rank) against hop count on the synthetic set, with one curve per model. The prediction fails if $J=1$ does not drop sharply at two hops while $J=2$ holds up. It also fails if the first-hop distractor does not account for most of the $J=1$ errors. In either case the chaining story is wrong.
The second check is capacity. We plot R@1 against the ratio of stored facts to $d$. We expect degradation to begin near the delta rule's capacity limit, since the memory can hold roughly $d$ nearly orthogonal keys per head. We expect it to be steeper with plain decay.
The third check is transfer to real data: the MuMu-QA-derived items, plus standard Ubuntu and E-commerce to confirm that nothing regresses on ordinary selection.
Finally, as a secondary test, we will use the score margin between the top two candidates to decide adaptively when to stop reading. We accept this rule only if it matches fixed $J=3$ accuracy with fewer reads on average.
References
- Gated Delta Networks: Improving Mamba2 with Delta Rule
- ConveRT: Efficient and Accurate Conversational Representations from Transformers
- Multi-hop Selector Network for Multi-turn Response Selection in Retrieval-based Chatbots
- CMT-RAG: Complementary Memory Traces for Multi-turn Multi-hop RAG
- MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval
- Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs
- Memory Augmented Sequential Paragraph Retrieval for Multi-hop Question Answering
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition