Language models and the memory systems attached to them keep acting on facts that are no longer true. The problem persists even when the correct, current fact is sitting in the prompt. Agents given the full history of a conversation still answer with a budget the user revised three sessions ago. Editing methods that teach a model the new office-holder erase who held the office before. We propose that both failures share one mechanism. The retrieval step in these systems, whether it is transformer attention or a Hopfield-style associative memory (a store that reconstructs a whole memory from a partial cue by pulling it toward the most similar stored pattern), adds up evidence across everything it has stored. Under additive readout, a value mentioned five times beats a value mentioned once, however recent the single mention is. Stale facts win because they are loud.
We therefore propose an associative memory in which a new value for an existing "slot" (an entity-attribute pair such as user's budget or UK prime minister) does not add a competing pattern. It supersedes the old one. The old pattern stays stored with a closed validity interval, so historical questions can still complete it. Repeated mentions of the same value merge into one pattern instead of piling up weight. Pattern completion from a vague cue ("what am I working with now?") then returns the one value valid at the query time, together with its chain of predecessors. The project has two parts. The first is a falsifiable diagnostic: under additive readout, the advantage of a stale value should grow roughly with the logarithm of how often it was repeated. The second is a drop-in memory layer evaluated on state-tracking and temporal-editing benchmarks against symbolic and retrieval baselines.
Core Claim
The phenomenon of state drift under oracle retrieval was documented in Can Agent Memory Systems Track Evolving State?. In that paper, models anchor on frequent or salient superseded facts even when everything relevant is in the prompt, and chain-of-thought makes it worse. We claim this drift is not a reasoning failure. It is the expected behavior of softmax associative retrieval over repeated patterns.
Take a modern Hopfield or attention readout with query $q$, stored keys $k_i$, and inverse temperature $\beta$. The probability mass assigned to value $v$ is
$$ P(v \mid q) \propto \sum_{i:\,\text{value}(i)=v} \exp(\beta\, k_i^\top q). $$
Suppose the old value has $n$ near-identical stored copies with similarity $s_{\text{old}}$ and the new value has one copy with similarity $s_{\text{new}}$. The log-odds favoring the stale value is then $\log n + \beta(s_{\text{old}} - s_{\text{new}})$. Any recency signal must overcome a $\log n$ penalty that grows with repetition. This is the same "attractor deepening" that associative-memory theory treats as a feature (Huang et al. describe stored configurations as basins that pull degraded inputs to the nearest codeword). It is also exactly what consolidation designs such as Memini build in on purpose.
Our fix keeps completion but changes the readout inside a slot. Let $c(i)$ be the slot a stored pattern belongs to, and let $[a_i, b_i)$ be its validity interval. We set $b_i = \infty$ until a different value is written to the same slot. Duplicates are merged into a single pattern with a count $m_i$. The readout at query time $t_q$ is
$$ P(i \mid q, t_q) \propto \exp(\beta\, k_i^\top q)\;\mathbb{1}[a_i \le t_q < b_i], $$
and the default is $t_q$ = now. The count $m_i$ is kept but used only where consolidation helps, namely completing a very weak or noisy cue when no valid pattern clears a similarity threshold. It never enters the competition between values of the same slot. Consolidation strength and susceptibility to being superseded are thereby decoupled. The file identifies this tension (Gap 2) and no cited system resolves it.
Relation to Prior Work
Several papers motivate the proposal by documenting the failure. Can Agent Memory Systems Track Evolving State? shows drift under perfect retrieval. History Matters shows that ROME, MEMIT, MEND and CFT drive historical recall to near zero, because they treat a valid temporal transition as an error correction. RealTime QA shows models falling back on outdated knowledge when retrieved evidence is incomplete.
The closest work bridging the two directions supplies pieces of the method but not the mechanism. Memini gives external association-graph edges fast and slow synaptic variables so that repeated associations consolidate. Its edges are unlabeled, so it cannot tell which value of which attribute is current. Its design also deliberately strengthens repeated associations, which is the salience trap. We invert that choice within a slot. SHARC completes full hidden-feature patterns from sparse cues and prunes old patterns with Bayesian forgetting. We reuse its idea of doing associative completion in a model's hidden space, but age-based pruning deletes exactly the history we need to keep. Learning with Partial Forgetting in Modern Hopfield Networks makes Hopfield readout non-bijective by thresholding low activations. That forgetting is instantaneous and unrelated to time, whereas ours is keyed to validity intervals. A generative model of memory construction and consolidation pairs a Hopfield "hippocampus" with a generative "neocortex" and shows memories drifting toward prototypes as they consolidate. That work constrains our design: it predicts that any consolidated store will bias recall toward the typical old value, which is why counts stay out of within-slot competition.
The symbolic competitors are StateMem, which parses turns into state units with explicit dependency edges and re-checks derived facts, and TEMPLE-MQA, which builds a time-aware graph with explicit date matching. Both get supersession right by construction, but only when an LLM parser correctly canonicalizes each mention into a slot. Our proposal learns slot membership as a pattern-separation problem. Mentions of the same slot, possibly phrased very differently, must land in one basin, while different slots of the same entity must stay apart. This is the separation/completion trade-off attributed to the hippocampal CA3 region in Doeller et al..
Research Program
The first stage is diagnostic and needs no new architecture. We take an open-weight model and construct oracle-context prompts in which a slot's old value appears $n \in \{1,2,4,8,16\}$ times and the new value appears once. Position, phrasing, and distance are controlled. We measure the log-odds of the old value in the output and in the attention mass reaching the value tokens. The additive-superposition account makes three predictions: log-odds rises approximately linearly in $\log n$; the slope is roughly invariant to where the update appears; and deduplicating the stale mentions in the prompt removes most of the drift. If the slope is flat, drift is not a readout effect and the rest of the program is poorly motivated. We would report that result as the finding.
The second stage builds the memory layer. Each conversation turn or document sentence is encoded into a key from the frozen LLM's hidden states. A small key head is trained contrastively so that mentions of the same slot cluster and distinct slots separate, and a value head extracts the asserted value. Writing works as follows. If a new pattern falls in an existing slot basin with a matching value, it merges and increments $m_i$. If the value differs, it closes the old interval and opens a new one. Otherwise it creates a new slot. Reading iterates Hopfield updates from the query for a few steps, since a partial cue may first need completing to the right slot, and then applies the interval gate above. The completed current state, plus the predecessor chain when the query is historical, is written into the prompt as a short structured block. Training data for the key head comes from SUMIE-style synthetic update streams and from StateMemBench training scenarios. The evaluation scenarios are held out.
The third stage tests retention of history on temporal knowledge editing. We write AToKe fact chains into the memory rather than into weights and check that current and historical questions both complete correctly.
Baselines
The naive reference point is the backbone LLM given the full interaction history in context. We also report it under oracle context, where only the relevant turns are included, following the StateMemBench protocol. This isolates readout failure from retrieval failure.
The strongest existing method on the central question is StateMem. On StateMemBench with a DeepSeek-V4-Flash backbone it raised current-state accuracy from 0.205 to 0.363 over the strongest baseline. Its prompt-level StateMemWrapper gave lifts of +32 to +67 points across six memory backends. We run StateMem, and our layer with and without the wrapper, on the same backbone, the same held-out scenarios, and closed-pool grading. Beating it means exceeding 0.363 current-state accuracy with the same backbone, while also matching or exceeding it on historical-state questions, which StateMem is not designed to prioritize.
For temporal editing, the reference points are METO, which lifted HES* on AToKe-ME from below 0.3 to over 15 for some editors, and TEMPLE-MQA, which reports an HES of 97.25 on AToKe-SE. TEMPLE-MQA's near-ceiling single-edit score means our claim there is limited to multiple and extending edits (AToKe-ME and AToKe-EE), where accumulated interference should matter. We run it with its published prompts.
For partial-evidence temporal QA on SituatedQA, the reference is the adaptive-inference system of Zhang & Choi at 20.5 Exact Match. The same file reports 23% EM for the best finetuned retrieval model; these come from different settings and we will re-run both on one split.
Memini is the associative comparison. No downstream QA result is reported on Lacuna for Memini, so we implement it over the same encoder, reading the entity with the highest activation as the answer.
Two ablations isolate the key ingredient. The first keeps everything but replaces the interval-gated readout with the additive readout, with and without merging duplicates. This tests whether supersession, rather than the learned encoder, produces the gain. The second replaces the trained key head with a frozen off-the-shelf sentence embedder. This tests whether learned slot separation matters.
Every system gets the same backbone, the same training scenarios for anything trainable, and an equal hyperparameter search budget of the same number of configurations, selected on the same validation split.
Why not just…?
Why not just use a dense retriever and pick the most recent matching passage? This is the simplest competitor and a serious one. It fails in principle whenever the most recent mention is not the most recent assertion. Examples are "back when it was $600…", a hypothetical, or a reaffirmation of the old value. It also inherits the top-$k$ salience problem when stale mentions crowd the retrieved set. We will build this baseline with our own trained slot encoder plus timestamp re-ranking, so the only difference from our method is mention-recency versus validity-interval readout. StateMemBench and SUMIE contain enough non-update mentions to separate the two. If they tie, we will conclude that the attractor dynamics add nothing beyond a good slot encoder and say so.
Why not just store timestamped (subject, relation, value) triples and let the latest win? On benchmarks that hand over clean triples, such as AToKe, this is essentially what TEMPLE-MQA does, and it will probably match us. We concede that regime. The open question is unstructured dialogue with implicit transitions, which Fan et al. list as unverified for StateMem. There, the symbolic store is only as good as the LLM's canonicalization of each mention into a slot. SUMIE shows how fragile that step is: strong models stay below 80.4% F1 on incremental entity updates, with 25% of extracted values unsupported by evidence. We report accuracy stratified by whether a scenario's updates are stated explicitly or only implied, and our claim is confined to the implicit stratum.
Why not just prompt the model to reconstruct the value chain, or rely on a longer context? The StateMemWrapper result shows prompting helps a lot, and we include it as a baseline and as a composable add-on. But long context is the oracle condition under which drift was observed, and chain-of-thought made drift worse in Fan et al.. The stage-one diagnostic tests directly whether the wrapper removes the $\log n$ repetition slope or only shifts its intercept. If the slope survives, heavily repeated stale values will still win at high $n$.
Why not just edit the weights? History Matters shows standard editors erase history. METO recovers some of it only with gradient access and a multi-objective loss that the authors do not show scaling to long edit sequences. An external store leaves the weights untouched. The AToKe-ME comparison will show whether it retains history better across many sequential edits.
Decisive Checks
The project succeeds if three things hold. The diagnostic shows a clear positive $\log n$ slope that deduplication removes. On the implicit-update stratum of StateMemBench, the gated layer beats both StateMem's 0.363 and the recency-reranked retriever with the same backbone. On AToKe-ME, historical and current recall both stay high as the number of edits grows. The additive-readout ablation carries the most weight. If adding merging and gating to the identical system does not close most of the gap between the naive baseline and our method, then the supersession mechanism is not what explains the results.
References
- Can Agent Memory Systems Track Evolving State?
- Clustering of neural code words revealed by a first-order phase transition
- History Matters: Temporal Knowledge Editing in Large Language Model
- RealTime QA: What's the Answer Right Now?
- Continual Knowledge Updating in LLM Systems: Learning Through Multi-Timescale Memory Dynamics
- Saliency-Guided Hidden Associative Replay for Continual Learning
- Learning with Partial Forgetting in Modern Hopfield Networks
- A generative model of memory construction and consolidation
- Multi-hop Question Answering under Temporal Knowledge Editing
- Human neuroimaging studies on the hippocampal CA3 region – integrating evidence for pattern separation and completion
- SituatedQA: Incorporating Extra-Linguistic Contexts into QA
- Mitigating Temporal Misalignment by Discarding Outdated Facts