Targeted Unlearning Against Stale Recall in Dense Associative Memory

Transformer attention can be read as a lookup in an associative memory, a store that returns the full content whose address best matches a partial cue. Under that reading, the familiar failure where language-model agents repeat a fact that has since been overwritten has a simple cause. Softmax retrieval adds up evidence across stored copies, so an old value written ten times outweighs its single correction. We propose a write rule for continuous dense associative memories (modern Hopfield networks) that removes this bias. When a new pattern arrives, the memory checks which stored patterns the new key would retrieve and lowers the weight of those whose values disagree with the new one. Patterns that agree are left alone. This is the continuous, online counterpart of the unlearning step that triples capacity in classical Hopfield networks. We expect it to address two problems at once: the memory keeps returning the current value, and it never fills past the number of facts that are actually still true, so its basins of attraction stay wide enough to tolerate noisy cues.

The test bed is streaming process discovery, the problem of keeping a live model of "what follows what" from an unbounded event log while the underlying process drifts. It offers what language benchmarks lack: ground-truth drift points, explicit memory budgets, and strong non-neural baselines that solve the same problem with counters. We then check whether the same write rule helps on agent state-tracking and time-sensitive question answering.

Core Claim

The "salience trap" documented in Can Agent Memory Systems Track Evolving State? can be stated as a property of the retrieval rule itself. Consider memory slots with keys $k_i$, values $v_i$ and log-masses $a_i$. Retrieval for a query $q$ is

$$r(q)=\sum_i \operatorname{softmax}_i\!\big(\beta\, k_i^\top q + a_i\big)\, v_i ,$$

where $\beta$ is the inverse temperature. With $a_i=0$ and $\beta=1$ this is the update of Hopfield Networks is All You Need, which is equivalent to attention. Suppose an append-only store holds $n$ copies of an obsolete value and one copy of its replacement under the same key. The weight on the stale value is then $n/(n+1)$, regardless of $\beta$. Sharpening the temperature cannot fix this, and frequency alone decides the answer.

Our write rule, applied when $(k,v)$ arrives, is

$$a_i \leftarrow a_i + \log\!\big(1-\epsilon\, p_i(k)\, c_i\big), \qquad p_i(k)=\operatorname{softmax}_i(\gamma\, k_i^\top k + a_i),\quad c_i = \tfrac{1}{2}\big(1-\cos(v_i,v)\big),$$

where $\epsilon\in(0,1]$ is the unlearning strength, $\gamma<\beta$ is a lower "unlearning temperature" that spreads the correction over nearby keys, and $c_i\in[0,1]$ measures how much slot $i$'s value disagrees with the new one. After this update the new slot is appended, or merged into a near-duplicate. A slot whose mass falls below a floor is deleted, which gives a hard memory budget.

The method comes from Analysis of the Hopfield Model Incorporating the Effects of Unlearning. That paper subtracts the network's own high-temperature correlations, $J'_{ij}=J_{ij}-\epsilon\langle s_is_j\rangle_\gamma$, and shows the critical load rises from about 0.14 to about 0.42 inside a crescent-shaped region of $(\epsilon,\gamma)$. Our rule likewise subtracts what the memory itself would retrieve at a softer temperature, with two changes. It is gated by value disagreement, and it runs online on continuous softmax memories. The cited analysis covers only binary pairwise networks in batch.

Why This Intersection, and What Is Already Done

Each of the four directions supplies a different piece.

The motivation comes from knowledge obsolescence. Updating the retrieval corpus leaves models 15 points less accurate on changed facts than on stable ones (SituatedQA). State drift persists even when the correct context is fully present in the prompt (Can Agent Memory Systems Track Evolving State?), which points at the retrieval rule rather than at retrieval coverage.

The constraint comes from capacity theory. Dense memories store exponentially many patterns, but attraction basins shrink as the load nears that limit, and Gaussian-like patterns with uneven norms split into well- and poorly-retrieved groups (Exponential Capacity of Dense Associative Memories). An append-only memory fed a stream therefore degrades on noisy cues over time, even before stale recall becomes visible.

Streaming discovery supplies the evaluation regime and the strongest non-neural competitor. Lossy Counting with Budget maintains activity, case and succession counts under a fixed memory budget and tracks drift better than a sliding window (Control-flow discovery from event streams).

The closest neural relatives each stop short of this proposal in a specific way. Semantically-correlated memories in a dense associative model uses anti-Hebbian self-inhibition to walk a memory graph, but the graph is fixed and nothing is ever forgotten. Unlearning-based sliding window for continual learning under concept drift unlearns the oldest chunk of a convolutional classifier. Its unlearning is keyed by age, not by contradiction, and it struggles under semantic drift. AURORA adapts a retrieval index to drift with meta-learning but has no notion of supersession or capacity. StateMem resolves invalidation with an external parser and a symbolic dependency graph. We ask whether the memory's own dynamics can do the work.

None of these papers performs contradiction-gated unlearning inside a continuous associative memory, and none measures basin size under streaming load.

Research Program

The first part is analysis. Using the signal-to-noise approach of The S/N Analysis of Associative Memory, carried over to the log-sum-exp energy, we derive the stationary number of live slots under three regimes: append-only writes, uniform exponential decay, and our rule. The analysis assumes a stream with volatility rate (the fraction of keys that change per unit time) and a skewed write frequency. We expect two results. Append-only load grows linearly with stream length, so basin radius shrinks steadily. Under our rule, load converges to roughly the number of currently valid key-value pairs plus a transient term set by $\epsilon$, so basin radius plateaus. We will also characterize the $(\epsilon,\gamma)$ region where stale recall is suppressed without erasing near-neighbor keys. Our hypothesis, to be tested rather than assumed, is that it forms a crescent analogous to the discrete case.

The second part is the memory module itself. We build it with merge-on-near-duplicate and a mass floor that enforces a slot budget, so that it can be compared one-to-one against Lossy Counting with Budget. The cost per write is one extra softmax over the slots, the same order as a read.

The third part embeds the module in two settings. In process streams, keys encode the current activity together with a short prefix embedding, and values encode the next activity. In agent memory, keys encode embedded entity–attribute mentions, values encode the asserted statement, and the top slots by mass are placed in the prompt.

Baselines

The naive reference point is an append-only modern Hopfield memory under the same slot budget, evicting first-in-first-out when full. It shows what happens when the retrieval rule is left alone, and the salience argument predicts it fails on frequently written, later-corrected facts.

The strongest generic competitor is the same memory with uniform exponential decay of $a_i$ per write. It has one hyperparameter, it is what most engineers would try first, and it bounds load. Its decay rate gets the same tuning budget as our $(\epsilon,\gamma)$: an equal number of grid points, selected on a held-out portion of each stream.

A second strong competitor is hard overwrite. On each write, the nearest stored key above a cosine threshold is replaced, with the threshold tuned on the same held-out portion. This is the discrete-thinking answer to supersession.

For process discovery, the strongest existing method is Lossy Counting with Budget from Control-flow discovery from event streams, run at equal memory measured in stored entries. That paper reports qualitatively that LC and LCB beat a sliding window on model-to-model similarity across artificial streams with two drifts. The file gives no numeric value: no reported result was found on Lacuna for model-to-model similarity or BPIC-12 fitness and precision numbers, so we rerun LCB ourselves.

For agent state, the strongest existing method is StateMem. On StateMemBench with a DeepSeek-V4-Flash backbone it reports current-state accuracy of 0.363 against 0.205 for the strongest prior baseline, and StateMemWrapper adds 32 to 67 points across six memory backends (Can Agent Memory Systems Track Evolving State?). We use the same backbone and grading. On SituatedQA, the reference is the adaptive system of Mitigating Temporal Misalignment by Discarding Outdated Facts at 20.5 exact match.

Two ablations isolate the key ingredient. The first sets $c_i\equiv 1$, which turns the rule into key-targeted decay and tests whether value-disagreement gating matters. The second sets $\gamma=\beta$, which tests whether the softer unlearning temperature matters, as it does in the discrete theory.

Beating the baselines means the following. On synthetic and process streams, the method must match or exceed the best tuned baseline on current-value accuracy and also be significantly better on long-lived stable facts or on recall from noisy cues, at equal memory. On StateMemBench, our memory plugged in as a backend must exceed the strongest non-StateMem backend. Matching StateMem's 0.363 without a symbolic rechecker would be a strong result, and adding StateMemWrapper on top should still help.

Why not just…?

Why not just decay old memories exponentially? This is the real threat. When every fact changes at a similar rate, decay will probably match our rule. Decay does not distinguish keys: it erases a fact that is old, frequent and still true just as fast as one that was corrected. The test is a stream mixing never-updated keys with volatile ones under heavy frequency skew. The prediction is that decay must trade stable-fact recall against stale suppression along a single curve, while our rule moves off that curve. If it does not, the gating claim fails, and the work reduces to the capacity analysis.

Why not just overwrite the nearest slot? With clean, discrete keys such as activity labels in BPIC-12, hard overwrite and counting are nearly exact, and we expect them to be competitive there. Hard overwrite has two weaknesses. It depends on a sharp threshold, which misfires when the same entity is described differently or when keys are noisy. It also cannot represent partial drift, where the successor distribution shifts rather than flips. We vary the key noise level and the drift type, abrupt versus gradual, to map where the soft rule pays off. Results for the regime where it does not are reported plainly.

Why not just use a sliding window or retrain? Burattin et al. found the sliding window both about twice as slow per event as lossy counting and the worst on model-to-model similarity under drift. Unlearning-based sliding window for continual learning under concept drift reports sliding-window recovery from drift taking 17 chunks against 6.33 with targeted unlearning. We still include a sliding window in the synthetic runs as a sanity check.

Why not just keep everything in a long context and let attention sort it out? By the argument in the Core Claim, this is the append-only memory at its largest. StateMemBench shows drift persists even with oracle context, and that chain-of-thought reasoning sometimes makes it worse, so we expect no improvement from length alone.

Proposed Experiments

The first experiment is synthetic. We generate key-value streams of up to $10^6$ writes with controlled volatility, Zipfian write frequency, key noise and value correlation. We report three quantities. Current-value accuracy is split into volatile and stable keys. Basin radius is the largest cue corruption at which retrieval stays correct, tracked against stream length. Live slot count is tracked over time. The decisive check is whether basin radius under our rule plateaus while append-only shrinks, as the capacity argument predicts, and whether the ablations lose this property.

The second experiment uses process streams. On the three PLG artificial processes with two drifts each, and on BPIC-12 (over 262,000 events), we measure next-activity accuracy after each drift, recovery time in events, and model-to-model similarity of the recovered succession graph, which is read off as the high-mass slots. LCB serves as reference at equal memory. A cyclic-drift variant, where an old regime returns, tests whether soft mass reduction keeps recoverable traces that pruning discards.

The third experiment is language. The memory serves as a backend on StateMemBench under its closed-pool grading and the DeepSeek-V4-Flash backbone, with and without StateMemWrapper. On SituatedQA we compare exact match on updated versus stable facts, the 15-point gap, against the 20.5 adaptive system.

The decisive outcome is the stable-versus-volatile split. If contradiction-gated unlearning narrows the updated-versus-stable gap on SituatedQA and StateMemBench, while exponential decay can only narrow it by also giving up stable-fact accuracy, the core claim stands. If decay matches it everywhere, the claim does not.