Long-running chat assistants and agents accumulate far more conversation history than fits in accelerator memory. A common systems answer is to offload the stored attention state (the key–value cache: the per-token vectors a transformer reuses instead of re-reading old text) to host RAM, then copy back over the slow CPU–GPU link only the blocks that look relevant to the current query. The trouble is that "looks relevant" is measured by attention similarity. When a user changes a fact ("my budget is now $1,200, not $600"), the outdated version is about the same topic as the current one, so it scores just as high, and it is often mentioned more times. Relevance-only fetching therefore spends scarce link bandwidth moving superseded facts to the GPU. There, those facts produce the failure that Can Agent Memory Systems Track Evolving State? calls the salience trap: models keep repeating loud, stale values even when the update is in context.
We propose giving every offloaded memory block a validity score that the fetch policy multiplies into its relevance score. Blocks whose facts have been superseded are dropped before they cross the link. The score is maintained lazily. When a fact changes, we append one entry to a small update log on the GPU and do not touch the stored blocks. A block's accumulated invalidation is computed in closed form only when that block becomes a fetch candidate. This is the same catch-up trick that sparse optimizers use to apply deferred regularization to rarely active features. The claim is that one mechanism reduces link traffic and improves accuracy on changed facts, at a per-update cost that does not grow with how many times the stale fact was repeated.
Core Claim
In offloaded memory, relevance and validity diverge exactly where it matters. Stale versions of a fact are high-relevance, and the more often they were stated, the more blocks they occupy. A fetch policy that scores blocks by relevance times a lazily maintained validity term should transfer fewer bytes than relevance-only fetching. It should also answer knowledge-update questions more accurately, without hurting questions about old facts that remain true. Recency-based decay does hurt those questions. The new element is placing fact supersession at the hardware fetch decision, with maintenance cost proportional to the number of distinct facts rather than the number of stored mentions.
Where This Sits
Three strands of prior work each cover part of this and miss the rest.
Offloading systems supply the fetch machinery and motivate the bandwidth problem. Yao et al. show that whether transfer can hide behind compute depends on the ratio of link bandwidth to per-layer compute. Synk offloads long-context caches to a CPU vector store and fetches under 2% of tokens by top-k attention. SpeCache (Jie et al.) keeps a low-precision "scout" copy on the GPU to predict which blocks to prefetch. AtlasKV organizes a billion knowledge triples into a three-level hierarchy, of which only the top level stays on the GPU. All four select by relevance and treat stored content as permanently valid. AtlasKV in particular would need re-clustering when facts change. We reuse their fetch structure and change only the selection score and its maintenance.
Learned eviction supplies a retention score but not the right one. TRIM-KV (Cache What Lasts) attaches a small gate to each attention head that emits a retention value $\beta$ with decay $\beta^{t-i}$. It evicts the lowest-scoring tokens inside GPU memory and reports a 198.4% relative gain over SnapKV on AIME24 under tight budgets. Its gates learn static token roles and have no notion that a later token invalidates an earlier one. Evicted tokens are simply deleted rather than tiered. Darwinian Memory scores symbolic action snippets by reuse, decay and failure rate. Both constrain the design. Decay- and usage-based scores will evict a root fact mentioned once, even though other facts still depend on it, which is exactly the case this proposal must not break.
Obsolescence work motivates the problem and supplies the invalidation logic, but only above the model. StateMem (Can Agent Memory Systems Track Evolving State?) parses each turn with an LLM into structured state units that carry derived_from links. A rechecker invalidates dependents eagerly. The paper reports that state drift explains up to 63.5% of confirmed errors and that current-state accuracy on StateMemBench rises from 0.205 to 0.363 on a DeepSeek-V4-Flash backbone. Mitigating Temporal Misalignment by Discarding Outdated Facts predicts how long a fact type stays true and routes to retrieval once that time has elapsed. We borrow StateMem's dependency propagation and optionally that paper's lifespan prior. Unlike StateMem, we operate on cached attention state at fetch time, need no full LLM parse per turn, and never rewrite the store eagerly.
Lazy optimization supplies the maintenance trick. Tsuruoka et al. track a global cumulative $\ell_1$ penalty plus each feature's last-update time, so that a dormant weight receives its exact accumulated penalty in one step when it reappears. FOBOS (Duchi et al.) shows why this is exact for separable penalties. AdamAR (Adaptive Regularization) shows that rarely touched parameters go stale relative to what surrounds them. Our memory blocks behave the same way: they are dormant, rarely fetched, and silently outdated. The analogy is structural, not a claim that AdamAR's regularizer transfers.
Mechanism
Each offloaded block $b$ carries a few fact keys: entity–attribute signatures such as (user, budget). A lightweight tagger reads the model's hidden states at write time and emits these keys. When a newly written block asserts a value for an existing key $k$, the tagger also emits a supersession strength $s \in [0,1]$. We then update a per-key cumulative log-validity $L_k \leftarrow L_k + \log(1-s)$. The update also propagates along a small key-level dependency graph, so derived keys inherit their parent's invalidation. Each block stores a snapshot $L_k^{(b)}$ taken when it was written. Its validity when it becomes a fetch candidate is
$$ v_b = \exp\Big(\sum_{k \in K(b)} \big(L_k - L_k^{(b)}\big)\Big), $$
where $K(b)$ is the block's key set. The fetch score is relevance times $v_b$, optionally times a lifespan prior. Blocks with $v_b$ below a threshold are skipped and demoted in host storage.
Because updates touch keys and not blocks, a stale fact repeated in fifty blocks costs one log entry. With eager invalidation, all fifty blocks would have to be found and rewritten. Blocks whose keys have no update events keep $v_b=1$ forever. This is how the design protects the once-mentioned, still-true root fact that decay-based eviction would drop.
Research Program
We would build this on top of a relevance-based offloading pipeline in the style of Synk or SpeCache, running on an open-weight model whose cache we control. The first deliverable is a diagnostic rather than a method. On LongMemEval_S and LongMemEval_M (LongMemEval, about 115k and 1.5M tokens of multi-session history), we would measure what fraction of bytes transferred under relevance-only fetching belong to blocks superseded by the time of the query. We would also measure how that fraction scales with the number of times the stale value was repeated.
Next we train the tagger on synthetic dialogues with scripted updates and dependency chains, with the base model frozen. We also build an oracle-key variant that uses ground-truth keys, so that tagger error can be separated from the value of the fetch policy. Finally we compare lazy and eager maintenance as the store and the repetition counts grow.
Baselines
The naive reference is full-context decoding with no offloading. LongMemEval already reports that long-context models drop 30% to 60% relative to oracle evidence. We rerun it on our backbone for a matched number.
The strongest systems competitor is relevance-only offloaded fetching (Synk-style top-k, with SpeCache-style scout prefetch). It is our pipeline minus the validity term and is run with identical transfer budgets, block sizes and hardware. Beating it means lower bytes transferred per query together with higher accuracy on the Knowledge Updates and Temporal Reasoning categories, and no loss on Information Extraction.
TRIM-KV is the strongest learned-retention competitor. It is trained with its published distillation and capacity losses on the same base model and given the same memory budget. We found no reported LongMemEval number for it on Lacuna, so we report our own rerun.
StateMem and StateMemWrapper are the strongest accuracy competitors on changed facts. They must be rerun on our backbone, because the 0.205 to 0.363 StateMemBench result is on DeepSeek-V4-Flash. Beating StateMem means matching its current-state accuracy at substantially lower per-turn ingestion cost; exceeding it outright is not required. We will also test whether stacking StateMemWrapper on our fetch policy adds accuracy.
A recency-weighted fetch baseline uses the same relevance score times an exponential age decay, tuned on the same validation split. Two ablations isolate the key ingredients. Eager validity recomputes $v_b$ for every affected block at each update, which isolates the lazy mechanism's cost. Oracle-key validity provides an accuracy upper bound. Every tuned method gets the same hyperparameter search budget.
Why not just…?
Why not just decay old memories by recency? Recency cannot distinguish a superseded fact from an old fact that is still true, and the literature points to the second case as the fragile one. StateMem's dependency analysis shows that single-mention root facts drive many later answers, and decay-based scores such as TRIM-KV's $\beta^{t-i}$ push those facts toward eviction. Our experiment reports per-category LongMemEval accuracy for the tuned recency baseline. We expect it to help on Knowledge Updates and lose on Information Extraction and Multi-session Reasoning, while validity scoring should help on the first without losing on the other two. If recency matches us across all categories, the key machinery is unnecessary and we will say so.
Why not just run StateMem at the text level and ignore the cache? This is the most serious threat on accuracy. StateMem already gives a 1.8x current-state gain, and prompt-level invalidation may be all the accuracy that is available. Our case rests on cost and bandwidth. StateMem needs an LLM parse of every turn, and it does nothing to stop stale cache blocks from being moved when the history lives in an offloaded cache. We report accuracy and wall-clock ingestion cost side by side, plus the combination of the two. If StateMem alone dominates on both axes at LongMemEval_M scale, the proposal reduces to a bandwidth optimization and should be scoped that way.
Why not just invalidate eagerly when a fact changes? For small stores, eager rewriting of host-side blocks is cheap and gives identical scores. The lazy scheme's advantage is that update cost scales with distinct keys rather than stored mentions, which only matters when stale facts are heavily repeated or stores are large. The eager ablation, measured on LongMemEval_S versus _M and on the repetition stress test, settles whether that advantage is real or negligible. We will report either outcome.
Why not just use a bigger context window or TRIM-KV's learned gates? Long context is already shown to fall 30% to 60% short of oracle evidence on LongMemEval_S, and it leaves both versions of a fact visible, which is the salience-trap setting. TRIM-KV's gates are trained without any supersession signal. The head-to-head comparison at matched budget shows whether its gates learn invalidation implicitly. If they do, the tagger is redundant.
Decisive Checks
Three results decide the idea, in order.
First, the diagnostic must show that superseded blocks take a non-trivial and repetition-growing share of relevance-only transfers on LongMemEval and StateMemBench. If that share is negligible, the bandwidth half of the claim fails and we stop.
Second, the oracle-key variant must raise Knowledge Updates and StateMemBench current-state accuracy over relevance-only fetching while holding Information Extraction accuracy within noise. This separates the policy from tagger quality.
Third, the learned tagger must recover most of the oracle gain. Lazy maintenance must also show flat per-update cost as repetition grows, where eager cost rises linearly.
A controlled stress test varies how many times a stale value is restated before its update. It should show the relevance-only baseline's accuracy falling and its stale-byte share rising, with validity-aware fetching staying flat on both.
References
- Can Agent Memory Systems Track Evolving State?
- Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer
- Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs
- SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
- AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAM
- Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
- Darwinian Memory: A Training-Free Self-Regulating Memory System for GUI Agent Evolution
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- Stochastic Gradient Descent Training for L1-regularized Log-linear Models with Cumulative Penalty
- Efficient Learning using Forward-Backward Splitting
- Adaptive Regularization for Large-Scale Sparse Feature Embedding Models
- LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory