Horizon-Matched Forgetting for Agent Memory on Shared GPUs

Long-lived chat agents keep each user's conversation history in fast GPU memory. Part of it is stored as the cached internal states of past turns (the "KV cache", the attention keys and values a transformer stores so it need not reprocess old text). Part is stored as summaries in a memory hierarchy. When many users' agents share one GPU, a spike from a co-located job, the "noisy neighbor", suddenly shrinks the room available. The server must then decide, per item, whether to keep it, move it to slower host memory, collapse it into a summary, or throw it away. Current systems decide this in one of two ways. Some look at what the model is attending to right now: prompt-guided compression such as Finch, and sampling-based eviction such as ImportanceAttention. Others refuse to touch the cache and make room elsewhere, as Oneiros/MIRAGE does by evicting model weights. Neither asks the question that human-memory research has found to matter: will this item still be needed after a delay, and how fast does that need fade?

Research on how memorable videos are shows that what is salient now and what survives later are only loosely related. In VideoMem, short-term and long-term memorability correlate at only ρ = 0.305. Newman et al. show that each item has its own baseline and its own decay slope. We propose to carry this idea into memory management for agents. Each cached memory item gets a small forgetting curve: a predicted probability that it will be referenced again at each future delay. Under contention, the server sheds memory by expected loss over the horizon the contention episode is predicted to last, not by current attention. The neighbor's pressure enters twice. It sets how much must go, and, by saturating memory bandwidth, it sets how costly it is to bring a demoted item back.

Core Claim

The claim has two parts, each testable on its own. The first is empirical, and the proposal depends on it. For long-horizon agent memory, the signal current eviction methods use (attention mass during the current turn) predicts cross-session reuse poorly. It predicts it the way perceptual salience predicts multi-day video recall: moderately at best. Naim et al. found that items easiest to recall immediately can be the worst retained. Our analogue is that tokens that dominate attention now, such as greetings, formatting and the current task's scaffolding, are often not the facts a user will ask about three sessions later.

The second part is systems-level. Let $p_i(t)$ be the predicted probability that memory item $i$ is referenced at delay $t$, parameterized by a base rate $m_i$ and decay slope $\alpha_i$ as in Newman et al. The value of keeping item $i$ in fast memory through a contention episode of predicted length $H$ is then

$$U_i(H) = c_i(b)\int_0^{H} p_i(t)\,dt ,$$

where $c_i(b)$ is the cost of needing the item when it is not resident. That cost equals the reload latency at the current available bandwidth $b$ if the item was moved to host memory, and a quality penalty if it was collapsed to a summary or dropped. Items whose need arrives after $H$ should be moved to host memory rather than kept or dropped. Items with high $m_i$ and low $\alpha_i$ should be consolidated into a durable summary rather than discarded. Because $c_i$ rises as neighbors eat bandwidth, the best choice between moving and summarizing shifts with contention. No existing method models this coupling.

Research Program

The memory unit is a round-level segment of conversation, following LongMemEval's finding that round-level granularity helps retrieval. Each segment is held in one of four states: resident KV in GPU memory, KV moved to host memory, a node in a TiMem-style temporal summary tree, or deleted.

We first build a reuse dataset. We replay LongMemEval_S/M and LoCoMo conversations turn by turn, in timestamp order. At each later turn and at question time, we record which earlier segments are needed. There are two sources of labels. Gold labels come from the benchmarks' annotated evidence sessions. Denser proxy labels come from which segments a full-context reference model and a retriever actually draw on at each later turn. The gold labels are few (LongMemEval has 500 questions and LoCoMo 10 user groups), so the proxy labels do most of the training, and gold labels are held out for evaluation.

On this data we fit a small predictor that outputs $(m_i, \alpha_i)$ from features computed when the segment is written: fact and entity density, overlap with the user-profile tier, and current-turn attention mass. The first of these follows the finding that semantically rich content resists decay. We pick the curve's shape (linear, exponential or power-law) on validation data rather than assuming one.

The policy has three parts. A contention monitor tracks free GPU memory and neighbor-induced bandwidth use, predicts episode length $H$ from recent history, and reads reload cost from measured transfer rates; interference is read from runtime counters in the spirit of VELTAIR and iGniter. A planner allocates per-tenant budgets and selects actions by $U_i(H)$ per byte. A consolidation step writes a summary node before deleting any leaf and deletes across all tiers at once. That rule addresses the residue trap in Deployment-Time Memorization, where raw-only deletion leaves recoverable traces in about 20% of cases. It also supplies the storage-level forgetting that TiMem's authors say their tree lacks.

Baselines

The naive reference is vLLM-style serving with no contention awareness: the KV cache is swapped or recomputed on pressure, and summaries are never pruned. It shows how much any policy helps at all.

The strongest existing competitor on latency is Oneiros/MIRAGE, which evicts idle models' weights to host memory and leaves KV intact. Its authors report a 44.8%–82.5% reduction in P99 time-between-tokens and 6.6%–86.7% higher throughput than vLLM on ShareGPT, Alpaca and Azure traces. We will rerun it on the same hardware and traces. Beating it means equal or lower P99 time-between-tokens at no loss in memory QA accuracy in regimes where its assumptions weaken: PCIe-class interconnects, and neighbors that are not idle LLMs.

The strongest competitors on which content to shed are attention-based eviction. The first is Finch, which keeps 90% of reference accuracy on SQuAD v2 at 3.76x compression. The second is ImportanceAttention; no reported result was found on Lacuna for ImportanceAttention on LongMemEval or LoCoMo, and its LongBench and RULER results are only given as curves. Both will be given the same memory budget trajectory as our policy, so only the selection rule differs.

The strongest memory-quality reference is TiMem, at 76.88% on LongMemEval-S and 75.30% on LoCoMo in isolation. That is an upper target for accuracy under contention, and we will report how far each policy falls below it. Oracle Belady replacement, which evicts the item needed furthest in the future using the true trace, gives a ceiling.

Two ablations isolate the key ingredient. One replaces the $(m_i, \alpha_i)$ curve with a single learned reuse score trained on the same labels. The other fixes $H$ rather than tracking the contention episode. Every learned component gets the same tuning budget, selected on the same validation split.

Why not just…?

Why not just use attention scores, which Finch and ImportanceAttention already exploit well? This is the main threat. The first experiment below tests it directly. If current-turn attention mass correlates highly with cross-session reuse on held-out gold evidence, the forgetting-curve predictor adds little. In that case we drop it and keep only the horizon and bandwidth coupling. Our reason to expect a gap is the evidence from memorability research. Within LLMs, LongMemEval reports a 30%–60% decline for long-context models relative to oracle evidence, which suggests that what the model attends to is not what later questions need.

Why not just make room elsewhere, as Oneiros does? When an idle model's weights can be moved over a 450 GB/s interconnect, that is cheaper than touching memory. The paper itself notes that its gains shrink on PCIe systems, and it frees nothing when the neighbor is not an LLM whose weights can be evicted. We will run both methods, and their combination, across interconnect classes and neighbor types. We expect Oneiros to win where its assumptions hold, and the combination to beat either alone.

Why not just learn one reuse score, as learned cache-replacement policies do? This is plausible, which is why it is an ablation and not a straw man. The curve's structure should only pay off when episode length varies. With a single score, an item needed in ten minutes and one needed in ten days look alike, although one should stay resident and the other should be moved to host memory. If the single score matches our policy when $H$ varies widely, we will report that the decay parameterization is unnecessary.

Why not just give each tenant a fixed hardware slice? Static partitioning removes neighbor noise, but ParvaGPU measures 20%–50% underutilization from coarse MIG slices. Re-slicing is too slow for bursty agent traffic. We include a fixed per-tenant quota as a baseline to measure this cost.

Proposed Experiments

The decisive first check is the dissociation measurement. On held-out gold evidence from LongMemEval and LoCoMo, we compute the Spearman correlation between write-time attention mass and actual reuse at short delays (the same session) and long delays (later sessions). We compare it with the correlation achieved by the fitted forgetting curve. The analogy predicts moderate correlation at short delays that falls off at long ones, mirroring VideoMem. A flat, high correlation would falsify the premise.

The main experiment co-locates many tenants of an 8B-class model replaying LongMemEval_S/M and LoCoMo conversations. Arrivals follow the ShareGPT and Azure trace shapes. Neighbors are drawn from the iGniter/ParvaGPU vision-and-BERT mix under spatial sharing, run on both a high-bandwidth host link and a PCIe system. Contention episodes are injected with controlled length distributions. We report memory QA accuracy, P99 time-between-tokens, time-to-first-token, and SLO compliance for each baseline. Results are plotted as accuracy against tail latency at matched memory budgets.

We also check tier consistency after consolidation-driven deletion using the canary protocol of Deployment-Time Memorization. Success means our policy sits on or above the accuracy–latency frontier of every baseline in at least the PCIe and non-LLM-neighbor regimes, and closes a clear part of the gap to Belady. It must also keep residue below the raw-only deletion rate.