Causal Memory Policy: Making Memory Utility Identifiable by Intervening on Retrieval

Arman Behnam, Binghui WangarXiv 2026 · arXiv

TL;DR

Memory-augmented LLMs often fail to identify which memories are actually useful because their retrieval systems are deterministic—if a memory isn't retrieved, its utility remains invisible. This paper introduces the Causal Memory Policy (CMP), which uses randomized retrieval to restore "identifiability," significantly improving the model's ability to distinguish between essential information and noise.

What Problem Does This Paper Address?

As Large Language Models (LLMs) are deployed for longer interactions, they rely on external memory stores to bypass fixed context windows. A critical design challenge is deciding what to keep and what to evict when memory fills up.

Current systems try to estimate "memory utility" by looking at how often a memory contributes to a successful answer. However, the authors identify a fundamental flaw: retrieval-level positivity violation. In causal inference terms, you can only measure the effect of a "treatment" (a memory) if there's a non-zero probability of the model seeing it. If a deterministic retriever never surfaces a specific memory, the system can never observe its utility.

This creates a "dead zone" where identification fails. The system might mistake a crucial but unretrieved memory for a useless one, leading to "destructive eviction"—permanently forgetting information that was actually necessary.

Figure 1: Causal DAG of memory-augmented LLMs across two timesteps.

Methods

The authors propose the Causal Memory Policy (CMP), a framework designed to make memory utility identifiable by intervening directly on the retrieval process.

  1. Randomized Retrieval Exposure: Instead of letting the ranker fill the entire context window, CMP reserves a fixed number of slots ($k$) for memories sampled randomly from the store with known probabilities (propensities).
  2. Balanced Assignment: To ensure every memory is tested fairly, CMP uses a "balanced round-robin" design. This guarantees that over a horizon of queries, each candidate memory is included in the context a specific number of times.
  3. Utility Estimation: CMP uses a self-normalized inverse propensity weighting (the Hájek estimator) to calculate utility. Because the sampling probabilities are known by design, the estimator is unbiased.
  4. Bayes-Optimal Decision Rule: The authors derive a specific rule for forgetting. Since forgetting is irreversible (you can't re-estimate the utility of a deleted memory) but keeping is reversible, the optimal policy is "one-sided." The system only evicts a memory if it is confidently negative or useless, otherwise, it defaults to keeping it.
Figure 2: Overview of CMP. (a) Balanced exposure assigns the Tk exposure slots evenly across the exposure pool, fixing each memory’s propensity by design. (b) At each timestep, the context consists of the top-(B−k) retrieved memories together with k randomly exposed memories. (c) CMP compares three designs that differ only in the randomized variable while using the same estimator.

Datasets

The authors evaluated CMP across several challenging benchmarks to test different aspects of memory:

  • LongMemEval: A benchmark for long-horizon conversational memory (histories of ~490 turns).
  • LoCoMo: A multi-session dialogue benchmark where the same memory might be relevant to multiple different queries.
  • HotpotQA & MuSiQue: Used to test multi-hop reasoning and retrieval.
  • Mem0: A real-world, deployed memory system used to demonstrate that these identification failures persist in production environments.

Key Insights and Ideas

  • Retrieval is a Mediator: The paper formalizes the memory system as a Structural Causal Model (SCM). The "aha moment" is realizing that the memory state only affects the output through the retriever. If the retriever fails, the causal chain is broken.
  • The "Zero Utility" Trap: Without randomization, an unretrieved memory produces a utility estimate near zero. This makes "identification failure" indistinguishable from "genuinely low utility," which is dangerous for eviction policies.
  • Identification is Not Free: Randomizing retrieval means occasionally displacing a "high-rank" memory for a random one. This can slightly lower immediate task performance (a "cost of exposure") to gain the information needed for better long-term memory management.
  • Query-Specific Utility: A major finding is that memory utility is highly dependent on the specific query. A memory that is vital for one question might be useless for the next 29. This suggests that a single "utility score" per memory is insufficient for a perfect retention policy.

Key Results

The empirical results highlight how widespread this problem is:

  • High Failure Rates: On the LongMemEval and LoCoMo benchmarks, 54% and 67% of required memories, respectively, were never surfaced by standard retrievers.
  • Improved Discrimination: CMP improved the ability to distinguish between required and non-required memories, raising the Area Under the Curve (AUC) from 0.54 (near random) to 0.66.
  • Eviction Quality: CMP halved the "gold loss" (the rate of accidentally deleting required memories) compared to standard store-level randomization methods.
Table 2: Estimation and decision quality on two designs (only differ in randomized vari- able) and LongMemEval independent subset.

Limitations and Future Work

  • Pool Selection: CMP requires a "pool" of memories to randomize. If the total memory store is massive (millions of items), you can't randomize everything. Finding a "pool selector" that captures the memories that might be useful remains an open challenge.
  • Aggregation: While CMP identifies utility for a specific query (reaching 0.78 AUC), it struggles to predict if a memory will be useful for future, unseen queries. The authors note that no current aggregation method (like taking the mean or max utility) effectively solves this.
  • Write-Time Policies: The paper focuses on what to forget (eviction). However, in systems like Mem0, the authors found that the way memories are written (e.g., accumulating contradictory facts) also limits performance, suggesting a need for better "write-time" causal policies.

Summary updated

Abstract

Memory-augmented large language models must decide which memories to retain, and recent systems do so by estimating each memory's effect on task performance. However, these estimates rely entirely on retrieved memories. When a memory is never retrieved, store-level interventions produce identical outcomes, leaving its utility unidentified. This is a retrieval-level positivity violation, invisible to diagnostics that examine only memory operations. We introduce Causal Memory Policy (CMP), a causal framework that restores identification by intervening on retrieval itself, reserving a fixed number of context slots for memories sampled with known propensities. CMP estimates memory utility by self-normalized inverse propensity weighting under a balanced assignment design. We prove the causal factorization of memory utility through retrieval, the unbiasedness and exact variance of the estimator, and the optimal decision rule under irreversible operations. Empirically, identification fails for 54% of required memories on LongMemEval and 67% on LoCoMo, and the failure persists in a deployed memory system. CMP improves discrimination between required and non-required memories from 0.54 to 0.66 AUC. Finally, we show that identified memory utility alone is insufficient for retention decisions: per-query utility reaches 0.78 AUC on the query for which it is estimated, yet no aggregation available to a retention policy predicts a memory's value on unseen queries. Code is available at: https://anonymous.4open.science/r/cmp-release-D0C3/.
BibTeX
@misc{behnam2026causal,
  author = {Arman Behnam and Binghui Wang},
  title = {{Causal Memory Policy: Making Memory Utility Identifiable by Intervening on Retrieval}},
  year = {2026},
  eprint = {2610.02070},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.02070},
}
Figure