MemShift: Separating Detection from Repair in LLM-Derived Personal Memory

Zichuan Xu, Haochen Zhang, Yuyang Dong, Makoto Onizuka, Chuan XiaoarXiv 2026 · arXiv

TL;DR

MemShift is a new benchmark designed to evaluate how AI assistants maintain "derived" personal memories—facts like "important contacts" that are calculated from a user's records. It introduces a crucial distinction between detection (noticing an input has changed) and repair (re-calculating the correct value), revealing that even when AI models notice a change, they often fail to update the information correctly unless they use program-aided reasoning.

What Problem Does This Paper Address?

AI assistants are increasingly used to store synthesized information derived from a user's digital life, such as "my active projects" or "VIP senders." However, these memories often become "stale" or contradictory when the underlying data changes.

The authors identify two major gaps in how we currently evaluate these systems:

  1. Conflated Failure Steps: Existing benchmarks look only at the final answer. If an assistant gives a wrong answer, we don't know if it failed to notice the data changed (detection failure) or if it noticed but couldn't figure out the new correct answer (repair failure).
  2. Text-Centric Updates: Most benchmarks provide updates as new dialogue or text. In reality, many "shifts" happen without new text—for example, a user might change their definition of what an "active project" is, or the assistant might switch to a different underlying LLM.

Methods

The researchers developed MemShift, a benchmark that uses "counterfactual branches." They create synthetic worlds representing 12 months of a person's life. At a specific point, the world forks into branches that are identical except for one of five "shifts":

  • Source Revision: A record is updated (e.g., a meeting is moved).
  • Deletion: A record is removed.
  • Redefinition: The user changes a rule (e.g., "Important contacts now need 3 meetings instead of 5").
  • Writer Switch: The LLM responsible for writing the memory is replaced.
  • Export: The memory is moved to a different product.
Figure 1: Detection and repair are separate steps. (a) One definition item and the five shifts. A source revision and a deletion change records, and a redefinition, a writer switch, and an export add no text. (b) The four HoldMem cells (𝐷, 𝐵) after a redefinition. With a key (𝐵=1), the memory invalidates the item and re-derives it. Without a key (𝐵=0), it keeps the old list and derives a new item only when a new record arrives. The writer derives a wrong list (𝐷=0), and exact recomputation the right one (𝐷=1). Each bar splits the changed definition items as in Equation (1): right (gray at 𝐷=0, blue at 𝐷=1), wrong after a re-derivation (red), and wrong after a missed change (amber). 𝑚: share of changes missed. 𝑟: share of re-derived items that are right.

To validate the benchmark, they built HoldMem, a reference memory system where they can manually toggle detection (using "keys" that invalidate items when inputs change) and repair (using either the LLM's own re-derivation or exact programmatic re-computation). Crucially, MemShift uses a deterministic scorer rather than an LLM judge, ensuring that evaluations are objective and reproducible.

Datasets

The benchmark uses 40 synthetic worlds (20 for the main study, 20 for replication). Each world includes:

  • Six Sources: E-mail, chat, calendar, contacts, tasks, and notes.
  • 48 Derived Items: 42 simple facts (like a contact's city) and 6 "definition items" that require complex aggregation (like "overdue tasks").
  • Scale: Each world contains roughly 1,500–1,650 events and 2.2 million tokens.

Key Insights and Ideas

  • The Detection-Repair Interaction: Detection is only as useful as the repair step allows. If a model is bad at calculating a value, noticing that it needs to be updated doesn't actually improve the final accuracy.
  • LLMs Struggle with Logic, Not Just Memory: Open-weight models (like Llama-3 or Qwen) are surprisingly poor at re-deriving values that require following a rule over many records. They often fail even when they have all the necessary evidence in their context window.
  • Programs are the "Silver Bullet": When LLMs are asked to write a small Python program to calculate a memory item instead of just "guessing" the answer, the accuracy gap between the LLM and perfect re-computation almost disappears.
  • The "Phantom" Problem: Simple citation-based tracking (noticing when a record the model previously cited changes) is insufficient. It misses "phantom" records—new data that should now be included in a set but wasn't there when the memory was first written.

Key Results

  • Repair Failure: On changed definition items, open-weight writers re-derived the correct value at most 25% of the time. In contrast, exact re-computation reached over 91%.
  • The Value of Keys: Using a "key" to detect changes raised accuracy by only +0.2 percentage points when the model re-derived the answer itself, but by +26.9 points when paired with exact re-computation.
  • Model Performance: Stronger models like Claude Sonnet 5.5 performed significantly better, coming close to exact re-computation when given access to all relevant records as typed fields.
  • Writer Switches: When switching models, 97% of correct simple facts survived, but the majority of failures occurred on definition-based items.
Figure 2: 𝐵×𝐷on definition items against the share of programs right before the shift (a) and against 𝑟0 (b). Computing the value helps most when the programs are right (a) and the writer’s own re-derivation is poor (b). (a) The 18 program sets of the 6 writers after a redefinition, one square per set. Gray hulls enclose the sets of Qwen3.5-9B, of gemma-4-12B, and of Llama-3.1-8B and gpt-oss-20b together. Blue line and band: exact recomputation and its 95% interval. (b) 𝐵×𝐷with exact recomputation for three writers of definition items after a redefinition (squares) and a source revision (circles). Bars: 95% intervals.

Limitations and Future Work

  • Synthetic Nature: The worlds are template-generated and definitions are strictly codified. Real-world data is messier, and user definitions are often vaguer.
  • Programmatic Constraints: While programs fixed many issues, the study used a restricted interpreter. Real-world implementations would need to handle security and edge cases in LLM-generated code.
  • Future Directions: The authors suggest future work should focus on memories where the LLM itself decides when to invalidate information, rather than relying on the deterministic keys used in this study's reference system.

Summary updated

Abstract

AI assistants store values that a language model derived from a person's records under the person's own definitions, such as their important contacts. Keeping such a value correct takes two steps: detection, noticing that an input changed, and repair, producing the correct new value. Benchmarks that score only the final answer cannot tell which step failed, and they state every change as new text. Seeing this limitation, we present MemShift, a benchmark of five shifts: source revision, deletion, redefinition, writer switch, and export. The first three fork a synthetic world into branches that differ only in the shift. Gold values and dependencies are fixed before any text, and a deterministic scorer replaces an LLM judge. For a memory that reports its invalidations, MemShift shows for each changed item whether the memory noticed the change and whether its repair was right. Moreover, we build a reference memory with known detection and repair to validate the measure. By evaluating four open-weight LLMs, we find that detection helps only as much as repair allows: the LLMs re-derive few values computed by the person's definitions correctly, and programs that the LLM writes over typed records close most of the gap to exact recomputation. Code and data are available at https://github.com/zichuanxu/memshift.
BibTeX
@misc{xu2026memshift,
  author = {Zichuan Xu and Haochen Zhang and Yuyang Dong and Makoto Onizuka and Chuan Xiao},
  title = {{MemShift: Separating Detection from Repair in LLM-Derived Personal Memory}},
  year = {2026},
  eprint = {2611.02338},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02338},
}
Figure