TL;DR
MemShift is a new benchmark designed to evaluate how AI assistants maintain "derived" personal memories—facts like "important contacts" that are calculated from a user's records. It introduces a crucial distinction between detection (noticing an input has changed) and repair (re-calculating the correct value), revealing that even when AI models notice a change, they often fail to update the information correctly unless they use program-aided reasoning.
What Problem Does This Paper Address?
AI assistants are increasingly used to store synthesized information derived from a user's digital life, such as "my active projects" or "VIP senders." However, these memories often become "stale" or contradictory when the underlying data changes.
The authors identify two major gaps in how we currently evaluate these systems:
- Conflated Failure Steps: Existing benchmarks look only at the final answer. If an assistant gives a wrong answer, we don't know if it failed to notice the data changed (detection failure) or if it noticed but couldn't figure out the new correct answer (repair failure).
- Text-Centric Updates: Most benchmarks provide updates as new dialogue or text. In reality, many "shifts" happen without new text—for example, a user might change their definition of what an "active project" is, or the assistant might switch to a different underlying LLM.
Methods
The researchers developed MemShift, a benchmark that uses "counterfactual branches." They create synthetic worlds representing 12 months of a person's life. At a specific point, the world forks into branches that are identical except for one of five "shifts":
- Source Revision: A record is updated (e.g., a meeting is moved).
- Deletion: A record is removed.
- Redefinition: The user changes a rule (e.g., "Important contacts now need 3 meetings instead of 5").
- Writer Switch: The LLM responsible for writing the memory is replaced.
- Export: The memory is moved to a different product.
To validate the benchmark, they built HoldMem, a reference memory system where they can manually toggle detection (using "keys" that invalidate items when inputs change) and repair (using either the LLM's own re-derivation or exact programmatic re-computation). Crucially, MemShift uses a deterministic scorer rather than an LLM judge, ensuring that evaluations are objective and reproducible.
Datasets
The benchmark uses 40 synthetic worlds (20 for the main study, 20 for replication). Each world includes:
- Six Sources: E-mail, chat, calendar, contacts, tasks, and notes.
- 48 Derived Items: 42 simple facts (like a contact's city) and 6 "definition items" that require complex aggregation (like "overdue tasks").
- Scale: Each world contains roughly 1,500–1,650 events and 2.2 million tokens.
Key Insights and Ideas
- The Detection-Repair Interaction: Detection is only as useful as the repair step allows. If a model is bad at calculating a value, noticing that it needs to be updated doesn't actually improve the final accuracy.
- LLMs Struggle with Logic, Not Just Memory: Open-weight models (like Llama-3 or Qwen) are surprisingly poor at re-deriving values that require following a rule over many records. They often fail even when they have all the necessary evidence in their context window.
- Programs are the "Silver Bullet": When LLMs are asked to write a small Python program to calculate a memory item instead of just "guessing" the answer, the accuracy gap between the LLM and perfect re-computation almost disappears.
- The "Phantom" Problem: Simple citation-based tracking (noticing when a record the model previously cited changes) is insufficient. It misses "phantom" records—new data that should now be included in a set but wasn't there when the memory was first written.
Key Results
- Repair Failure: On changed definition items, open-weight writers re-derived the correct value at most 25% of the time. In contrast, exact re-computation reached over 91%.
- The Value of Keys: Using a "key" to detect changes raised accuracy by only +0.2 percentage points when the model re-derived the answer itself, but by +26.9 points when paired with exact re-computation.
- Model Performance: Stronger models like Claude Sonnet 5.5 performed significantly better, coming close to exact re-computation when given access to all relevant records as typed fields.
- Writer Switches: When switching models, 97% of correct simple facts survived, but the majority of failures occurred on definition-based items.
Limitations and Future Work
- Synthetic Nature: The worlds are template-generated and definitions are strictly codified. Real-world data is messier, and user definitions are often vaguer.
- Programmatic Constraints: While programs fixed many issues, the study used a restricted interpreter. Real-world implementations would need to handle security and edge cases in LLM-generated code.
- Future Directions: The authors suggest future work should focus on memories where the LLM itself decides when to invalidate information, rather than relying on the deterministic keys used in this study's reference system.
Summary updated