TL;DR
LEDGER is a fact-centric memory layer for LLM agents that uses explicit, deterministic rules to manage changing information, source reliability, and data deletion. By moving the logic for "what is true now" out of the model's context and into a structured ledger, it achieves higher accuracy than full-history replay while using only 1% of the tokens.
What Problem Does This Paper Address?
Most current LLM agent memory systems (like RAG or full-history replay) are evaluated on static histories. In the real world, facts are dynamic: a project's database might migrate from Postgres 14 to 16, a README might become stale, or a user might ask the agent to "forget" a specific API key.
Standard approaches struggle here. Full-history replay sends the entire past to the model, forcing it to decide which of several conflicting values is current. RAG (Retrieval-Augmented Generation) might retrieve an outdated turn while missing the update. Neither handles "deletion" well; even if a fact is removed from a database, it often remains "resurrected" in the raw conversation transcripts used for retrieval. The authors aim to build a memory system that is auditable, source-aware, and deletion-compliant.
Methods
LEDGER moves the decision-making process for memory management from the LLM's internal reasoning into a structured Ledger Core.
- Extraction & Routing: An LLM extracts "claims" (entity/attribute/value) from events. A router then maps questions or deletion requests to specific "slots" in the memory.
- The Ledger Core: Instead of just storing text, the core applies five fixed write rules—restate, supersede, dispute, introduce, hold—to every incoming claim. These rules use timestamps and a Source Trust Policy (e.g., User > Document > Assistant guess) to decide if a value is current or contested.
- Temporal Tracking: Every fact is stored with a validity interval. This allows the agent to answer questions about the current state or "as of" a past date.
- LEDGER+EP: This variant adds an "Episode Store" for raw conversation turns. It serves as a fallback for questions the structured ledger can't answer. Crucially, when a fact is deleted, LEDGER+EP "scrubs" all raw turns in the Episode Store that mention that fact.
Datasets
The authors evaluated their system using two primary workloads:
- Coding-Agent Workload: A newly introduced synthetic benchmark simulating a multi-user, multi-session coding environment. It features "planted" ground truth with controlled updates, conflicting sources (e.g., a stale README vs. a user message), and explicit deletion requests.
- LongMemEval: A 210-question subset of an existing benchmark for long-term memory in chat assistants.
Key Insights and Ideas
- The Evidence-Answer Gap: A key finding is that LLMs are surprisingly good at "repairing" noisy evidence. Even if a memory system sends stale data, the model might still guess the right answer. However, this repair fails as history grows, and it doesn't solve the privacy/deletion problem—the model still sees the forbidden data.
- Rules Over Reasoning: By using deterministic rules for ingestion (see Figure 6), the system avoids the "hallucination" of old facts. If a trusted source says a value has changed, the old value is closed off and never sent as "current" evidence again.
- Scrubbing is Essential: Deletion is a major challenge for RAG. The paper demonstrates that simply deleting a record isn't enough; you must also scrub the raw transcripts to prevent the model from "remembering" deleted info through retrieval.
Key Results
- Superior Accuracy: On the coding workload, LEDGER+EP achieved 94.3% accuracy, significantly outperforming the strongest baseline, full-history replay (82.1%).
- Extreme Efficiency: On LongMemEval, LEDGER+EP matched the accuracy of full-history replay (74.8%) while using only 1% of the context tokens (approx. 1,000 vs 100,000+).
- Robustness to Growth: As shown in Figure 2, as the conversation history grows, the accuracy of standard RAG and full replay drops significantly due to "forbidden" (stale or conflicting) evidence. LEDGER's accuracy remains stable because its evidence stays clean.
- Perfect Deletion: In all evaluated probes, LEDGER+EP successfully prevented 100% of deleted values from reaching the model's context.
Limitations and Future Work
The primary limitation is that the coding workload is synthetic and template-based, which may not capture the full linguistic complexity of real-world developer interactions. Additionally, the study focuses on question-answering rather than measuring the agent's success on downstream tasks (like actually fixing a bug).
Future work could explore how these structured memory rules perform with different model families and in more diverse, non-coding domains. The authors also suggest that "scrubbing" raw turns might lead to a loss of useful context, a trade-off that warrants further measurement.
Summary updated