LEDGER: Auditable Agent Memory Under Change

Md Aidul Islam, Md Nasir Uddin Shuvo, Muhammad Waseem, Pekka AbrahamssonarXiv 2026 · arXiv

TL;DR

LEDGER is a fact-centric memory layer for LLM agents that uses explicit, deterministic rules to manage changing information, source reliability, and data deletion. By moving the logic for "what is true now" out of the model's context and into a structured ledger, it achieves higher accuracy than full-history replay while using only 1% of the tokens.

What Problem Does This Paper Address?

Most current LLM agent memory systems (like RAG or full-history replay) are evaluated on static histories. In the real world, facts are dynamic: a project's database might migrate from Postgres 14 to 16, a README might become stale, or a user might ask the agent to "forget" a specific API key.

Standard approaches struggle here. Full-history replay sends the entire past to the model, forcing it to decide which of several conflicting values is current. RAG (Retrieval-Augmented Generation) might retrieve an outdated turn while missing the update. Neither handles "deletion" well; even if a fact is removed from a database, it often remains "resurrected" in the raw conversation transcripts used for retrieval. The authors aim to build a memory system that is auditable, source-aware, and deletion-compliant.

Methods

LEDGER moves the decision-making process for memory management from the LLM's internal reasoning into a structured Ledger Core.

  1. Extraction & Routing: An LLM extracts "claims" (entity/attribute/value) from events. A router then maps questions or deletion requests to specific "slots" in the memory.
  2. The Ledger Core: Instead of just storing text, the core applies five fixed write rules—restate, supersede, dispute, introduce, hold—to every incoming claim. These rules use timestamps and a Source Trust Policy (e.g., User > Document > Assistant guess) to decide if a value is current or contested.
  3. Temporal Tracking: Every fact is stored with a validity interval. This allows the agent to answer questions about the current state or "as of" a past date.
  4. LEDGER+EP: This variant adds an "Episode Store" for raw conversation turns. It serves as a fallback for questions the structured ledger can't answer. Crucially, when a fact is deleted, LEDGER+EP "scrubs" all raw turns in the Episode Store that mention that fact.
The LEDGER+EP architecture showing how events are processed into structured facts and raw turns, with explicit rules governing the Ledger Core.

Datasets

The authors evaluated their system using two primary workloads:

  • Coding-Agent Workload: A newly introduced synthetic benchmark simulating a multi-user, multi-session coding environment. It features "planted" ground truth with controlled updates, conflicting sources (e.g., a stale README vs. a user message), and explicit deletion requests.
  • LongMemEval: A 210-question subset of an existing benchmark for long-term memory in chat assistants.

Key Insights and Ideas

  • The Evidence-Answer Gap: A key finding is that LLMs are surprisingly good at "repairing" noisy evidence. Even if a memory system sends stale data, the model might still guess the right answer. However, this repair fails as history grows, and it doesn't solve the privacy/deletion problem—the model still sees the forbidden data.
  • Rules Over Reasoning: By using deterministic rules for ingestion (see Figure 6), the system avoids the "hallucination" of old facts. If a trusted source says a value has changed, the old value is closed off and never sent as "current" evidence again.
  • Scrubbing is Essential: Deletion is a major challenge for RAG. The paper demonstrates that simply deleting a record isn't enough; you must also scrub the raw transcripts to prevent the model from "remembering" deleted info through retrieval.
Table of desired memory properties and the errors they prevent.

Key Results

  • Superior Accuracy: On the coding workload, LEDGER+EP achieved 94.3% accuracy, significantly outperforming the strongest baseline, full-history replay (82.1%).
  • Extreme Efficiency: On LongMemEval, LEDGER+EP matched the accuracy of full-history replay (74.8%) while using only 1% of the context tokens (approx. 1,000 vs 100,000+).
  • Robustness to Growth: As shown in Figure 2, as the conversation history grows, the accuracy of standard RAG and full replay drops significantly due to "forbidden" (stale or conflicting) evidence. LEDGER's accuracy remains stable because its evidence stays clean.
  • Perfect Deletion: In all evaluated probes, LEDGER+EP successfully prevented 100% of deleted values from reaching the model's context.
Performance over time: LEDGER maintains high accuracy and low "forbidden evidence" even as history grows, unlike RAG or full replay.

Limitations and Future Work

The primary limitation is that the coding workload is synthetic and template-based, which may not capture the full linguistic complexity of real-world developer interactions. Additionally, the study focuses on question-answering rather than measuring the agent's success on downstream tasks (like actually fixing a bug).

Future work could explore how these structured memory rules perform with different model families and in more diverse, non-coding domains. The authors also suggest that "scrubbing" raw turns might lead to a loss of useful context, a trade-off that warrants further measurement.

Summary updated

Abstract

Agent memory systems are mostly evaluated on fixed histories, and only their final answers are scored. In real use, facts change, sources differ in reliability, and users ask the agent to forget things. We present LEDGER, a memory layer in which a language model proposes facts and explicit rules decide which value is current, which source to trust, and what to delete. Each fact keeps its source and the period in which it held. LEDGER+EP adds a search over raw conversation turns for questions the ledger does not cover, and scrubs these turns on deletion. We compare both against no memory, full-history replay, retrieval over raw transcripts, and Mem0, with the same answering model and prompt, on a synthetic multi-session coding-agent workload and a 210-question subset of LongMemEval, and we score both the evidence each memory sends to the model and the answer. On the coding workload, LEDGER+EP in fallback mode answers 94.3% of questions correctly on five development seeds and 91.1% on five held-out seeds, against 82.1% and 83.3% for full replay, the strongest baseline. When a deletion request only describes the fact, no deleted value reached our systems' evidence in any evaluated probe. On LongMemEval, LEDGER+EP in augment mode has the same observed accuracy as full replay (74.8%) with 1% of its context. The main limitation is that the coding workload is synthetic and built by us; on LongMemEval, the differences between the strongest systems are within the confidence intervals.
BibTeX
@misc{islam2026ledger,
  author = {Md Aidul Islam and Md Nasir Uddin Shuvo and Muhammad Waseem and Pekka Abrahamsson},
  title = {{LEDGER: Auditable Agent Memory Under Change}},
  year = {2026},
  eprint = {2611.20927},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20927},
}
Figure