KV-LTM: Persistent Per-Session KV Caches as Long-Term Memory for Personal Assistants

Ziv Kfir, Ido KfirarXiv 2026 · arXiv

TL;DR

KV-LTM proposes a new long-term memory (LTM) tier for personal assistants that stores the raw text of every conversation session along with its corresponding KV cache. By treating the KV cache as a "recompute-free prefill accelerator," the system avoids the information loss common in summarization and retrieval-based memory while significantly speeding up the processing of long histories.

What Problem Does This Paper Address?

Personal assistants (PAs) accumulate months of interactions, but current memory systems struggle to recall specific details accurately. Most existing LTM stacks rely on one of two strategies:

  1. Write-time derivation: Summarizing history or extracting facts into a database (e.g., Mem0). This often loses nuances, wording, and rationale.
  2. Read-time selection: Using RAG (Retrieval-Augmented Generation) to pick relevant snippets. This often fails to find the "needle in the haystack" if the retrieval index is imperfect.

The authors highlight a "content loss" gap: commercial assistants often perform significantly worse when using their memory pipelines compared to a model reading the entire history offline.

Figure 1: Performance gap between offline full-history reading (L3) and online memory systems (L2).

Methods

The core proposal is KV-LTM, a persistent memory tier that stores:

  • The raw session text: The "memory of record."
  • The native KV cache state: The internal activations of a fixed "reader" model.

When a user asks a question, the system restores the cached state of relevant sessions and appends the new question. This avoids the "prefill" compute cost—the most expensive part of processing long contexts—while ensuring the model sees the original, un-summarized evidence.

Key technical components include:

  • The Exactness Lemma: A formal statement that restoring a cache and appending a suffix is mathematically identical to prefilling the whole sequence from scratch, provided the model and prefix remain identical.
  • Budget-Derived Retention: Since KV caches are large, the system uses a storage and latency budget to decide which sessions to keep in the cache tier. If a session is evicted from the cache, the system "degrades gracefully" to reading the raw text.
  • Bounded Aggregation: A protocol for taking short answers from multiple independent session reads and synthesizing them into a final response.

Datasets

The paper defines an evaluation protocol to test this logic on existing long-term memory benchmarks without requiring a full persistent-cache implementation. They use:

  • LongMemEval: For testing multi-session reasoning.
  • LoCoMo: For long-context conversation modeling.
  • PersonaMem, MemoryAgentBench, and BEAM: To cover persona consistency and agentic reasoning.

Key Insights and Ideas

  • Cache as an Accelerator, Not Just Storage: The authors frame the KV cache not as a new way to store information, but as a way to trade storage space for avoided compute.
  • The Impact of Model Architecture: The feasibility of this approach depends heavily on how the model handles attention. Modern architectures like MLA (Multi-Head Latent Attention) used in DeepSeek-V2/V3 drastically reduce the size of the KV cache, making persistent storage much more affordable.
Table 1: Comparison of KV compression mechanisms and their suitability for KV-LTM.
  • Fidelity Over Compression: Unlike summarization, which tries to guess what is important at "write-time," KV-LTM preserves everything, deferring the interpretation of what's relevant to "read-time."

Key Results

The paper provides an extensive analytical cost model rather than just accuracy scores:

  • Storage Requirements: For an 8B parameter model (GQA), a year of conversation (at 30K tokens/day) requires ~1.44 TB of storage. For models using MLA, this drops to ~0.34 TB.
  • The Storage-to-Text Ratio: KV state is massive compared to raw text—roughly 29,000 times larger for standard 8B models.
  • Latency Trade-offs: At a transfer speed of 5 GB/s, loading 20 days of history (~79 GB of state) takes about 15.7 seconds. This motivates the use of a "hybrid" mode where a lightweight retriever selects which caches to load.

Limitations and Future Work

  • Model Version Lock-in: A stored KV cache is only valid for the specific model version that created it. Updating the model requires re-prefilling the entire history from the raw text.
  • Cross-Session Synthesis: Because sessions are read independently, the model cannot "attend" across different sessions simultaneously. This requires a robust aggregation step or a secondary "joint-read" tier for complex questions.
  • Privacy and Data Protection: Storing internal model states alongside text increases the "data surface" for potential leaks, requiring careful consideration of deletion and redaction (e.g., GDPR compliance).
  • Session Bounding: The system assumes interactions can be cleanly segmented into "sessions," which is a difficult problem in itself.

Summary updated

Abstract

Personal assistants accumulate months of conversations, but today's long-term memory stacks put derivation or selection between a question and its evidence. In a system-level comparison on LongMemEval, two commercial assistants scored well below the same model reading the history offline [1], consistent with, but not isolating, content loss between write and read. We propose KV-LTM: a memory tier storing each session's raw text plus its KV cache under a fixed reader model, answering by restoring cached sessions and appending the question. KV-LTM is complementary to, not a replacement for, retrieval and indexing. Text is the memory of record; the cache is a recompute-free prefill accelerator, degrading gracefully to text. We specify the design: storage record, budget-derived retention, append-only read path, and bounded aggregation. We size it analytically: across pinned BF16 full-attention (GQA) and latent-cached MLA configurations, per-token state spans approximately 31K-328K bytes (the pinned KDA+MLA hybrid differs: approximately 8.1K bytes/token plus a fixed per-session recurrent term), giving approximately 0.34-3.6 TB per user-year at a declared 30K-token/day scenario, against approximately 49 MB of raw text; per-query cost is modelled as regime-conditional transfer, attention, decode, aggregation, and compute. We define an evaluation protocol needing no persistent-cache implementation for logical QA evaluation: per-session prefill emulates valid cache reuse under the same model/input-state contract and ideal arithmetic, with named arms, controlled baselines, and a preregistered reader-model gate; a separate minimal real-cache replay tests numerical equivalence and measures the cost axis, with outcomes not assumed. KV-LTM trades storage for avoided re-prefill compute and write-time fidelity, within a declared retention budget.
BibTeX
@misc{kfir2026kv,
  author = {Ziv Kfir and Ido Kfir},
  title = {{KV-LTM: Persistent Per-Session KV Caches as Long-Term Memory for Personal Assistants}},
  year = {2026},
  eprint = {2611.21998},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21998},
}
Figure