RMem: Bridging Memory Retention and Retrieval via Reversible Compression

Xiaoqiang Wang, Suyuchen Wang, Yun Zhu, Bang Liu

TL;DR

$R^3$Mem bridges the gap between explicit external memory (like vector databases) and implicit parametric memory in LLMs using a reversible Transformer architecture. By treating memory retention as context compression and retrieval as backward context expansion, it stores long histories directly within model parameters while maintaining highly reliable retrieval.

What Problem Does This Paper Address?

Large Language Models (LLMs) are inherently stateless and limited by fixed context windows, making it difficult for them to leverage past experiences in long-horizon tasks or multi-turn interactions. Existing solutions generally fall into two categories, each presenting a tough trade-off:

  1. Explicit Memory: Storing information in external mediums (e.g., vector databases). This offers reliable retrieval but suffers from massive storage overhead and complex management (like merging and purging old data).
  2. Implicit Memory: Encoding information directly into model parameters via fine-tuning or model editing. This is highly compact but suffers from unreliable retrieval, often leading to hallucinations because the model struggles to precisely decode its own parameterized knowledge.
Comparison of explicit, implicit, and $R^3$Mem designs.

The authors introduce $R^3$Mem to achieve the best of both worlds: storing memory compactly within parameters while optimizing for reliable, explicit retrieval.

Methods

$R^3$Mem introduces a duplex network architecture that unifies retention and retrieval through three core components:

  • Virtual Memory Tokens: Trainable summary vectors ($\theta$) are appended to text segments. They act as "read" and "write" tokens that cache compressed representations and propagate them across context windows, allowing the model to scale beyond standard context length limits.
  • Hierarchical Compression: To handle varying semantic granularities, documents are progressively decomposed into document-to-paragraph, paragraph-to-sentence, and sentence-to-entity pairs using an oracle LLM (GPT-4o). This structures the memory from high-level summaries down to atomic facts.
Figure 2: Overview of R3Mem’s architecture: The model employs a reversible framework that integrates context compression and expansion mechanisms. For the forward model, raw textual data is hierarchically encoded into compact representations at various levels—document, paragraph, and entity—using virtual memory tokens. In the backward model, the model reconstructs the original information by reversing the compression process.
  • Reversible Transformer Architecture: Standard Transformers are not mathematically invertible. $R^3$Mem modifies a pre-trained LLaMA 3.1-8B model using adapter-based (LoRA) modifications to make each layer reversible (bijective). The forward pass compresses raw text into memory tokens (Retention), while running the model in reverse reconstructs the exact raw text from those tokens (Retrieval), acting like a neural "zip" and "unzip" process.
Figure 3: The architecture of the reversible Transformer. Left: The general reversible neural architecture. Right: The components of the reversible Transformer.
  • Cycle-Consistent Training: The network is trained bidirectionally using a combination of forward compression loss, backward expansion loss, and a cycle consistency loss to ensure the reconstructed text matches the original input.

Datasets

  • UltraDomain: A diverse long-context dataset spanning question-answering and summarization tasks (NarrativeQA, Qasper, GovReport, MultiNews) used for training and RAG evaluation.
  • Long-Context Benchmarks: PG19 (English books), Pile arXiv (technical papers), and C4 (4K+ web corpus) used to evaluate language modeling perplexity.
  • SiliconFriend: A real-world conversational chatbot dataset containing 10 days of interaction history across 15 virtual users, used to test long-horizon agent memory.

Key Insights and Ideas

  • Memory as a Duplex Network: Retrieval shouldn't just be a search query; it can be modeled as the exact mathematical inverse of compression. By forcing the model to be reversible, retention and retrieval naturally reinforce each other.
  • The Failure of Scaling Token Length: Interestingly, simply increasing the number of virtual memory tokens (memory size) does not significantly improve performance. Memory tokens behave differently than prompt-tuning tokens; they summarize sample-level context rather than stimulating broad pre-trained knowledge, making them easier to overfit if scaled naively.
  • The In-Domain vs. Out-of-Domain Dilemma: Extended training dramatically improves the model's ability to memorize and retrieve in-domain training contexts, but it severely degrades out-of-domain generalization. This reveals a fundamental stability risk in implicit memory: fine-tuning to inject new memories can overwrite or interfere with pre-trained parametric knowledge.

Key Results

  • State-of-the-Art Language Modeling: $R^3$Mem achieved the lowest perplexity across all long-context benchmarks. On the challenging C4 (4K+) dataset, it reduced perplexity to 13.38 (a ~13% reduction over the next-best baseline, MemoRAG at 15.37).
  • Superior Agent Retrieval: When integrated into the SiliconFriend chatbot agent, $R^3$Mem significantly outperformed both explicit memory banks (MemoryBank) and implicit baselines (MemoRAG) in Memory Retrieval Accuracy and Response Correctness.
  • RAG Performance: It demonstrated robust in-domain RAG performance on UltraDomain, showing that better context compression directly translates to superior generation quality.

Limitations and Future Work

  • Pipeline Complexity and Cost: Generating the high-quality hierarchical context-query pairs currently relies on expensive oracle models like GPT-4o. Developing lightweight or adaptive data construction pipelines is a necessary next step.
  • Lifelong Learning Instability: As noted in their training iterations analysis, excessive fine-tuning risks catastrophic forgetting or degradation of the model's core common-sense reasoning. Future work will need to explore more controllable architectures, such as expert network routing mechanisms (MoE), to safely isolate newly integrated memories from core parametric knowledge.

Summary updated

Abstract

Memory plays a key role in enhancing LLMs' performance when deployed to real-world applications. Existing solutions face trade-offs: explicit memory designs based on external storage require complex management and incur storage overhead, while implicit memory designs that store information via parameters struggle with reliable retrieval. In this paper, we propose R$^3$Mem, a memory network that optimizes both information Retention and Retrieval through Reversible context compression. Specifically, R$^3$Mem employs virtual memory tokens to compress and encode infinitely long histories, further enhanced by a hierarchical compression strategy that refines information from document- to entity-level for improved assimilation across granularities. For retrieval, R$^3$Mem employs a reversible architecture, reconstructing raw data by invoking the model backward with compressed information. Implemented via parameter-efficient fine-tuning, it can integrate seamlessly with any Transformer-based model. Experiments demonstrate that our memory design achieves state-of-the-art performance in long-context language modeling and retrieval-augmented generation tasks. It also significantly outperforms conventional memory modules in long-horizon interaction tasks like conversational agents, showcasing its potential for next-generation retrieval systems.
BibTeX
@inproceedings{wang2025rmem,
  author = {Xiaoqiang Wang and Suyuchen Wang and Yun Zhu and Bang Liu},
  title = {{RMem: Bridging Memory Retention and Retrieval via Reversible Compression}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=rUAJuQ8J1B},
}
Figure