TL;DR
Memory of Amortized Contexts (MAC) is a new framework for keeping Large Language Models (LLMs) up-to-date without the need for expensive retraining or gradient-based online finetuning. By compressing new documents into a memory bank of compact "modulations" via meta-learning, MAC allows a frozen LLM to adapt to new information on-the-fly during inference, achieving superior accuracy, speed, and knowledge retention.
What Problem Does This Paper Address?
Large Language Models are typically static artifacts; their knowledge is frozen at the moment training ends. In a world where information is generated at a breakneck pace, LLMs quickly become outdated.
Current solutions have significant drawbacks:
- Retraining from scratch is computationally prohibitive for modern LLMs.
- Retrieval-Augmented Generation (RAG) can struggle with counterfactual information and becomes computationally expensive as the number of retrieved documents grows.
- Online Finetuning (updating weights on a stream of new data) often leads to catastrophic forgetting, where the model loses previously learned information while trying to incorporate the new. It also requires expensive gradient calculations and is highly sensitive to hyperparameters.
The authors of MAC ask: Can we retain knowledge from incoming documents without needing gradient-based learning at test time?
Methods
MAC bridges the gap between retrieval and finetuning using a "complementary learning systems" approach. The framework consists of three main components:
- Amortization Network ($\theta_{amort}$): Instead of performing an iterative optimization to find the best parameters for a new document, MAC uses an encoder-decoder (T5-based) hypernetwork. This network takes a new document and, in a single forward pass, predicts a set of modulations (specifically, P-Tuning v2 soft prompts). This process is called "amortization" because it spreads the "cost" of learning across the initial training phase.
- Memory Bank ($\mathcal{M}$): The compact modulations for every new document are stored in a memory bank. Unlike RAG, which stores raw text, MAC stores the instructions on how to adapt the model.
- Aggregation Network ($\psi$): When a user asks a question, this network uses cross-attention to scan the memory bank, select relevant modulations, and aggregate them into a single target modulation ($\phi^*$). This aggregated modulation is then fed into the frozen base LM to generate the answer.
To make this scalable, the authors introduced two efficiency tricks:
- Backpropagation Dropout: During training, they only compute gradients for a random subset of documents, which drastically reduces memory usage while maintaining performance.
- Hierarchical Modulation Aggregation: A divide-and-conquer strategy for inference that subgroups modulations to keep memory costs manageable even with massive memory banks.
Datasets
The researchers evaluated MAC on three key question-answering (QA) benchmarks adapted for sequential document streams:
- StreamingQA: News articles from 2007 to 2020, requiring the model to adapt to changing facts over time.
- SQuAD-Seq: A sequential version of the classic Stanford Question Answering Dataset.
- ArchivalQA-Seq: A large-scale benchmark based on historical news collections.
Key Insights and Ideas
- Zero-Gradient Adaptation: The most powerful aspect of MAC is that the base LLM remains frozen. Adaptation happens entirely through forward passes of the auxiliary networks. This eliminates the instability and compute cost of online gradient updates.
- Solving Catastrophic Forgetting: Because the base model isn't being overwritten and documents are stored as discrete modulations in a bank, MAC effectively bypasses the forgetting problem. The model can "remember" an old document simply by attending to its modulation in the bank.
- Model Soups for Context: The aggregation network essentially creates a "model soup" on the fly. By interpolating between different document modulations based on the question, the model can synthesize information across multiple sources better than picking just one.
Key Results
MAC demonstrated clear superiority over existing online finetuning methods like CaMeLS:
- Performance: On StreamingQA, MAC improved the F1 score from 18.97% to 21.79%. On SQuAD-Seq, it rose from 18.66% to 21.14%.
- Efficiency: Compared to the state-of-the-art CaMeLS baseline, MAC reduced memory usage by 68% and adaptation time by 90.31% (reducing a 28.58-minute task to just 2.5 minutes).
- Knowledge Retention: As shown in the forgetting analysis, MAC retained 96.2% of its initial performance after processing 1,400 additional documents, whereas baselines dropped to ~70%.

Limitations and Future Work
- Memory Bank Growth: While modulations are compact, the memory bank still grows linearly with the number of documents. The authors suggest that future work could explore neural compression or more advanced "merging" techniques to prune the bank.
- Privacy: Storing modulations derived from user documents still carries privacy risks. The authors note that learning to "anonymize" these modulations during the amortization process is a promising research direction.
- Complex Reasoning: While MAC excels at retrieving and applying facts, its ability to perform multi-step reasoning across many amortized contexts remains an open question.
Summary updated