Online Adaptation of Language Models with a Memory of Amortized Contexts

Jihoon Tack, Jaehyung Kim, Eric Mitchell, Jinwoo Shin, Yee Whye Teh, Jonathan Richard Schwarz

TL;DR

Memory of Amortized Contexts (MAC) is a new framework for keeping Large Language Models (LLMs) up-to-date without the need for expensive retraining or gradient-based online finetuning. By compressing new documents into a memory bank of compact "modulations" via meta-learning, MAC allows a frozen LLM to adapt to new information on-the-fly during inference, achieving superior accuracy, speed, and knowledge retention.

What Problem Does This Paper Address?

Large Language Models are typically static artifacts; their knowledge is frozen at the moment training ends. In a world where information is generated at a breakneck pace, LLMs quickly become outdated.

Current solutions have significant drawbacks:

  1. Retraining from scratch is computationally prohibitive for modern LLMs.
  2. Retrieval-Augmented Generation (RAG) can struggle with counterfactual information and becomes computationally expensive as the number of retrieved documents grows.
  3. Online Finetuning (updating weights on a stream of new data) often leads to catastrophic forgetting, where the model loses previously learned information while trying to incorporate the new. It also requires expensive gradient calculations and is highly sensitive to hyperparameters.

The authors of MAC ask: Can we retain knowledge from incoming documents without needing gradient-based learning at test time?

Methods

MAC bridges the gap between retrieval and finetuning using a "complementary learning systems" approach. The framework consists of three main components:

  • Amortization Network ($\theta_{amort}$): Instead of performing an iterative optimization to find the best parameters for a new document, MAC uses an encoder-decoder (T5-based) hypernetwork. This network takes a new document and, in a single forward pass, predicts a set of modulations (specifically, P-Tuning v2 soft prompts). This process is called "amortization" because it spreads the "cost" of learning across the initial training phase.
  • Memory Bank ($\mathcal{M}$): The compact modulations for every new document are stored in a memory bank. Unlike RAG, which stores raw text, MAC stores the instructions on how to adapt the model.
  • Aggregation Network ($\psi$): When a user asks a question, this network uses cross-attention to scan the memory bank, select relevant modulations, and aggregate them into a single target modulation ($\phi^*$). This aggregated modulation is then fed into the frozen base LM to generate the answer.
Figure 1: An overview of MAC: we amortize each context document into PEFT modulation ϕ and learn to aggregate modulations into a single target modulation ϕ∗based on the given question input x to adapt the frozen LM θbase. During online adaptation, we store the amortized contexts into a memory bank M, then adapt the LM via aggregating the memory bank based on the given question.

To make this scalable, the authors introduced two efficiency tricks:

  1. Backpropagation Dropout: During training, they only compute gradients for a random subset of documents, which drastically reduces memory usage while maintaining performance.
  2. Hierarchical Modulation Aggregation: A divide-and-conquer strategy for inference that subgroups modulations to keep memory costs manageable even with massive memory banks.

Datasets

The researchers evaluated MAC on three key question-answering (QA) benchmarks adapted for sequential document streams:

  • StreamingQA: News articles from 2007 to 2020, requiring the model to adapt to changing facts over time.
  • SQuAD-Seq: A sequential version of the classic Stanford Question Answering Dataset.
  • ArchivalQA-Seq: A large-scale benchmark based on historical news collections.

Key Insights and Ideas

  • Zero-Gradient Adaptation: The most powerful aspect of MAC is that the base LLM remains frozen. Adaptation happens entirely through forward passes of the auxiliary networks. This eliminates the instability and compute cost of online gradient updates.
  • Solving Catastrophic Forgetting: Because the base model isn't being overwritten and documents are stored as discrete modulations in a bank, MAC effectively bypasses the forgetting problem. The model can "remember" an old document simply by attending to its modulation in the bank.
  • Model Soups for Context: The aggregation network essentially creates a "model soup" on the fly. By interpolating between different document modulations based on the question, the model can synthesize information across multiple sources better than picking just one.

Key Results

MAC demonstrated clear superiority over existing online finetuning methods like CaMeLS:

  • Performance: On StreamingQA, MAC improved the F1 score from 18.97% to 21.79%. On SQuAD-Seq, it rose from 18.66% to 21.14%.
  • Efficiency: Compared to the state-of-the-art CaMeLS baseline, MAC reduced memory usage by 68% and adaptation time by 90.31% (reducing a 28.58-minute task to just 2.5 minutes).
  • Knowledge Retention: As shown in the forgetting analysis, MAC retained 96.2% of its initial performance after processing 1,400 additional documents, whereas baselines dropped to ~70%.

Comparison of memory and time efficiency (left) and catastrophic forgetting analysis (right). Catastrophic forgetting analysis showing MAC's superior retention.

Limitations and Future Work

  • Memory Bank Growth: While modulations are compact, the memory bank still grows linearly with the number of documents. The authors suggest that future work could explore neural compression or more advanced "merging" techniques to prune the bank.
  • Privacy: Storing modulations derived from user documents still carries privacy risks. The authors note that learning to "anonymize" these modulations during the amortization process is a promising research direction.
  • Complex Reasoning: While MAC excels at retrieving and applying facts, its ability to perform multi-step reasoning across many amortized contexts remains an open question.

Summary updated

Abstract

Due to the rapid generation and dissemination of information, large language models (LLMs) quickly run out of date despite enormous development costs. To address the crucial need to keep models updated, online learning has emerged as a critical tool when utilizing LLMs for real-world applications. However, given the ever-expanding corpus of unseen documents and the large parameter space of modern LLMs, efficient adaptation is essential. To address these challenges, we propose Memory of Amortized Contexts (MAC), an efficient and effective online adaptation framework for LLMs with strong knowledge retention. We propose a feature extraction and memory-augmentation approach to compress and extract information from new documents into compact modulations stored in a memory bank. When answering questions, our model attends to and extracts relevant knowledge from this memory bank. To learn informative modulations in an efficient manner, we utilize amortization-based meta-learning, which substitutes an otherwise required optimization process with a single forward pass of the encoder. Subsequently, we learn to choose from and aggregate selected documents into a single modulation by conditioning on the question, allowing us to adapt a frozen language model during test time without requiring further gradient updates. Our experiment demonstrates the superiority of MAC in multiple aspects, including online adaptation performance, time, and memory efficiency. In addition, we show how MAC can be combined with and improve the performance of popular alternatives such as retrieval augmented generations (RAGs). Code is available at: https://github.com/jihoontack/MAC.
BibTeX
@inproceedings{tack2024online,
  author = {Jihoon Tack and Jaehyung Kim and Eric Mitchell and Jinwoo Shin and Yee Whye Teh and Jonathan Richard Schwarz},
  title = {{Online Adaptation of Language Models with a Memory of Amortized Contexts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=RIfgKCknTu},
}
Figure