FastMem: Fast Memorization of Prompt Improves Context Awareness of Large Language Models

Junyi Zhu, Shuochen Liu, Yu Yu, Bo Tang, Yibo Yan, Zhiyu Li, Feiyu Xiong, Tong Xu, Matthew B. Blaschko

TL;DR

FastMem is a lightweight, inference-time technique that improves an LLM's context awareness by "memorizing" the prompt right before generating a response. By fine-tuning only the last Feed-Forward Network (FFN) module on the provided context, the model significantly reduces its internal confusion (perplexity), leading to much higher accuracy in tasks like RAG and structured data extraction.

What Problem Does This Paper Address?

Large Language Models (LLMs) are incredibly knowledgeable, but they are also stubborn. When you provide a prompt with specific context—such as a document for Retrieval-Augmented Generation (RAG)—the model often faces a "knowledge conflict." If its pre-trained "parametric" knowledge contradicts the provided text, it might ignore the context and hallucinate based on what it learned during training.

For example, if an LLM "knows" a celebrity works at Google, but your prompt includes a recent news article saying they resigned, the model might still insist they are at Google. This lack of context awareness makes LLMs unreliable for applications requiring strict adherence to provided data or specific output formats.

Figure 1: An illustrative example where an LLM cannot follow the reference due to its conflict with the LLM’s prior knowledge (a). In such cases, LLM often exhibits high perplexity on the conflicted information. Our pro- posed FastMem addresses this issue by enabling the model to memorize the reference text (thereby reducing perplexity) before conducting inference (b).

Methods

The authors propose FastMem, a "test-time training" approach that happens in the seconds between receiving a prompt and generating an answer.

  1. Memorization Phase: Before inference, the model performs a few iterations of training on the prompt's reference text. It uses a standard next-token prediction objective to maximize the likelihood of the prompt.
  2. Targeted Updates: To keep this process fast and prevent the model from "breaking" its general abilities, FastMem only updates the parameters of the very last Feed-Forward Network (FFN) module in the model. All other layers remain frozen.
  3. Regularization: To ensure the model doesn't overfit to a tiny snippet of text, they use a Kullback-Leibler (KL) divergence loss to keep the updated model's behavior close to the original.
  4. Control Tokens: They use specific chat templates and control tokens (like <|eot|>) during this mini-training phase so the model maintains its identity as an assistant and doesn't just start mindlessly repeating the prompt.
Figure 2: Overall frameworks of our proposed FastMem and its integration with decoding strategies.

Datasets

The researchers evaluated FastMem across several challenging benchmarks:

  • NQ-SWAP: A modified version of Natural Questions where entity names are swapped to create intentional knowledge conflicts.
  • MemoTrap: A dataset designed to "trap" models into following common proverbs or patterns instead of specific, counter-intuitive instructions.
  • Standard Q&A: Natural Questions (NQ) and HotpotQA.
  • Summarization: CNN-DM, XSUM, and WikiHow.

Key Insights and Ideas

  • The Perplexity Connection: The authors discovered a strong negative correlation between the model's perplexity on a context and its accuracy. If a model finds the context "surprising" (high perplexity), it is much more likely to fail. FastMem works by explicitly lowering this perplexity.
  • Why the Last FFN? The FFN layers are known to act as "key-value memories." Updating only the last one is computationally efficient (it's basically a small MLP update) and provides the best balance between learning new context and preserving general language skills.
  • Efficiency is Key: Because only the last FFN is tuned, the hidden states from all previous layers can be cached. This makes the "training" phase extremely fast—taking only about 1.6 seconds for an 8B model, which is roughly the time it takes to generate 20 tokens.
Figure 3: Model performance and perplexity on NQ and NQ-SWAP datasets.

Key Results

FastMem consistently outperformed standard zero-shot and few-shot baselines:

  • Knowledge Conflicts: On NQ-SWAP, FastMem boosted Llama 3-8B-Instruct accuracy from 59.1% to 71.6%.
  • Instruction Following: On MemoTrap, it improved performance from 77.9% to 97.2%.
  • Structured Output: For smaller models like Qwen 1.5-4B, which often struggle with formatting, FastMem reduced the output structure failure rate from 34.9% to 25.5%.
  • Compatibility: FastMem can be combined with other decoding strategies like Contrastive Decoding (CD) or DoLa for even further gains, particularly in summarization tasks.
Table 1: Results of Q&A tasks using Llama 3-8B- Instruct. The best results are in bold, and the second-best results are underlined.

Limitations and Future Work

While promising, FastMem has a few hurdles to clear:

  • Noisy Context: The method assumes the provided reference is correct. If a RAG system retrieves a "noisy" or incorrect document, FastMem might actually make the model more likely to believe the wrong information.
  • Harmful Instructions: There is a risk that "memorizing" a prompt could make a model more susceptible to jailbreaking or harmful instructions if they are embedded in the context.
  • Scaling to Long Context: As context windows grow to millions of tokens, the latency of the memorization phase may become a bottleneck.
  • Future Directions: The authors suggest exploring other parameter-efficient methods like LoRA to see if updating more layers (while staying efficient) could yield even better results.

Summary updated

Abstract

Large language models (LLMs) excel in generating coherent text, but they often struggle with context awareness, leading to inaccuracies in tasks requiring faithful adherence to provided information. We introduce FastMem, a novel method designed to enhance instruction fine-tuned LLMs' context awareness through fast memorization of the prompt. FastMem maximizes the likelihood of the prompt before inference by updating only the last Feed-Forward Network (FFN) module. This targeted approach ensures efficient optimization without overfitting, significantly improving the model's ability to comprehend and accurately follow the context. Our experiments demonstrate substantial gains in reading comprehension, text summarization and adherence to output structures. For instance, FastMem improves the accuracy of Llama 3-8B-Inst on the NQ-SWAP dataset from 59.1% to 71.6%, and reduces the output structure failure rate of Qwen 1.5-4B-Chat from 34.9% to 25.5%. Extensive experimental results highlight FastMem's potential to offer a robust solution to enhance the reliability and accuracy of LLMs in various applications. Our code is available at: https://github.com/IAAR-Shanghai/FastMem
BibTeX
@inproceedings{zhu2024fastmem,
  author = {Junyi Zhu and Shuochen Liu and Yu Yu and Bo Tang and Yibo Yan and Zhiyu Li and Feiyu Xiong and Tong Xu and Matthew B. Blaschko},
  title = {{FastMem: Fast Memorization of Prompt Improves Context Awareness of Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=Ud3CKCZRuN},
}
Figure