Abstract
Modern Large Language Models (LLMs) predominantly utilize decoder-only architectures, which lack inherent statefulness and struggle with long-term memory. This paper introduces a computationally efficient method to augment decoder-only models, such as LLaMA, with a specialized memory encoder. By compressing external knowledge or conversation history into a small set of latent tokens, the proposed architecture achieves superior performance compared to traditional context-window expansion methods.
Introduction
The "stateless" nature of decoder-only LLMs requires developers to re-feed entire conversation histories into the model with every new prompt. This approach faces two primary challenges: quadratic scaling of computational costs as context length increases, and the difficulty of integrating post-training external knowledge. While increasing context windows is a common industry trend, it remains computationally expensive. This work proposes a "memory encoder" as a practical alternative to provide LLMs with long-term memory capabilities.
Methodology
The proposed architecture bridges the gap between encoder-decoder and decoder-only models by attaching a small, specialized memory encoder to a pretrained LLaMA model.
Architecture and Dimensionality Reduction
The memory encoder is a scaled-down version of the LLaMA architecture, mirrored but modified. Specifically, attention masking is removed to allow the encoder to process the entire memory simultaneously, and cross-attention is omitted. To maintain efficiency, internal dimensions were reduced from 4096 to 128, and the depth was limited to eight layers.
Memory Injection and Training
The encoder processes memory inputs (e.g., Wikipedia snippets) and produces an encoded representation consisting of just 16 tokens. These tokens are then prepended to the frozen decoder's input. The training process involves two stages:
- Pretraining: A reconstruction task to develop robust memory representations.
- Fine-tuning: Training the encoder to assist the decoder in query-answering tasks based on the encoded memory.
Experimental Setup
The approach was evaluated across four datasets requiring external memory:
- Curiosity: Knowledge-grounded conversations.
- DREAM: Dialogue-based reading comprehension.
- Schema-Guided Dialogue (SGD): Task-oriented assistant interactions.
- Wizard of Wikipedia (WoW): Wikipedia-grounded dialogue.
Key Results and Insights
The memory encoder demonstrates a significant performance advantage over baseline methods. Notably, the specialized encoder's 16-token distillation outperformed the "brute force" method of prepending raw text.
As shown in the results, the memory encoder achieved a Cross-Entropy Loss of 7.73, drastically lower than the "No memory" baseline (19.97). Interestingly, prepending plain memory text resulted in the worst performance (21.37), suggesting that raw text can act as a distractor or exceed the model's effective processing range. This highlights the importance of specialized "adapters" that can distill information into a format optimized for the decoder.
Conclusion and Future Work
This work demonstrates that a small, separately trained encoder can effectively augment massive decoder-only models without the need for extensive retraining. Future research will focus on increasing dataset diversity during pretraining and exploring joint fine-tuning of the encoder and decoder to further enhance performance.
Summary updated