Abstract
One of the prominent issues stifling the current generation of large language models is their limited context length. Recent proprietary models such as GPT-4 and Claude 2 have introduced longer context lengths, 8k/32k and 100k, respectively; however, despite the efforts in the community, most common models, such as LLama-2, have a context length of 4k or less. Unlimiformer (Bertsch et al., 2023) is a recently popular vector-retrieval augmentation method that offloads cross-attention computations to a kNN index. However, its main limitation is incompatibility with decoder-only transformers out of the box. In this work, we explore practical considerations of adapting Unlimiformer to decoder-only transformers and introduce a series of modifications to overcome this limitation. Moreover, we expand the original experimental setup on summarization to include a new task (i.e., free-form Q&A) and an instruction-tuned model (i.e., a custom 6.7B GPT model). Our results showcase the effectiveness of these modifications on summarization, performing on par with a model with 2x the context length. Moreover, we discuss limitations and future directions for free-form Q&A and instruction-tuned models.
Introduction
Large Language Models (LLMs) are notoriously constrained by their context windows. While proprietary models have pushed these boundaries, many widely used open-source models like Llama-2 remain capped at 4k tokens. This limitation causes models to "forget" early information in long documents, such as books or technical reports, as they process subsequent tokens.
Unlimiformer was originally proposed to address this by using a k-Nearest Neighbors (kNN) index to store hidden states, enabling retrieval from a theoretically unlimited input. However, Unlimiformer was designed for encoder-decoder architectures (e.g., BART, T5). Modern LLMs are predominantly decoder-only and lack the explicit cross-attention mechanism that Unlimiformer utilizes. This work bridges this gap, adapting Unlimiformer for the decoder-only paradigm.
Methodology
Adapting Unlimiformer to decoder-only models requires addressing the lack of a separate encoder and the dynamic nature of decoder hidden states. The authors implemented four primary modifications:
- Artificial Cross-Attention: Specific layers within the decoder (e.g., layer 16) are designated to perform cross-attention over the kNN index, mimicking the behavior of an encoder-decoder model.
- Information Fusion: Retrieved vectors from the kNN index are fused with the current layer's input, allowing the model to dynamically weigh long-range context against immediate tokens.
- Layer-Specific kNN Indices: To account for the varying data distributions across different layers of a decoder, separate kNN indices are maintained for each layer performing cross-attention.
- Dynamic Index Updates: To prevent "index staleness," newly generated tokens are added to the kNN index in real-time, ensuring the model's own recent outputs are available for future retrieval.
Experimental Setup and Results
The approach was evaluated on summarization (GovReport, BookSum) and free-form Q&A (NarrativeQA, Qasper) using a custom 6.7B GPT model in both fine-tuned (GPT-Summ) and instruction-tuned (GPT-Inst) variants.
Performance Gains
A key finding is the "2x rule": a model with a 1k context window using these modifications performs comparably to a standard model with a 2k context window. On the GovReport dataset, the adapted Unlimiformer consistently outperformed baseline models that simply truncated the input.
Task-Specific Variations
The modifications were highly effective for summarization but showed less impact on free-form Q&A. The authors hypothesize that instruction-tuned models may be "biased" toward the immediate prompt, potentially ignoring retrieved long-range context. Additionally, ablation studies confirmed that dynamic index updates (fixing staleness) are crucial for maintaining coherence.
Discussion and Conclusion
While the adaptation shows promise, challenges remain. Maintaining multiple kNN indices introduces computational latency, which might be mitigated by GPU-accelerated indices or approximate search methods. Furthermore, the "query bias" in retrieval suggests that if the model's current state is inaccurate, it may retrieve irrelevant information.
Future work will focus on "un-biasing" instruction-tuned models to better integrate retrieved context and optimizing the efficiency of multi-layer retrieval. This research demonstrates that with targeted architectural modifications, the benefits of retrieval-augmented "unlimited" context can be extended to the current generation of decoder-only LLMs.
Summary updated