TL;DR
This paper introduces Neurocache, a novel approach to extend the context window of Large Language Models (LLMs) by efficiently retrieving and integrating past states from an external vector cache. Neurocache significantly improves inference speed and language modeling accuracy, particularly for long-document tasks, by using compressed states and a single retrieval operation per token.
What Problem Does This Paper Address?
Large Language Models (LLMs) have demonstrated remarkable capabilities, but their effectiveness is often limited by a fixed, relatively short context window. This limitation becomes a bottleneck for tasks requiring the processing of long documents, such as summarization or academic literature review, where hundreds of thousands of tokens might need to be considered. While various methods exist to extend context windows, they often struggle with full utilization or introduce significant computational overhead. This paper addresses the challenge of efficiently processing long documents by enhancing LLMs with an external memory mechanism.
Methods
Neurocache introduces a vector retrieval approach that leverages a k-nearest-neighbor (kNN) algorithm to retrieve relevant past states from a compressed external vector cache. Here's a breakdown of its key components:
- State Compression: Hidden states from Transformer decoder layers are compressed into a lower-dimensional representation using a learned projection matrix. This reduces the cache size and speeds up retrieval.
- Efficient kNN Retrieval: For each token, Neurocache performs a single kNN search to find the most similar past states in the cache. This is a significant improvement over methods that require multiple queries per token.
- Cache Updating: The cache is updated with new compressed states using a First-In-First-Out (FIFO) strategy to maintain a fixed size.
- Cache-Augmented Layers: Specialized layers integrate retrieved states into the attention mechanism. These layers use the retrieved states to generate keys and values for a "cache attention" operation, which is then combined with the standard self-attention output.
- Contextual Retrieval Window & Extended Cache-Attention: Neurocache not only retrieves the top-k most similar states but also considers neighboring states within a defined window. It also extends cache-attention to include retrievals from preceding tokens, providing a richer context.
Datasets
The paper uses the following datasets for training and evaluation:
- PG-19: A benchmark for long-form language modeling, consisting of books published before 1919.
- LongPile: A diverse dataset derived from The Pile, with documents exceeding 20K tokens.
- LongBench: A suite of seven downstream tasks for evaluating long-document processing, including single-document QA, multi-document QA, and few-shot learning.
Key Insights and Ideas
- Compression is Key: Compressing hidden states before caching is crucial for managing cache size and improving retrieval efficiency.
- Single Query, Big Impact: Performing only one kNN query per token, rather than one per attention head or layer, dramatically reduces computational overhead.
- Contextual Neighbors Matter: Extending the retrieval window to include neighboring states, not just the absolute closest ones, improves language modeling and downstream task performance.
- Adaptability is a Strength: Neurocache can be effectively adapted to existing pre-trained models like Llama2-7B and Mistral-7B, significantly enhancing their long-context capabilities.
Key Results
- Improved Perplexity: Neurocache consistently achieves lower token perplexity compared to baselines like Memorizing Transformers on both PG-19 and LongPile datasets, across various cache sizes. For example, when adapted to Llama2-7B, Neurocache achieved a perplexity of 7.078 on PG-19 (16K cache) compared to 7.359 for the base model.
- Faster Inference: Figure 1 demonstrates that Neurocache offers faster batched inference times than Memorizing Transformers, especially as cache size increases.
- Enhanced Downstream Performance: On the LongBench benchmark, Neurocache-adapted models (Llama2-7B and Mistral-7B) showed superior performance in single-document QA and few-shot learning tasks. For instance, Neurocache achieved an F1 score of 41.23 in MultiFieldQA for Llama2-7B, outperforming the base model's 33.39.
- Scalability: Neurocache scales well with larger cache sizes, maintaining its performance advantage.
Limitations and Future Work
- Multi-document QA Challenges: Neurocache showed competitive performance but fell short of Text Retrieval methods in multi-document QA tasks. This suggests potential limitations in integrating information from multiple disparate sources.
- Domain Specificity: The evaluation was primarily on general text corpora (PG-19, LongPile). Performance on specialized domains like technical documents or source code might differ.
- Zero-Shot Evaluation: The study relied on zero-shot evaluation. Fine-tuning on downstream tasks could potentially yield different results and better capture the model's full adaptability.
- Bias Analysis: The paper acknowledges that Neurocache's outputs are influenced by the underlying language models and datasets, and an explicit analysis of model biases was not conducted.
Future work could explore extending Neurocache to multi-document tasks, adapting it to different model architectures, and conducting more extensive domain-specific evaluations.
Summary updated