TL;DR
This paper introduces MixAttention, a novel architecture that combines sliding window attention with KV cache sharing to significantly reduce memory usage and speed up inference for large language models (LLMs). By strategically employing these techniques, MixAttention achieves competitive performance on both short and long-context tasks, making LLMs more practical for real-world applications.
What Problem Does This Paper Address?
Modern Transformer-based language models, while powerful, face a significant challenge during inference: the ever-growing Key-Value (KV) cache. This cache stores intermediate computations for attention mechanisms, and its size scales linearly with both the model's context length and the number of attention heads. This leads to substantial memory consumption and slower inference, especially for long inputs. This problem is particularly acute in applications like Retrieval Augmented Generation (RAG), which inherently require processing lengthy documents. The authors aim to alleviate these inference bottlenecks without sacrificing model performance.
Methods
The core of the proposed solution is MixAttention, an architecture that blends two key ideas:
- Sliding Window Attention: Instead of attending to all previous tokens, each token only attends to a fixed-size window of recent tokens. This drastically reduces the number of KV pairs that need to be stored and processed. The paper uses a window size of 1024 tokens.
- KV Cache Sharing: This technique allows multiple layers in the Transformer to reuse the same KV cache computed by an earlier layer. This effectively reduces the total KV cache size by a factor related to the number of layers sharing the cache.
The authors explore various configurations of MixAttention, combining these two techniques in different ways and positions within the model architecture. They also retain a few standard attention layers, recognizing their importance for long-context abilities. The models are built upon Mixture of Experts (MoE) architectures with 1.64B active and 5.21B total parameters, using RoPE positional embeddings and Grouped Query Attention (GQA).
Datasets
Training: * A large corpus of 101B tokens for initial pretraining with a 4k context length. * A mix of natural language and code data (9B tokens) for increasing context length to 32k. * A synthetic, long-context Question Answering (QA) dataset (500M tokens) for fine-tuning on long-context tasks.
Evaluation: * Mosaic Evaluation Gauntlet v0.3.0: A suite of benchmarks for evaluating model quality across various tasks like reading comprehension, commonsense reasoning, world knowledge, and symbolic problem-solving. * RULER [12]: A benchmark specifically designed for evaluating long-context abilities, used at a 32k context length. RULER includes tasks like Needle-in-a-haystack (NIAH), Variable Tracking (VT), Common and Frequent Word Extraction (CWE/FWE), and Question Answering (QA).
Key Insights and Ideas
- Strategic Placement of Standard Attention is Crucial: Simply having a standard attention layer at the beginning of the model (even with KV cache reuse) doesn't seem to confer long-context abilities. Placing standard attention layers deeper in the network (like in MA-Offset) or having multiple standard attention layers, at least one of which is deeper, is essential for strong long-context performance.
- Over-Sharing KV Cache in Sliding Window Layers Hurts Performance: While KV cache sharing is beneficial, aggressively sharing across all sliding window layers can degrade long-context abilities. The paper suggests that a moderate sharing pattern (e.g., 2-3 layers) strikes a good balance.
- MixAttention Achieves a Sweet Spot: The proposed MixAttention architecture, particularly configurations like MA-Offset and MA-Pair, demonstrates a compelling balance between inference speed, memory efficiency, and model quality across both short and long contexts.
Key Results
- Significant Speedup and Memory Reduction: MixAttention models are substantially faster and consume less memory during inference compared to standard attention models. Figure 8 shows a significant reduction in inference time and an increase in the maximum total number of tokens that can be processed.
- Competitive Performance on Long Context: Configurations like MA-Offset and MA-Pairs maintain performance comparable to standard attention models on RULER evals, indicating strong long-context abilities.
- Trade-offs Exist: While MixAttention generally performs well, some configurations (e.g., MA and MA-EndSlide) showed worse performance on long-context QA tasks, highlighting the importance of architectural choices. The paper also notes that on certain challenging long-context tasks like Variable Tracking, neither MixAttention nor standard attention models perform exceptionally well, suggesting room for further improvement.
- Gauntlet Evals Show Similar Short-Context Abilities: On standard tasks evaluated by the Mosaic Eval Gauntlet, MixAttention models exhibit similar metrics to baseline models, suggesting they retain good shorter-context capabilities. However, a slight performance dip was observed in reading comprehension tasks.
Limitations and Future Work
- Further Exploration of Architectures: The authors encourage further research into more MixAttention configurations, as the hyperparameter space is vast.
- Longer Training for Specific Tasks: For tasks where performance was suboptimal (e.g., Variable Tracking), longer training or different training data might be necessary.
- Reading Comprehension Improvement: An open question is whether different MixAttention configurations or extended training can recover the observed dip in reading comprehension abilities.
- Optimizing Sliding Window Implementation: The paper notes that the current implementation of sliding window attention in SGLang did not optimize memory consumption, suggesting that further optimizations could increase the number of tokens MixAttention can support.
- Sharing KV Cache Between Consecutive Layers: While explored, the benefits of sharing standard attention KV caches between consecutive layers were mixed, with some tasks showing slight degradation. Further research is needed to fully understand these effects.
Summary updated