Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths

Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang

TL;DR

Long-context Large Language Model (LLM) inference is severely bottlenecked by the massive memory and compute overhead of the Key-Value (KV) Cache. Mixture of Attention Spans (MoA) solves this by automatically assigning customized, dynamic sliding-window lengths to individual attention heads and layers. This training-free approach slashes GPU memory usage and boosts decode throughput by up to 8.2× while maintaining near-full attention accuracy.

What Problem Does This Paper Address?

Processing long contexts in LLMs is notoriously resource-intensive because the KV-Cache grows linearly with sequence length. Existing solutions like sliding-window attention bound this growth by enforcing a fixed, uniform window size across all attention heads.

However, this "one-size-fits-all" approach is deeply flawed. Different attention heads naturally exhibit highly heterogeneous behaviors: some focus strictly on local context, while others track global dependencies. Forcing global heads into a narrow local window severely degrades the model's effective context window and retrieval accuracy.

Figure 2: Examples of attention matrices from different attention heads of the Vicuna-7B model. Each attention matrix is averaged over 256 data items from the LongEval dataset.

Methods

The authors introduce Mixture of Attention Spans (MoA), a training-free framework that optimizes inference by finding the ideal attention span for every single head:

  1. Heterogeneous Elastic Rules: Instead of a fixed window, MoA defines a linear scaling rule for each head ($S_h = \alpha_h + \beta_h \cdot N$), allowing the attention span $S_h$ to dynamically adjust based on the input length $N$.
  2. Attention Influence Profiling: To determine which heads need large windows, MoA uses a first-order Taylor expansion based on gradients ($\partial L / \partial A$) over a calibration dataset. This quantifies the exact impact that masking a specific attention value would have on the model's loss.
  3. Automatic Configuration Search: MoA frames the selection of optimal $\alpha$ and $\beta$ parameters as a multi-objective mixed-integer programming (MIP) problem. This is solved efficiently using standard solvers (like Gurobi) to minimize loss under a target average density constraint.

The entire pipeline is illustrated below: Overview of the MoA pipeline

Figure 3: Overview of MoA. (a) The search space includes heterogeneous elastic rules of the attention span on sliding-window masks. (b) The automatic rule search pipeline begins with a calibration dataset, which includes long-dependency contexts and supervision texts generated by the original dense LLM. MoA profiles each attention value’s impact on model predictions within this dataset, revealing accuracy losses for different candidate elastic rules across various input lengths. The final optimization step selects elastic rules for each attention head to minimize the total prediction loss while adhering to specified density constraints.

Datasets

  • Calibration & Validation: MultiNews (a long-context multi-document summarization dataset). Crucially, the authors use summaries generated by the original dense LLM rather than human-written ones to ensure perfect model alignment during profiling.
  • Evaluation: LongEval (for key-value retrieval accuracy), LV-Eval and LongBench (benchmarks for long-context understanding across subtasks like QA, summarization, and coding), and a custom perplexity test set sampled from Qasper, MultiNews, TREC, and LCC.

Key Insights and Ideas

  • Heads scale differently: Some attention heads maintain a fixed local focus regardless of sequence length, while others expand their focus rapidly as the input grows. Uniform windowing wastes resources on local heads and chokes global ones.
  • Data engineering matters for compression: Profiling attention with standard language modeling datasets (like RedPajama) or human responses leads to poor configurations due to misalignment. Using long-range dependency data with model-generated responses provides much cleaner gradient signals for optimization, boosting retrieval accuracy by up to 60%.
  • Layer-wise trends: Initial and middle layers generally require higher KV-cache density (larger spans), whereas final layers can get away with much lower density, except for a few critical outlier heads.

Key Results

  • Massive Speedups: MoA achieves a 6.6–8.2× decode throughput improvement over FlashAttention2 and a 1.7–1.9× improvement over the highly optimized vLLM framework at 50% density.
  • Memory Reduction: It reduces GPU memory consumption by 1.2–1.4×, allowing a single A100 GPU to process 128k context lengths that would otherwise require multiple GPUs using standard baselines.
  • Accuracy Preservation: MoA increases the effective context length by 3.9× compared to uniform baselines. Across long-context benchmarks, it shrinks the relative performance drop from 9–36% (in baselines) to within 5% of full attention.
Figure 1: Retrieval accuracy of the Vicuna-7B model using different attention methods across varying input lengths and retrieval positions on the LongEval benchmark (Li et al., 2023a). The benchmark takes massive key-value pairs as inputs and tests the accuracy to retrieve values based on given keys from diverse positions. (a) Original model with a full attention span; (b) StreamingLLM with half the attention span, showing reduced effectiveness beyond the span; (c) MoA with half the attention span, maintaining effectiveness beyond the span.

Limitations and Future Work

  • Low-Density Failure: Under extremely aggressive memory constraints (very low-density budgets), MoA's performance begins to degrade.
  • Linear Constraints: The current implementation relies on linear elastic rules; exploring non-linear scaling rules with bounded maximum spans could yield better efficiency.
  • Dynamic Adaptation: MoA is currently static per head/length after the search phase. A promising future direction is developing a fully dynamic MoA that adapts on-the-fly to specific input contents.

Summary updated

Abstract

Sliding-window attention offers a hardware-efficient solution to the memory and throughput challenges of Large Language Models (LLMs) in long-context scenarios. Existing methods typically employ a single window length across all attention heads and input sizes. However, this uniform approach fails to capture the heterogeneous attention patterns inherent in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this challenge, we propose *Mixture of Attention Spans* (MoA), which automatically tailors distinct sliding-window length configurations to different heads and layers. MoA constructs and navigates a search space of various window lengths and their scaling rules relative to input sizes. It profiles the model, evaluates potential configurations, and pinpoints the optimal length configurations for each head. MoA adapts to varying input sizes, revealing that some attention heads expand their focus to accommodate longer inputs, while other heads consistently concentrate on fixed-length local contexts. Experiments show that MoA increases the effective context length by 3.9x with the same average sliding-window length, boosting retrieval accuracy by 1.5-7.1x over the uniform-window baseline across Vicuna-{7B, 13B} and Llama3-{8B, 70B} models. Moreover, MoA narrows the performance gap with full attention, reducing the maximum relative performance drop from 9%-36% to within 5% across three long-context understanding benchmarks. MoA achieves a 1.2-1.4x GPU memory reduction, boosting decode throughput by 6.6-8.2x and 1.7-1.9x over FlashAttention2 and vLLM, with minimal performance impact. Our code is available at: https://github.com/thu-nics/MoA
BibTeX
@inproceedings{fu2025mixture,
  author = {Tianyu Fu and Haofeng Huang and Xuefei Ning and Genghan Zhang and Boju Chen and Tianqi Wu and Hongyi Wang and Zixiao Huang and Shiyao Li and Shengen Yan and Guohao Dai and Huazhong Yang and Yu Wang},
  title = {{Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths}},
  booktitle = {COLM},
  year = {2025},
  url = {https://openreview.net/forum?id=n3rZJrWPLE},
}
Figure