Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths

Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang

Abstract

Sliding-window attention offers a hardware-efficient solution to the memory and throughput challenges of Large Language Models (LLMs) in long-context scenarios. Existing methods typically employ a single window length across all attention heads and input sizes. However, this uniform approach fails to capture the heterogeneous attention patterns inherent in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this challenge, we propose Mixture of Attention Spans (MoA), which automatically tailors distinct sliding-window length configurations to different heads and layers. MoA constructs and navigates a search space of various window lengths and their scaling rules relative to input sizes. It profiles the model, evaluates potential configurations, and pinpoints the optimal length configurations for each head. MoA adapts to varying input sizes, revealing that some attention heads expand their focus to accommodate longer inputs, while other heads consistently concentrate on fixed-length local contexts. Experiments show that MoA increases the effective context length by 3.9x with the same average sliding-window length, boosting retrieval accuracy by 1.5-7.1x over the uniform-window baseline across Vicuna-{7B, 13B} and Llama3-{8B, 70B} models. Moreover, MoA narrows the performance gap with full attention, reducing the maximum relative performance drop from 9%-36% to within 5% across three long-context understanding benchmarks. MoA achieves a 1.2-1.4x GPU memory reduction, boosting decode throughput by 6.6-8.2x and 1.7-1.9x over FlashAttention2 and vLLM, with minimal performance impact.

Introduction

As Large Language Models (LLMs) are increasingly deployed for long-context tasks such as document analysis and multi-turn dialogue, the "attention bottleneck" has become a critical challenge. Standard self-attention requires storing a Key-Value (KV) cache that grows linearly with sequence length, quickly exhausting GPU memory and slowing down generation. While sliding-window attention (e.g., StreamingLLM) mitigates this by limiting focus to a fixed number of recent tokens, these methods typically apply a uniform window size to every attention head.

The authors of "Mixture of Attention Spans" (MoA) argue that this uniform approach is fundamentally flawed. LLM attention heads are heterogeneous: some focus on local patterns, while others act as "global anchors" that require a broader context to function correctly. A uniform window size either starves global heads of necessary information or wastes memory on local heads. MoA solves this by automatically assigning custom, "elastic" sliding-window lengths to each attention head.

Methodology: The MoA Framework

MoA is a training-free framework that tailors attention spans to the specific needs of each head through a systematic profiling and optimization process.

Heterogeneous Elastic Rules

Instead of a fixed window, MoA defines the attention span ($S_h$) for each head ($h$) as a linear function of the input length ($N$): $$S_h = \alpha_h + \beta_h \cdot N$$ This formulation allows for "elasticity": some heads can expand their focus as the conversation grows ($\beta_h > 0$), while others remain strictly local ($\beta_h = 0$).

Profiling and Optimization Pipeline

The MoA framework follows a structured pipeline to determine the optimal $\alpha$ and $\beta$ parameters for every head:

  1. Calibration: The model is run on a calibration dataset to observe attention patterns.
  2. Attention Influence Profiling: MoA uses gradient-based profiling to calculate how much each attention head contributes to the final prediction loss.
  3. Automatic Configuration Search: The selection process is formulated as a multi-objective optimization problem. Using Mixed-Integer Programming (MIP), MoA identifies "Pareto-optimal" configurations that minimize accuracy loss for a given memory (KV-cache density) constraint.
The MoA pipeline: from calibration and profiling to multi-objective optimization for optimal head configurations.

Key Results and Insights

The study evaluates MoA across several prominent long-context benchmarks, including LongEval, LV-Eval, and LongBench, using Vicuna and Llama3 models.

Effective Context Extension

A primary metric for long-context models is the "effective context length"—the point where retrieval accuracy remains high. By protecting critical global heads, MoA extends this length by 3.9x compared to uniform-window methods with the same average memory usage.

Retrieval accuracy comparison: MoA maintains high accuracy (green) across the entire input length, whereas StreamingLLM's accuracy collapses beyond its fixed attention span.

Efficiency and Performance

MoA demonstrates significant efficiency gains with minimal performance trade-offs:

  • Throughput: Achieves 6.6–8.2x speedup over FlashAttention2 and 1.7–1.9x over the highly optimized vLLM framework.
  • Memory: Reduces GPU memory usage by 1.2–1.4x, allowing for larger batch sizes.
  • Accuracy: In long-context retrieval, MoA boosts accuracy by 1.5–7.1x over uniform-window baselines and narrows the performance gap with "full attention" models to within 5%.

Conclusion

Mixture of Attention Spans (MoA) provides a robust solution to the memory and compute costs of long-context LLM inference. By recognizing the specialized roles of different attention heads and applying elastic scaling rules, MoA achieves state-of-the-art efficiency without sacrificing the model's ability to understand vast contexts. Future research may explore non-linear scaling or input-dependent spans to further refine these accuracy-latency trade-offs.

Summary updated

Abstract

Sliding-window attention offers a hardware-efficient solution to the memory and throughput challenges of Large Language Models (LLMs) in long-context scenarios. Existing methods typically employ a single window length across all attention heads and input sizes. However, this uniform approach fails to capture the heterogeneous attention patterns inherent in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this challenge, we propose *Mixture of Attention Spans* (MoA), which automatically tailors distinct sliding-window length configurations to different heads and layers. MoA constructs and navigates a search space of various window lengths and their scaling rules relative to input sizes. It profiles the model, evaluates potential configurations, and pinpoints the optimal length configurations for each head. MoA adapts to varying input sizes, revealing that some attention heads expand their focus to accommodate longer inputs, while other heads consistently concentrate on fixed-length local contexts. Experiments show that MoA increases the effective context length by 3.9x with the same average sliding-window length, boosting retrieval accuracy by 1.5-7.1x over the uniform-window baseline across Vicuna-{7B, 13B} and Llama3-{8B, 70B} models. Moreover, MoA narrows the performance gap with full attention, reducing the maximum relative performance drop from 9%-36% to within 5% across three long-context understanding benchmarks. MoA achieves a 1.2-1.4x GPU memory reduction, boosting decode throughput by 6.6-8.2x and 1.7-1.9x over FlashAttention2 and vLLM, with minimal performance impact. Our code is available at: https://github.com/thu-nics/MoA
BibTeX
@inproceedings{fu2025mixture,
  author = {Tianyu Fu and Haofeng Huang and Xuefei Ning and Genghan Zhang and Boju Chen and Tianqi Wu and Hongyi Wang and Zixiao Huang and Shiyao Li and Shengen Yan and Guohao Dai and Huazhong Yang and Yu Wang},
  title = {{Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths}},
  booktitle = {COLM},
  year = {2025},
  url = {https://openreview.net/forum?id=n3rZJrWPLE},
}
Figure