TL;DR
HiSA (Hidden Sparse Attention) is a novel approach that sparsifies the hidden dimension of attention rather than dropping tokens. By leveraging the structure of Rotary Position Embeddings (RoPE), HiSA computes only about a third of the query-key dot product—dropping high-frequency components for distant tokens—resulting in lower validation loss, faster decoding, and a significantly smaller memory footprint.
What Problem Does This Paper Address?
Standard attention in Large Language Models (LLMs) is computationally expensive because it calculates a full dot product between every query and every key across the entire hidden dimension ($d$). While many researchers have tried to make attention "cheaper" by dropping tokens (e.g., sliding windows or KV-cache eviction), very few have explored sparsifying the hidden dimension itself.
The authors observe that modern LLMs use Rotary Position Embeddings (RoPE) to encode position. RoPE decomposes the hidden dimension into frequency pairs that rotate at different speeds. High-frequency pairs rotate many times over a few tokens, making them extremely sensitive to exact distances. However, language structure at long distances is usually "coarse" and insensitive to exact offsets. This raises a fundamental question: Do we really need to compute high-precision, high-frequency components for tokens that are thousands of positions apart?
Methods
The authors propose Hidden Sparse Attention (HiSA). Instead of a full dot product, HiSA computes a partial one based on a static, parameter-free rule:
- The HiSA Rule: A frequency pair $l$ in the dot product is kept only if its rotation angle $\Delta\theta_l$ (induced by the distance between query and key) is below a fixed budget $\Phi$ (defaulting to 90°).
- Static Sparsity: Because this depends only on token positions and RoPE frequencies, the sparsity pattern is known ahead of time. No learned routers or runtime selection steps are required.
- Hardware Optimization: The authors implement two versions:
- Exact Rule: A Triton-based kernel that follows the rule precisely.
- Flash-HiSA: An extension of FlashAttention-3 that applies the rule to blocks of tokens and chunks of features, maintaining high hardware utilization on GPUs.
Datasets
The models were trained and evaluated using:
- OpenWebText: Used for pretraining the 135M and 360M parameter models from scratch.
- PG19: A dataset of books used for long-context extension (up to 16K tokens).
- SmolLM Architectures: The experiments utilized the SmolLM-135M and SmolLM-360M setups as baselines.
Key Insights and Ideas
- Fast Pairs See Near, Slow Pairs See Far: High-frequency RoPE pairs act as "local experts," while low-frequency pairs provide the "global view." HiSA formalizes this by letting fast pairs drop out as distance increases.
- Beneficial Inductive Bias: Surprisingly, HiSA often achieves lower validation loss than dense attention. This suggests that forcing the model to ignore high-frequency noise at long distances acts as a helpful regularizer.
- Dense Models "Want" to be Sparse: Through an interpretability analysis, the authors show that standard dense models naturally learn to suppress the same high-rotation components that HiSA removes by construction. HiSA simply builds this learned behavior directly into the architecture.
- KV Cache Efficiency: Because different parts of a key are only needed for specific distances, HiSA can evict chunks of the KV cache at different times, leading to significant memory savings during inference.
Key Results
- Performance: A 135M model trained with HiSA reached a validation loss of 2.880, outperforming the dense baseline of 2.906, despite computing only 29.3% of the dot products.
- Inference Speed: At a 16K context length, Flash-HiSA improved decoding throughput by 42% compared to dense FlashAttention-3.
- Memory Footprint: HiSA reduced the KV cache size by approximately 33% at 4K context and even more at longer contexts.
- Retrofitting: The authors demonstrated that a pretrained dense model can be converted to HiSA via continued training (1.35B tokens) without losing quality, matching the performance of a continued-dense baseline.
Limitations and Future Work
- Scale: The study focused on relatively small models (up to 360M parameters). Whether these gains hold for 7B+ or 70B+ parameter models remains to be seen.
- Downstream Benchmarks: While validation loss improved, the paper does not report performance on standard downstream benchmarks (like MMLU or GSM8K).
- Kernel Complexity: While Flash-HiSA is fast, the authors note that better kernel design could further close the gap between the "Exact" rule's superior loss and the "Block" rule's superior speed.
Summary updated