TL;DR
Vortex is an open-source system designed to accelerate the development and deployment of sparse attention algorithms for large language models (LLMs). By combining a high-level Python frontend language with a page-centric tensor abstraction, Vortex allows both human researchers and AI agents to rapidly prototype and execute optimized sparse attention mechanisms directly within production serving stacks, yielding up to 4.7× throughput improvements on massive models.
What Problem Does This Paper Address?
As LLMs handle increasingly long context lengths, the memory movement of the Key-Value (KV) cache during decoding becomes a massive system bottleneck. Sparse attention algorithms mitigate this by computing attention over only a subset of tokens. However, deploying these algorithms in production is highly engineering-intensive.
Modern serving frameworks use paged attention to store KV caches in non-contiguous memory blocks to prevent memory fragmentation. This breaks standard contiguous tensor assumptions found in frameworks like PyTorch, making it incredibly difficult to implement dynamic sparsity patterns on the fly. As a result, researchers face a steep engineering barrier, lacking the programmability to easily iterate on new sparse attention ideas or seamlessly integrate them into evolving production serving stacks.
Methods
The authors introduce Vortex, a unified framework that bridges the gap between high-level algorithmic design and low-level paged memory layouts through three core components:
- vFlow: A Python-embedded frontend language that provides a simple, single-request mental model. It separates sparse attention into two stages: query-independent preprocessing (building the cache) and query-dependent execution (dynamic block indexing).
- vTensor: A page-centric tensor abstraction that extends PyTorch tensors with explicit layout metadata (batch size, index pointers, and ragged page structures). This allows complex sparse operations to be built compositionally without exposing low-level memory details.
- Execution Backend: An optimized backend integrated directly into production serving stacks (like SGLang). It incorporates automated workload planning, greedy kernel fusion to eliminate redundant memory traffic, and a custom stochastic radix top-$k$ operator that trades a tiny fraction of exactness for substantial indexing speedups. It natively supports advanced attention architectures like Grouped-Query Attention (GQA) and Multi-Head Latent Attention (MLA).
Datasets
The system was evaluated using standard long-context and reasoning benchmarks:
- RULER: Used for initial 4K retrieval validation.
- AMC23 and AIME24: Used to evaluate reasoning accuracy and server-side decoding throughput on models up to 8B parameters.
- AIME26: Used for ultra-long context scaling evaluations (up to 32K tokens) on massive models.
No new datasets were introduced in this work.
Key Insights and Ideas
- AI Agents as Autonomous Systems Researchers: By providing clean, programmable abstractions, Vortex allows AI agents (such as Claude Code and GPT-5) to autonomously innovate. In an 18-hour autonomous optimization loop, an agent successfully explored diverse algorithmic hypotheses, tuned block sizes, and adjusted system parameters to maximize throughput while preserving baseline accuracy.
- Sparsity Routing is Highly Concentrated: Using Vortex as a research instrument, the authors discovered that routing signals are concentrated in very few channels. In Qwen3, masking just two specific query-key channel groups completely collapsed model accuracy to near-random, while keeping only those two groups was virtually lossless.
- RoPE is Vital for MLA Routing: In Multi-Head Latent Attention (MLA), ignoring the decoupled rotary position embedding (RoPE) component during block selection causes a massive drop in accuracy. This proves that positional context, not just compressed content, carries the critical signal needed for accurate block selection.
Key Results
- Agent-Driven Optimization: Autonomous AI agents utilizing Vortex generated sparse attention variants that achieved up to a 3.46× higher throughput than full attention on Qwen3-1.7B while maintaining accuracy.
- Architectural Speedups: On the MLA-based GLM-4.7-Flash model running on an NVIDIA B200 GPU, a custom rope-aware sparse attention mechanism designed in vFlow delivered a 4.7× end-to-end throughput speedup over full attention.
- Massive Scale Deployment: When scaling to the 229B-parameter MiniMax-M2.7 model across four NVIDIA B200 GPUs, Vortex achieved up to a 1.37× speedup and matched or slightly exceeded full-attention accuracy.
- Latency Reduction: Vortex dramatically reduced user-side latency under heavy workloads, achieving up to an 11.7× reduction in P95 latency for block top-$k$ attention and 12.8× for Quest.
Limitations and Future Work
- Prefill Phase Support: Vortex currently focuses exclusively on the decoding stage of LLM serving, where the KV cache memory movement is the primary bottleneck. Extending these programmable abstractions to accelerate the prefill phase remains an open challenge.
- Inference-Only Optimization: The system is strictly designed for forward-pass inference serving and does not support the backward pass or gradient layouts required for training sparse attention models from scratch.
- Alternative Architectures: Vortex currently centers on attention-based models with explicit KV caches. The authors note that extending the vTensor abstraction to recurrent or state-space architectures like RNNs and Mamba is a promising future direction.
Summary updated