Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents

Zhuoming Chen, Xinrui Zhong, Qilong Feng, Ranajoy Sadhukhan, Yang Zhou, Michael Qizhe Shieh, Zhihao Jia, Beidi ChenarXiv 2026 · arXiv

TL;DR

Vortex is an open-source system designed to accelerate the development and deployment of sparse attention algorithms for large language models (LLMs). By combining a high-level Python frontend language with a page-centric tensor abstraction, Vortex allows both human researchers and AI agents to rapidly prototype and execute optimized sparse attention mechanisms directly within production serving stacks, yielding up to 4.7× throughput improvements on massive models.

What Problem Does This Paper Address?

As LLMs handle increasingly long context lengths, the memory movement of the Key-Value (KV) cache during decoding becomes a massive system bottleneck. Sparse attention algorithms mitigate this by computing attention over only a subset of tokens. However, deploying these algorithms in production is highly engineering-intensive.

Modern serving frameworks use paged attention to store KV caches in non-contiguous memory blocks to prevent memory fragmentation. This breaks standard contiguous tensor assumptions found in frameworks like PyTorch, making it incredibly difficult to implement dynamic sparsity patterns on the fly. As a result, researchers face a steep engineering barrier, lacking the programmability to easily iterate on new sparse attention ideas or seamlessly integrate them into evolving production serving stacks.

Illustration of physical vs. logical views in paged layouts

Methods

The authors introduce Vortex, a unified framework that bridges the gap between high-level algorithmic design and low-level paged memory layouts through three core components:

  1. vFlow: A Python-embedded frontend language that provides a simple, single-request mental model. It separates sparse attention into two stages: query-independent preprocessing (building the cache) and query-dependent execution (dynamic block indexing).
  2. vTensor: A page-centric tensor abstraction that extends PyTorch tensors with explicit layout metadata (batch size, index pointers, and ragged page structures). This allows complex sparse operations to be built compositionally without exposing low-level memory details.
  3. Execution Backend: An optimized backend integrated directly into production serving stacks (like SGLang). It incorporates automated workload planning, greedy kernel fusion to eliminate redundant memory traffic, and a custom stochastic radix top-$k$ operator that trades a tiny fraction of exactness for substantial indexing speedups. It natively supports advanced attention architectures like Grouped-Query Attention (GQA) and Multi-Head Latent Attention (MLA).
The agentic workflow enabled by Vortex

Datasets

The system was evaluated using standard long-context and reasoning benchmarks:

  • RULER: Used for initial 4K retrieval validation.
  • AMC23 and AIME24: Used to evaluate reasoning accuracy and server-side decoding throughput on models up to 8B parameters.
  • AIME26: Used for ultra-long context scaling evaluations (up to 32K tokens) on massive models.

No new datasets were introduced in this work.

Key Insights and Ideas

  • AI Agents as Autonomous Systems Researchers: By providing clean, programmable abstractions, Vortex allows AI agents (such as Claude Code and GPT-5) to autonomously innovate. In an 18-hour autonomous optimization loop, an agent successfully explored diverse algorithmic hypotheses, tuned block sizes, and adjusted system parameters to maximize throughput while preserving baseline accuracy.
  • Sparsity Routing is Highly Concentrated: Using Vortex as a research instrument, the authors discovered that routing signals are concentrated in very few channels. In Qwen3, masking just two specific query-key channel groups completely collapsed model accuracy to near-random, while keeping only those two groups was virtually lossless.
  • RoPE is Vital for MLA Routing: In Multi-Head Latent Attention (MLA), ignoring the decoupled rotary position embedding (RoPE) component during block selection causes a massive drop in accuracy. This proves that positional context, not just compressed content, carries the critical signal needed for accurate block selection.

Key Results

  • Agent-Driven Optimization: Autonomous AI agents utilizing Vortex generated sparse attention variants that achieved up to a 3.46× higher throughput than full attention on Qwen3-1.7B while maintaining accuracy.
Performance of agent-generated sparse attention algorithms
  • Architectural Speedups: On the MLA-based GLM-4.7-Flash model running on an NVIDIA B200 GPU, a custom rope-aware sparse attention mechanism designed in vFlow delivered a 4.7× end-to-end throughput speedup over full attention.
  • Massive Scale Deployment: When scaling to the 229B-parameter MiniMax-M2.7 model across four NVIDIA B200 GPUs, Vortex achieved up to a 1.37× speedup and matched or slightly exceeded full-attention accuracy.
  • Latency Reduction: Vortex dramatically reduced user-side latency under heavy workloads, achieving up to an 11.7× reduction in P95 latency for block top-$k$ attention and 12.8× for Quest.

Limitations and Future Work

  • Prefill Phase Support: Vortex currently focuses exclusively on the decoding stage of LLM serving, where the KV cache memory movement is the primary bottleneck. Extending these programmable abstractions to accelerate the prefill phase remains an open challenge.
  • Inference-Only Optimization: The system is strictly designed for forward-pass inference serving and does not support the backward pass or gradient layouts required for training sparse attention models from scratch.
  • Alternative Architectures: Vortex currently centers on attention-based models with explicit KV caches. The authors note that extending the vTensor abstraction to recurrent or state-space architectures like RNNs and Mamba is a promising future direction.

Summary updated

Abstract

Sparse attention is becoming increasingly important for serving large language models (LLMs) as generation lengths continue to grow. However, deploying and evaluating new sparse attention algorithms at scale remains highly engineering-intensive, slowing both human researchers and AI agents in exploring the sparse attention design. To address this challenge, we present Vortex, a system that combines a Python-embedded frontend language atop a page-centric tensor abstraction for expressing a broad range of sparse attention algorithms, with an efficient backend tightly integrated into modern LLM serving stacks. Vortex enables rapid prototyping, deployment, and evaluation of sparse attention algorithms, effectively translating their theoretical efficiency gains into real-world throughput improvements. As a result, Vortex substantially accelerates the design and iteration of sparse attention algorithms. First, AI agents use Vortex to automatically generate and refine diverse algorithms, the best reaching up to $3.46\times$ higher throughput than full attention while preserving accuracy. Second, Vortex extends sparse attention to emerging architectures and very large models that are otherwise hard to experiment with, reaching up to $4.7\times$ higher throughput on the MLA-based GLM-4.7-Flash and $1.37\times$ on the 229B-parameter MiniMax-M2.7 on NVIDIA B200 GPUs.
BibTeX
@misc{chen2026vortex,
  author = {Zhuoming Chen and Xinrui Zhong and Qilong Feng and Ranajoy Sadhukhan and Yang Zhou and Michael Qizhe Shieh and Zhihao Jia and Beidi Chen},
  title = {{Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents}},
  year = {2026},
  eprint = {2606.06453},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.06453},
}

Loading related content...

Figure