TL;DR
JAXBench is the first TPU-native benchmark suite designed to evaluate AI-driven kernel optimization using Google's Pallas language. By testing language models on 50 compute-heavy JAX workloads, the paper demonstrates that providing target-specific hardware and API documentation is far more critical for optimizing low-level TPU code than simply scaling up model size.
What Problem Does This Paper Address?
Optimizing low-level hardware kernels is a major bottleneck in deep learning. While the community has established robust benchmarks (like KernelBench and TritonBench) to evaluate AI-generated kernels for GPUs, no equivalent exists for Tensor Processing Units (TPUs).
TPUs differ fundamentally from GPUs—they rely on sequential execution with wide SIMD vector registers and systolic Matrix Multiply Units (MXUs) rather than a massively parallel SIMT model. Furthermore, writing TPU kernels requires Pallas, a domain-specific language (DSL) embedded in JAX that lowers to the Mosaic backend. Because Pallas is severely underrepresented in LLM training data, standard models frequently hallucinate APIs and fail to respect strict TPU hardware constraints (like memory hierarchies and block size divisibility). Standard compiler feedback doesn't explain these hardware-level violations, leaving AI agents stuck.
Methods
The authors build an automated evaluation harness that compiles candidate Pallas kernels, verifies numerical correctness against a JAX baseline on bf16 inputs, and profiles execution times directly on device using jax.profiler to eliminate host dispatch overhead.
Using this harness, they evaluate four LLM-driven optimization strategies:
- Best-of-N: One-shot generation without feedback.
- Iterative Refinement: An agent loop that receives compilation errors and profiler feedback to fix and optimize code over multiple turns.
- Iterative Refinement + Context: The same iterative loop, but augmented with curated TPU architecture summaries, Pallas API references, and code examples.
- Autocomp: A state-of-the-art, hardware-aware kernel optimizer that splits the task into a translation phase (beam search to get a correct Pallas kernel) and an optimization phase (refining block sizes and scheduling for speed).
Datasets
The paper introduces JAXBench, a suite of 50 JAX workloads specifically sized to push the hardware into compute-bound regimes (ensuring optimization headroom is meaningful):
- 17 Priority Production Kernels: Extracted from real-world LLM and biology architectures in Google's public MaxText library (including Llama-3.1, DeepSeek-V3, Mixtral, Mamba-2, and AlphaFold2). These cover attention variants (Flash, GQA, MLA), sparse MoE, and state-space models.
- 33 Fused Operators: Adapted from KernelBench Level 2, translated from PyTorch to JAX, and upscaled to ensure they achieve at least 60% MXU utilization on TPU v6e.
- Expert Baselines: For 8 of the priority workloads, they include hand-optimized Pallas kernels from the Tokamax library, block-size tuned via exhaustive grid search to serve as an expert performance upper bound.
Key Insights and Ideas
- Context > Scale: For niche or sparsely documented languages like Pallas, injecting high-quality documentation into the prompt is vastly more effective than switching to a larger model. Documentation bridges the "information gap" that reasoning alone cannot solve.
- The Compiler Feedback Blindspot: Standard compiler errors don't tell an LLM why a tiling layout violates systolic constraints or why a grid traversal order causes memory issues. Without explicit architectural rules in the context, iterative agents have nothing useful to iterate against.
- Correctness vs. Optimization Split: Getting a kernel to compile and pass correctness checks requires debugging, but making it fast requires structured search. Agents like Autocomp succeed because they explicitly pivot from a debugging mindset to an optimization mindset once a correct "seed" kernel is found.
Key Results
- Massive Correctness Boost: Adding curated TPU documentation to the iterative loop raises per-sample correctness from a dismal 5.8% to 37.3%, successfully solving 48 out of 50 benchmarks.
- Speedups over XLA: Autocomp's structured beam search achieves a 1.36× geometric mean speedup over XLA across the entire 50-workload suite.
- Approaching Expert Performance: On the 8 workloads with hand-tuned baselines, Autocomp achieves a 1.60× geomean speedup, recovering a significant portion of the 2.08× expert upper bound from Tokamax, though it still struggles with highly complex scheduling required for ragged and paged attention.
- Model Capacity Impact: Upgrading from Gemini 3 Flash to Gemini 3.1 Pro dramatically lifts all iterative methods, with the combination of Pro and context reaching up to a 3.82× geomean speedup on a 5-kernel subset.
Limitations and Future Work
- Single-Chip Focus: Currently, JAXBench evaluates workloads on a single TPU v6e chip. Real-world production training and inference run across massive multi-chip pods where sharding strategies (
shard_map) and collective communication overlap dominate performance. - Limited Expert References: Only 8 of the 17 priority production operators currently have hand-optimized Pallas reference baselines.
- Future Directions: The authors plan to extend JAXBench to multi-chip parallel regimes (tensor, pipeline, and expert parallel). Additionally, they suggest using JAXBench's correctness and latency signals as a reward mechanism to fine-tune LLMs or train reinforcement learning agents specifically for TPU optimization.
Summary updated