JAXBench: Benchmarking Autonomous TPU Kernel Optimization

Arya Tschand, Charles Hong, Julian Walker, Nina Cai, Shangkun Wang, Suvinay Subramanian, Sundar Dev, Vijay Janapa Reddi, Amir Yazdanbakhsh, Sethu Sankaran

TL;DR

JAXBench is the first TPU-native benchmark suite designed to evaluate AI-driven kernel optimization using Google's Pallas language. By testing language models on 50 compute-heavy JAX workloads, the paper demonstrates that providing target-specific hardware and API documentation is far more critical for optimizing low-level TPU code than simply scaling up model size.

What Problem Does This Paper Address?

Optimizing low-level hardware kernels is a major bottleneck in deep learning. While the community has established robust benchmarks (like KernelBench and TritonBench) to evaluate AI-generated kernels for GPUs, no equivalent exists for Tensor Processing Units (TPUs).

TPUs differ fundamentally from GPUs—they rely on sequential execution with wide SIMD vector registers and systolic Matrix Multiply Units (MXUs) rather than a massively parallel SIMT model. Furthermore, writing TPU kernels requires Pallas, a domain-specific language (DSL) embedded in JAX that lowers to the Mosaic backend. Because Pallas is severely underrepresented in LLM training data, standard models frequently hallucinate APIs and fail to respect strict TPU hardware constraints (like memory hierarchies and block size divisibility). Standard compiler feedback doesn't explain these hardware-level violations, leaving AI agents stuck.

Methods

The authors build an automated evaluation harness that compiles candidate Pallas kernels, verifies numerical correctness against a JAX baseline on bf16 inputs, and profiles execution times directly on device using jax.profiler to eliminate host dispatch overhead.

Using this harness, they evaluate four LLM-driven optimization strategies:

  1. Best-of-N: One-shot generation without feedback.
  2. Iterative Refinement: An agent loop that receives compilation errors and profiler feedback to fix and optimize code over multiple turns.
  3. Iterative Refinement + Context: The same iterative loop, but augmented with curated TPU architecture summaries, Pallas API references, and code examples.
  4. Autocomp: A state-of-the-art, hardware-aware kernel optimizer that splits the task into a translation phase (beam search to get a correct Pallas kernel) and an optimization phase (refining block sizes and scheduling for speed).
JAXBench evaluation harness and benchmark construction overview

Datasets

The paper introduces JAXBench, a suite of 50 JAX workloads specifically sized to push the hardware into compute-bound regimes (ensuring optimization headroom is meaningful):

  • 17 Priority Production Kernels: Extracted from real-world LLM and biology architectures in Google's public MaxText library (including Llama-3.1, DeepSeek-V3, Mixtral, Mamba-2, and AlphaFold2). These cover attention variants (Flash, GQA, MLA), sparse MoE, and state-space models.
  • 33 Fused Operators: Adapted from KernelBench Level 2, translated from PyTorch to JAX, and upscaled to ensure they achieve at least 60% MXU utilization on TPU v6e.
  • Expert Baselines: For 8 of the priority workloads, they include hand-optimized Pallas kernels from the Tokamax library, block-size tuned via exhaustive grid search to serve as an expert performance upper bound.

Key Insights and Ideas

  • Context > Scale: For niche or sparsely documented languages like Pallas, injecting high-quality documentation into the prompt is vastly more effective than switching to a larger model. Documentation bridges the "information gap" that reasoning alone cannot solve.
  • The Compiler Feedback Blindspot: Standard compiler errors don't tell an LLM why a tiling layout violates systolic constraints or why a grid traversal order causes memory issues. Without explicit architectural rules in the context, iterative agents have nothing useful to iterate against.
  • Correctness vs. Optimization Split: Getting a kernel to compile and pass correctness checks requires debugging, but making it fast requires structured search. Agents like Autocomp succeed because they explicitly pivot from a debugging mindset to an optimization mindset once a correct "seed" kernel is found.

Key Results

  • Massive Correctness Boost: Adding curated TPU documentation to the iterative loop raises per-sample correctness from a dismal 5.8% to 37.3%, successfully solving 48 out of 50 benchmarks.
  • Speedups over XLA: Autocomp's structured beam search achieves a 1.36× geometric mean speedup over XLA across the entire 50-workload suite.
  • Approaching Expert Performance: On the 8 workloads with hand-tuned baselines, Autocomp achieves a 1.60× geomean speedup, recovering a significant portion of the 2.08× expert upper bound from Tokamax, though it still struggles with highly complex scheduling required for ragged and paged attention.
  • Model Capacity Impact: Upgrading from Gemini 3 Flash to Gemini 3.1 Pro dramatically lifts all iterative methods, with the combination of Pro and context reaching up to a 3.82× geomean speedup on a 5-kernel subset.
Speedup and solved-rate trajectories across different optimization methods

Limitations and Future Work

  • Single-Chip Focus: Currently, JAXBench evaluates workloads on a single TPU v6e chip. Real-world production training and inference run across massive multi-chip pods where sharding strategies (shard_map) and collective communication overlap dominate performance.
  • Limited Expert References: Only 8 of the 17 priority production operators currently have hand-optimized Pallas reference baselines.
  • Future Directions: The authors plan to extend JAXBench to multi-chip parallel regimes (tensor, pipeline, and expert parallel). Additionally, they suggest using JAXBench's correctness and latency signals as a reward mechanism to fine-tune LLMs or train reinforcement learning agents specifically for TPU optimization.

Summary updated

Abstract

Rigorous benchmarks have driven progress in autonomous GPU kernel performance optimization by establishing a shared target to hillclimb on, but no equivalent exists for TPUs. We present JAXBench, a TPU-native benchmark suite for AI-generated kernel optimization on Google Cloud TPUs. JAXBench comprises 50 JAX workloads that are both relevant and provide headroom for optimization. We extract 17 production ML operators from architectures in the public MaxText library such as Llama-3.1, DeepSeek-V3, Mixtral, Mamba-2, and AlphaFold2, and translate 33 operators from KernelBench that are validated for correctness and set with new problem sizes that achieve high TPU v6e MXU utilization. Eight of the 17 production operators ship with hand-optimized Pallas kernels from the public Tokamax library and block-size tuned to establish an expert upper-bound baseline. We evaluate four feedback-driven methods on generating candidate Pallas kernels for JAXBench. Across the full suite with Gemini 3 Flash, we find that target-specific context matters more than model scale on a sparsely-documented DSL like Pallas. Conditioning on curated TPU documentation raises per-sample correctness from 5.8% to 37.3% and solves 48 of 50 benchmarks at a 1.28x geomean speedup. Search structure yields significant gains once correctness is achieved, with Autocomp's beam-search pipeline reaching a 1.36x geomean speedup over XLA. On the 8 hand-tuned kernels, Autocomp reaches 1.60x geomean over XLA, recovering most of the 2.08x Tokamax upper bound but trailing on the specialized paged and ragged attention operators. High-quality TPU kernel optimization remains a challenging task, and we release the JAXBench benchmark, evaluation harness, and baseline results to support open source contributions.
BibTeX
@inproceedings{tschand2026jaxbench,
  author = {Arya Tschand and Charles Hong and Julian Walker and Nina Cai and Shangkun Wang and Suvinay Subramanian and Sundar Dev and Vijay Janapa Reddi and Amir Yazdanbakhsh and Sethu Sankaran},
  title = {{JAXBench: Benchmarking Autonomous TPU Kernel Optimization}},
  booktitle = {DL4C @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=RFpZOuhIk3},
}
Figure