Abstract
Transformer-based models such as BERT and GPT2 have become the foundation of many modern applications, yet their execution requires substantial computational and memory resources. To address these challenges, recent advances in compiler technology and hardware accelerators have introduced new opportunities for performance portability. In this work, we evaluate JAX and TVM as high-level frameworks that combine a NumPy-like programming model with Just-In-Time (JIT) or Ahead-of-Time (AOT) code optimization and compilation, enabling efficient execution across CPUs or GPUs, and, in the case of JAX, on TPUs as well. We present systematic implementations of the core Transformer encoder and decoder blocks in JAX and TVM and compare their automatically optimized code against NumPy and CuPy baselines. Our experimental study covers heterogeneous hardware platforms (AMD CPU, NVIDIA GPUs, and Google TPUs) and multiple arithmetic precisions (FP32, INT8, and INT32). Results show that JAX and TVM deliver significant performance improvements over standard libraries, while reducing the programming effort required to adapt to different hardware. These findings demonstrate the potential of JIT- and AOT-oriented frameworks to serve as a portable and efficient solution for deploying Transformer workloads in diverse computing environments.
Introduction
Transformer models like BERT and GPT are the backbone of modern AI, but they are computationally intensive. To achieve high performance, developers often write specialized code for specific hardware, which is time-consuming and error-prone. Standard libraries like NumPy (for CPUs) or CuPy (for GPUs) are accessible but often inefficient because they lack operator fusion—the ability to combine multiple mathematical steps into a single kernel to minimize memory access overhead. This paper investigates whether auto-tuning compilers can provide a "write-once, run-fast-anywhere" solution.
Methodology
The researchers implemented the core building blocks of Transformers—the Multi-Head Attention (MHA) and Feed-Forward Network (FFN)—using two distinct frameworks:
- JAX (Just-In-Time/JIT): Uses the XLA (Accelerated Linear Algebra) compiler to optimize and compile code on the fly.
- Apache TVM (Ahead-of-Time/AOT): Uses an auto-scheduler (Ansor) to search for the fastest execution path before the code runs.
Experimental Setup
Performance was evaluated using synthetic workloads based on two standard model architectures:
- BERT Large: Represents the Encoder side, characterized by compute-bound matrix multiplications.
- GPT2 Large: Represents the Decoder side, characterized by memory-bound matrix-vector products.
Tests were conducted on AMD EPYC CPUs, NVIDIA A100 GPUs, and Google TPU v5e, covering FP32, INT8, and INT32 precisions.
Key Insights and Results
- The Power of Fusion: Significant gains came from fusing operations like Softmax and Layer Normalization directly into matrix multiplication steps, reducing data movement.
- Tuning Efficiency: JAX's optimization occurs in seconds, making it ideal for research. In contrast, TVM's auto-scheduler took nearly 30 hours to optimize the full Transformer block, though it produces highly efficient static binaries.
- Quantization: Auto-tuning frameworks are essential for low-precision math (INT8/INT32), as they generate native hardware instructions rather than emulating them.
- Performance Benchmarks:
- GPUs: JAX was 50x faster than CuPy for the BERT Large encoder in FP32.
- CPUs: JAX outperformed NumPy/OpenBLAS by 2.66x.
- TPUs: JAX achieved a 1652x speedup compared to uncompiled code, and a total 4084x reduction in runtime when using quantized arithmetic.
Conclusion
High-level frameworks like JAX and TVM effectively solve the performance portability challenge for Transformer models. While JAX offers superior development speed, TVM provides a path for deep optimization. Future work will focus on reducing TVM's tuning time and exploring the accuracy trade-offs of low-precision execution.
Summary updated