Abstract
Mixed-precision training is essential for scaling deep learning models, yet the JAX ecosystem has lacked a modern, maintained library for this purpose. We introduce MPX, an open-source toolbox that provides seamless mixed-precision support for JAX, specifically targeting integration with Equinox and Flax. MPX implements automated PyTree casting and dynamic loss scaling to prevent numerical instability. Our evaluation shows that MPX reduces VRAM usage by up to 1.8x and improves training throughput by up to 1.7x on consumer-grade hardware and 1.57x on high-performance clusters, without compromising model accuracy.
Introduction
While JAX has become a favorite for high-performance machine learning due to its just-in-time (JIT) compilation and sharding capabilities, it has historically lacked a robust, well-maintained solution for mixed-precision training. Mixed-precision training—using 16-bit floats (FP16 or BF16) for most computations while keeping master weights in 32-bit—is essential for training large-scale models. It reduces memory overhead and leverages specialized hardware like Tensor Cores.
The previous standard, a library called JMP, is no longer maintained and struggles with the complex "PyTree" data structures used in modern JAX libraries. Without a dedicated tool, developers often face the "gradient underflow" problem, where small gradients are rounded to zero in 16-bit precision, causing training to fail. MPX is designed to address these challenges as a flexible, "drop-in" replacement for standard training functions.
The MPX Toolbox
MPX is built to minimize friction for researchers by mimicking the functional style of libraries like Equinox. Its core features include:
- PyTree Casting: MPX provides utilities to automatically traverse and cast nested data structures (PyTrees) to the desired precision while intelligently ignoring integers, such as random seeds.
- Dynamic Loss Scaling: To prevent gradient underflow, MPX implements a heuristic that multiplies the loss by a scaling factor. If gradients overflow (become infinite), the scale is reduced and the step is skipped; if gradients remain stable, the scale is gradually increased.
- Equinox Integration: MPX offers versions of
filter_gradand optimizer wrappers that mirror the Equinox API. This allows users to convert a full-precision pipeline to mixed precision by changing only a few lines of code. - Selective Precision: Users can use
force_full_precisionto ensure numerically sensitive operations—like Softmax or large sums—are always executed in 32-bit to maintain stability.
Experimental Setup
The researchers evaluated MPX by training Vision Transformers (ViT) on two standard benchmarks:
- CIFAR-100: Used for testing on a mid-level desktop setup featuring an Nvidia RTX 4070.
- ImageNet-1k: Used for large-scale testing on a high-performance cluster equipped with 4x Nvidia H100 GPUs.
Results and Discussion
The empirical results demonstrate that MPX delivers on its promise of efficiency without sacrificing the model's final accuracy.
Memory Efficiency
On a desktop GPU, MPX reduced VRAM consumption by a factor of 1.8x. This allows for significantly larger batch sizes on consumer-grade hardware, which is often memory-constrained.
Training Throughput
Training speed improved significantly across different hardware configurations. Training was 1.7x faster on the RTX 4070, primarily due to reduced memory bandwidth bottlenecks. On the H100 cluster, where hardware is specifically optimized for 16-bit math, the speedup reached up to 1.57x.
Conclusion
MPX provides a modern, efficient solution for mixed-precision training in JAX. By leveraging JAX's internal type-promotion rules and providing a functional API, it removes the "boilerplate tax" associated with manual loss scaling. While extremely deep architectures may still require manual tuning of selective precision, MPX offers a robust starting point for the community. The source code is available on GitHub, making it a ready-to-use resource for JAX researchers.
Summary updated