MPX: Mixed Precision Training for JAX

Alexander Gräfe, Sebastian TrimpearXiv 2025 · DOI · arXiv

Abstract

Mixed-precision training is essential for scaling deep learning models, yet the JAX ecosystem has lacked a modern, maintained library for this purpose. We introduce MPX, an open-source toolbox that provides seamless mixed-precision support for JAX, specifically targeting integration with Equinox and Flax. MPX implements automated PyTree casting and dynamic loss scaling to prevent numerical instability. Our evaluation shows that MPX reduces VRAM usage by up to 1.8x and improves training throughput by up to 1.7x on consumer-grade hardware and 1.57x on high-performance clusters, without compromising model accuracy.

Introduction

While JAX has become a favorite for high-performance machine learning due to its just-in-time (JIT) compilation and sharding capabilities, it has historically lacked a robust, well-maintained solution for mixed-precision training. Mixed-precision training—using 16-bit floats (FP16 or BF16) for most computations while keeping master weights in 32-bit—is essential for training large-scale models. It reduces memory overhead and leverages specialized hardware like Tensor Cores.

The previous standard, a library called JMP, is no longer maintained and struggles with the complex "PyTree" data structures used in modern JAX libraries. Without a dedicated tool, developers often face the "gradient underflow" problem, where small gradients are rounded to zero in 16-bit precision, causing training to fail. MPX is designed to address these challenges as a flexible, "drop-in" replacement for standard training functions.

The MPX Toolbox

MPX is built to minimize friction for researchers by mimicking the functional style of libraries like Equinox. Its core features include:

  • PyTree Casting: MPX provides utilities to automatically traverse and cast nested data structures (PyTrees) to the desired precision while intelligently ignoring integers, such as random seeds.
  • Dynamic Loss Scaling: To prevent gradient underflow, MPX implements a heuristic that multiplies the loss by a scaling factor. If gradients overflow (become infinite), the scale is reduced and the step is skipped; if gradients remain stable, the scale is gradually increased.
  • Equinox Integration: MPX offers versions of filter_grad and optimizer wrappers that mirror the Equinox API. This allows users to convert a full-precision pipeline to mixed precision by changing only a few lines of code.
  • Selective Precision: Users can use force_full_precision to ensure numerically sensitive operations—like Softmax or large sums—are always executed in 32-bit to maintain stability.

Experimental Setup

The researchers evaluated MPX by training Vision Transformers (ViT) on two standard benchmarks:

  • CIFAR-100: Used for testing on a mid-level desktop setup featuring an Nvidia RTX 4070.
  • ImageNet-1k: Used for large-scale testing on a high-performance cluster equipped with 4x Nvidia H100 GPUs.

Results and Discussion

The empirical results demonstrate that MPX delivers on its promise of efficiency without sacrificing the model's final accuracy.

Memory Efficiency

On a desktop GPU, MPX reduced VRAM consumption by a factor of 1.8x. This allows for significantly larger batch sizes on consumer-grade hardware, which is often memory-constrained.

Comparison of GPU VRAM consumed for full precision and mixed precision as a function of batch size on a desktop PC.

Training Throughput

Training speed improved significantly across different hardware configurations. Training was 1.7x faster on the RTX 4070, primarily due to reduced memory bandwidth bottlenecks. On the H100 cluster, where hardware is specifically optimized for 16-bit math, the speedup reached up to 1.57x.

Comparison of training step times for full precision and mixed precision on (a) Desktop Computer (RTX 4070) and (b) High-Performance Cluster (4x H100).

Conclusion

MPX provides a modern, efficient solution for mixed-precision training in JAX. By leveraging JAX's internal type-promotion rules and providing a functional API, it removes the "boilerplate tax" associated with manual loss scaling. While extremely deep architectures may still require manual tuning of selective precision, MPX offers a robust starting point for the community. The source code is available on GitHub, making it a ready-to-use resource for JAX researchers.

Summary updated

Abstract

Mixed-precision training has emerged as an indispensable tool for enhancing the efficiency of neural network training in recent years. Concurrently, JAX has grown in popularity as a versatile machine learning toolbox. However, it currently lacks robust support for mixed-precision training. We propose MPX, a mixed-precision training toolbox for JAX that simplifies and accelerates the training of large-scale neural networks while preserving model accuracy. MPX seamlessly integrates with popular toolboxes such as Equinox and Flax, allowing users to convert full-precision pipelines to mixed-precision versions with minimal modifications. By casting both inputs and outputs to half precision, and introducing a dynamic loss-scaling mechanism, MPX alleviates issues like gradient underflow and overflow that commonly arise in half precision computations. Its design inherits critical features from JAX's type-promotion behavior, ensuring that operations take place in the correct precision and allowing for selective enforcement of full precision where needed (e.g., sums, means, or softmax). MPX further provides wrappers for automatic creation and management of mixed-precision gradients and optimizers, enabling straightforward integration into existing JAX training pipelines. MPX's source code, documentation, and usage examples are available at github.com/Data-Science-in-Mechanical-Engineering/mixed_precision_for_JAX .
BibTeX
@misc{grafe2025mpx,
  author = {Alexander Gräfe and Sebastian Trimpe},
  title = {{MPX: Mixed Precision Training for JAX}},
  year = {2025},
  eprint = {2507.03312},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.03312},
}
Figure