TL;DR
CertiSum introduces a method to achieve bitwise-identical training for Large Language Models using the 4-bit MXFP4 format across different GPU vendors and parallelism layouts. By certifying when native hardware accumulation is already exact and providing efficient fallbacks for the rare cases it isn't, CertiSum enables reproducible low-precision training with significantly lower overhead than traditional exact-summation techniques.
What Problem Does This Paper Address?
Training large models is increasingly moving toward narrower numerical formats like MXFP4 to save on compute and memory. However, these formats suffer from a major headache: non-determinism. Because floating-point addition is non-associative, the final result of a large sum depends on the order in which numbers are added. This order changes whenever you change your parallelism strategy (e.g., switching from 2 GPUs to 4) or even your hardware vendor (e.g., moving from NVIDIA to AMD).
As shown in Figure 1, this isn't just a theoretical concern. In MXFP4 training, changing the number of GPUs can cause nearly 38% of greedy next-token predictions to change, compared to only ~2% for BF16. This "numerical drift" makes debugging a nightmare, complicates training audits, and can cause training and inference engines to disagree in reinforcement learning setups.
Methods
The authors' core observation is that native FP32 accumulation of MXFP4 products is actually exact most of the time, regardless of the summation order. This is because MXFP4 elements have very few significand bits, so their products often fit within the 24-bit significand of an FP32 accumulator without needing to round.
CertiSum exploits this through a "certify-and-repair" workflow (Figure 3):
- Cheap Certification: During the quantization step, CertiSum collects simple row and block statistics (like sums of absolute values).
- Routing: It uses these statistics to evaluate a hierarchy of bounds. If a bound proves that the native FP32 sum will be exact, it uses the fast, native GPU kernels.
- Exact Repair: In the rare cases where certification fails (or for operations like attention where it's harder to guarantee), CertiSum routes the computation to specialized "exact" kernels that use techniques like FP64 accumulation or multi-word integer arithmetic to ensure a bit-perfect result.
- Layout Invariance: To handle multi-GPU setups, CertiSum ensures that quantization blocks are tied to global tensor positions and that partial sums from different GPUs are merged exactly without intermediate rounding.
Datasets
The researchers evaluated CertiSum using several prominent LLM training datasets:
- FineWeb-Edu: A high-quality corpus of educational web text.
- The Stack v3: A massive dataset of source code.
- C4: Used for smaller-scale validation.
They tested these on models ranging from 8B to 14B parameters, including Qwen3-8B-Base, Qwen3-14B-Base, and Llama-3.1-8B.
Key Insights and Ideas
- The 99.2% Rule: The most striking finding is that for a typical LLM training run, over 99.2% of GEMM outputs can be exactly accumulated in FP32 regardless of the order (Figure 2). This justifies the "certify-and-repair" approach—you almost never have to pay the full price for exactness.
- Hardware Invariance is Possible: By ensuring exactness before the final rounding step, CertiSum achieves bitwise identity even between different hardware architectures like AMD's MI350X and NVIDIA's GB200.
- Zero-Cost Statistics: Because the statistics needed for certification are gathered during the mandatory quantization pass, the overhead for the "check" itself is nearly zero.
Key Results
- Bitwise Identity: CertiSum successfully preserved training states bit-for-bit across AMD and NVIDIA GPUs, and across changes in data, tensor, and context parallelism.
- Performance: While adding 11.8–60.5% latency compared to "vanilla" (non-deterministic) MXFP4, CertiSum remained up to 1.24× faster than BF16 and 1.70× faster than deterministic BF16 implementations.
- Inference Consistency: In greedy generation tests, CertiSum produced identical token sequences regardless of batch size or GPU count, whereas standard MXFP4 failed to match its own single-GPU reference in every case.
Limitations and Future Work
- Small Shape Overhead: The relative overhead is higher for smaller matrix shapes because the fixed costs of extra kernel launches and host-side routing become a larger portion of the total step time.
- Format Specificity: While the method works well for MXFP4 and MXFP6, wider formats like MXFP8 have more significand bits, which would likely lead to more certification failures and higher repair costs.
- Framework Integration: Currently, CertiSum requires custom implementations of GEMM, attention, and collective operators. Wider adoption would require deeper integration into standard frameworks like PyTorch or JAX.
The authors suggest that future work could explore even cheaper certification methods or hardware-assisted exact accumulation to further drive down the overhead of reproducibility.
Summary updated