Numerical Non-Determinism in Parallel Deep Learning Hardware

63 concepts39 papers2 proposals

In deep learning, we often treat neural networks as deterministic mathematical functions. We expect that if we freeze the weights, fix the random seed, and feed in the exact same input, we will get the exact same output.

In reality, this assumption breaks down at the hardware level. When the same model runs on different GPU architectures, varying batch sizes, or alternative software backends, its outputs drift. This phenomenon—hardware-induced numerical non-determinism—is not driven by pseudo-random number generators, but by the fundamental physics of parallel computing and the limits of computer arithmetic.

The Core Mechanism: Non-Associative Addition

To understand why identical models diverge, we must look at how computers handle fractional numbers. Computer hardware represents real numbers using floating-point formats (such as FP32, FP16, or BF16), which have a fixed number of bits for the significand (mantissa) and the exponent. Because precision is finite, every floating-point operation requires a rounding step.

This introduces a critical mathematical quirk: floating-point addition is non-associative. In pure mathematics, $(a + b) + c$ always equals $a + (b + c)$. In floating-point arithmetic, it does not.

Consider a simple intuitive example in a system with limited precision:

  • Calculate $(1.0 + 10^{16}) - 10^{16}$: The first operation $1.0 + 10^{16}$ rounds down to $10^{16}$ because the smaller number is completely lost in the limited precision of the larger number's scale. Subtracting $10^{16}$ leaves us with $0.0$.
  • Calculate $1.0 + (10^{16} - 10^{16})$: The inner operation cancels out perfectly to $0.0$, leaving us with $1.0$.

The final result depends entirely on the order of operations.

Dynamic Parallel Reduction (Order A)      Dynamic Parallel Reduction (Order B)
       (Thread 1)   (Thread 2)                   (Thread 2)   (Thread 3)
         a  +  b       c  +  d                     b  +  c       a  +  d
          \   /         \   /                       \   /         \   /
         (a + b)       (c + d)                     (b + c)       (a + d)
             \           /                             \           /
           ((a + b) + (c + d))                       ((b + c) + (a + d))
             [Result Vector A]                         [Result Vector B]

In modern accelerators like GPUs, TPUs, and NPUs, operations like matrix multiplications and convolutions are split across thousands of parallel threads. To maximize throughput, these threads execute asynchronously. When computing a massive sum (a parallel reduction), the exact sequence in which threads finish and contribute their values depends on dynamic runtime factors: hardware scheduling, memory latency, power management, and batch sizes.

As a result, the reduction tree changes shape between runs or platforms. Every reordering introduces a slightly different sequence of rounding errors.

How Hardware and Software Platforms Diverge

As explored by Schlögl et al., these tiny rounding errors accumulate as data propagates through dozens of neural network layers. By the time the activations reach the final layer, the numerical deviations can be large enough to alter the model's final predictions, especially for inputs residing near a decision boundary.

This divergence manifests across several boundaries:

1. Cross-Architecture Variance

Different GPU generations (e.g., NVIDIA T4 vs. A100) or entirely different hardware classes (CPUs vs. GPUs) utilize distinct parallel reduction strategies and hardware instructions. Yuan et al. demonstrate that even with greedy decoding (where the temperature is set to zero), Large Language Model (LLM) outputs vary significantly across different hardware setups and batch sizes due to these underlying parallel reduction differences.

2. Software Backend Discrepancies

Deep learning frameworks rely on low-level linear algebra libraries (such as NVIDIA cuBLAS, Intel MKL, or Apple Accelerate) to execute matrix operations. Möller showed that these libraries implement mathematically equivalent operations differently. By engineering "Chimera examples"—adversarial inputs designed to exploit these library-specific rounding behaviors—Möller forced identical models to produce conflicting predictions depending solely on the linear algebra backend running the computation.

3. Framework and Kernel Fusion

Libraries like JAX, PyTorch, and TensorFlow interface with hardware kernels in unique ways. Optimization techniques like kernel fusion—which combines multiple mathematical operations into a single hardware instruction to save memory bandwidth—alter the precision and ordering of operations. Zhang et al. leveraged these subtle, platform-specific numerical signatures to develop Hardware and Software Platform Inference (HSPI), a technique capable of identifying the exact GPU model and software stack hosting a model simply by analyzing its raw numerical outputs.

The Cascading Impact on Downstream Applications

While a variance of $10^{-6}$ in a single logit might seem negligible, it creates severe failure modes in systems that require absolute bit-level consistency.

Brittle Pipeline Failures

In neural data compression, an encoder and a decoder must remain in perfect mathematical alignment. As Ballé et al. note, when a model is deployed across heterogeneous environments (such as compressing on a cloud GPU and decompressing on a mobile CPU), floating-point rounding errors break the probability synchronization required for entropy coding, causing catastrophic decoding failures. Similarly, Adler et al. highlight that LLM-based lossless compression algorithms fail if the encoder and decoder disagree on a token's probability by even a tiny fraction due to hardware non-determinism.

The Verification and Auditing Gap

When outsourcing model training or inference to third-party cloud providers, users often want to verify that the computation was executed correctly without re-running the entire workload. However, as Srivastava et al. explain, hardware-induced numerical divergence causes model weights to drift during training even when using identical random seeds, rendering simple weight-equivalence checks useless for auditing. Furthermore, a paper on neural network deployment reveals that these hardware quirks create a "soundness gap," allowing adversaries to construct deployment-sensitive backdoors that trigger malicious behavior on specific hardware platforms while remaining completely invisible to formal safety verifiers running on others.

Fairness and Performance Instability

Numerical non-determinism does not impact all data points equally. Nelaturu et al. discovered that switching the underlying hardware architecture introduces stochastic noise that disproportionately degrades the accuracy of minority demographic groups, turning hardware selection into an unintended fairness issue. In computer vision pipelines, Eryilmaz et al. observed that parallel execution orders in CUDA kernels can cause performance fluctuations of up to 5% on identical tasks, complicating model evaluation in high-stakes fields like medical imaging.

Mitigations and Solutions

To combat this cross-platform divergence, researchers have developed several hardware-aware software strategies:

  • Deterministic Integer Arithmetic: Ballé et al. introduced Integer Networks, which replace fragile floating-point operations with deterministic integer arithmetic, guaranteeing bit-exact reproducibility across any CPU or GPU platform at the cost of standard floating-point flexibility.
  • Hybrid Precision Storage: To stabilize LLM inference without the massive memory overhead of full FP32 execution, Yuan et al. proposed LayerCast. This approach performs critical parallel reductions in high-precision FP32 while maintaining model weights in BF16, neutralizing the rounding variations introduced by hardware-specific parallel reduction trees.
  • Boundary Logging: For verifiable training, Srivastava et al. designed a framework that tracks and logs specific rounding decisions only when floating-point operations occur dangerously close to a rounding boundary. This allows external auditors to perfectly synchronize their internal state and achieve bit-wise replication across entirely different GPU architectures.
  • Statistical Isolation of Noise: Rather than trying to eliminate non-determinism entirely, Kübler et al. developed a statistical framework based on McNemar's test. By evaluating accuracy shifts on a per-sample basis rather than using aggregate scores, their method successfully isolates harmless, hardware-induced numerical variations from true model degradation during optimization.

Key Foundational Work

Language model integration based on memory control for sequence to sequence speech recognition (2018) — contributed 1 concept elements to this direction.
Tensor2Tensor for Neural Machine Translation (2018) — contributed 1 concept elements to this direction.
Large Scale Evolution of Convolutional Neural Networks Using Volunteer Computing (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hardware_induced_non_determinism_b34f8397,
  title = {hardware-induced non-determinism},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hardware-induced-non-determinism/rdr_b34f8397c8f0498da3db6a59947726b4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hardware_induced_non_determinism_b34f8397_v1,
  title = {hardware-induced non-determinism},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hardware-induced-non-determinism/rdr_b34f8397c8f0498da3db6a59947726b4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake