Numerical Inconsistency Across Heterogeneous Machine Learning Hardware

This direction is split into other directions: hardware-induced non-determinism. Its last version stays readable and citable here.

Version 1 of 1145 papers231 conceptsfrozen 2026-10-04

In the idealized world of computer science, a neural network is a deterministic function: given a fixed set of weights and a fixed input, it should always produce the same output. However, as researchers like Alexander Schlögl, Nora Hofer, and Rainer Böhme demonstrate in Causes and Effects of Unanticipated Numerical Deviations in Neural Network Inference Frameworks, this assumption of bit-level determinism is false in modern high-performance computing.

When the same model is deployed across different GPUs, CPUs, or software stacks, the outputs diverge. These discrepancies are not just "noise"; they represent a fundamental challenge to the security, reproducibility, and reliability of artificial intelligence.

The Root of the Drift: Non-Associative Arithmetic

The primary culprit behind these deviations is the non-associative nature of floating-point arithmetic. In standard mathematics, $(a + b) + c$ is always equal to $a + (b + c)$. In the finite-precision world of computers, this is not guaranteed. Because floating-point numbers have limited significant digits, the order in which you add a sequence of numbers determines how rounding errors accumulate.

Modern deep learning relies on massive parallelism. To maximize speed, hardware like NVIDIA Tensor Cores or AMD Matrix Cores—reverse-engineered by Mao Yang and colleagues in MMA-Sim—reorders operations based on available threads, batch sizes, and kernel optimizations. As Hao Li notes in Give Me FP32 or Give Me Death?, even changing the number of GPUs or the batch size can lead to a 9% swing in accuracy for reasoning models because the summation order of gradients and activations changes.

Turning a Bug into a Fingerprint

While these numerical deviations are often seen as a nuisance for reproducibility, researchers have found ways to exploit them for system verification. Cheng Zhang, Ilia Shumailov, and their team developed Hardware and Software Platform Inference (HSPI), a technique that identifies the specific GPU and software stack running a model just by analyzing its outputs.

By measuring the subtle "numerical signature" left by hardware-specific rounding and kernel fusion, users can verify if a cloud provider is actually using the premium hardware they advertised. This transforms the "noise" of floating-point math into a cryptographic-like tool for auditing black-box APIs.

The Challenge of Trustworthy Inference

The lack of bit-level agreement creates a "noise floor" that makes it difficult to distinguish between legitimate hardware variance and malicious tampering. In decentralized AI marketplaces, a provider might try to save costs by serving a smaller, quantized model instead of the full version—a "bait-and-switch" tactic.

To combat this, protocols like DiFR by Adam Karvonen and VeriLLM by Xinyuan Song use statistical calibration. Instead of requiring an exact bit-for-bit match, which is impossible across heterogeneous hardware, these systems establish a "similarity threshold." They use compressed activation fingerprints to ensure the model's behavior remains within the expected range of natural numerical drift, effectively separating hardware-induced jitter from intentional model degradation.

Restoring Determinism

For applications where exactness is non-negotiable—such as neural data compression or verifiable training—researchers are moving toward deterministic alternatives. Jona Ballé and Johannes Ballé proposed Integer Networks, which replace floating-point math with fixed-point integer arithmetic to ensure identical results across CPUs and GPUs.

In the realm of Large Language Models, Jiayi Yuan and Hao Li introduced LayerCast, a hybrid approach. By performing critical accumulations in higher-precision FP32 while keeping weights in BF16, they restore deterministic behavior without the massive memory overhead of full 32-bit computation.

Why It Matters

The research direction across these 99 papers suggests that we can no longer treat machine learning hardware as a transparent utility. As A. Feder Cooper and Jonathan Frankle argue in Non-Determinism and the Lawlessness of Machine Learning Code, the stochasticity of the hardware itself undermines our ability to provide safety guarantees. Understanding the "why" of numerical inconsistency is the first step toward building AI systems that are not just fast, but provably consistent across the heterogeneous landscape of modern computing.

Papers in this version

Showing 60 of 145 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_numerical_inconsistency_across_heterogen_c80c5a58,
  title = {Numerical Inconsistency Across Heterogeneous Machine Learning Hardware},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/numerical-inconsistency-across-heterogeneous-machine-learning-hardware/rdr_c80c5a584ee54d09bcb3305d62c14db4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_numerical_inconsistency_across_heterogen_c80c5a58_v1,
  title = {Numerical Inconsistency Across Heterogeneous Machine Learning Hardware},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/numerical-inconsistency-across-heterogeneous-machine-learning-hardware/rdr_c80c5a584ee54d09bcb3305d62c14db4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}